Normal Distribution to Mean Squared Error Loss

Pre-requisite

This article assumes you are comfortable with the bridge between probability distributions and loss functions.

In particular, we will use these ideas:

• likelihood
• maximum likelihood
• IID: “same setup, fresh wiggle”
• negative log-likelihood

If those ideas feel familiar, keep going.

If they feel rusty, read this foundation article first:

Read the setup first From Probability Distributions to Loss Functions →

Normal and the Free Throw

Imagine a basketball player taking the same shot again and again.

Same player.
Same hoop.
Same spot on the court.

But the ball does not land in the exact same place every time.

One shot goes in.
One misses a little left.
One misses a little right.

A few miss farther than usual.

For the chart, we turn each shot into a signed error.

A shot near the middle of the hoop has error near 0.
A left miss gets a negative error.
A right miss gets a positive error.

That is the clue.

The shot is repeatable, but not identical.

You can spot this same pattern everywhere.

Weight reading: Step on the same scale a few times. The number wiggles a little, but most readings stay near your usual weight.

Pancake batter: Try pouring the same pancake again and again. Most pancakes come out close in size. A few are too small or too large.

Thermostat reading: A room does not jump wildly every second. Most readings stay close, with tiny electrical and environmental wiggles.

Here is the pattern hiding underneath all those examples.

You repeat the same kind of event.

A basketball player takes the same shot.
A scale measures the same body.
A thermostat reads the same room.

The setup is stable.

But the result is not a photocopy.

It wiggles.

Most wiggles stay close to the usual result.

A few wander farther away.

Very large wiggles are possible, but rare.

That is the Normal world:

Same setup.
Small wiggles.
Rare big swings.

The target stays put. Reality jitters.

The hoop is the center. The misses make the bell.

Normal Visualized

Use the component below to watch the basketball story become a distribution.

Each shot becomes one signed error:

$$\text{error} = \text{landing position} - \text{center of hoop}$$

Samples Play

Now notice the shape of that curve.

It is tall in the middle and thin in the tails.

The curve is tallest near the center because errors near the mean are most likely.

As errors move farther left or right, the curve gets lower because those errors are less likely.

The curve is symmetric around the center.

Fold it at the mean, and the left side would mirror the right.

In this plot:

• 0 means the middle of the hoop.
• Shots near 0 are the usual case.
• Left misses fall below 0.
• Right misses rise above 0.
• Wild misses live in the thin tails.

That mound is the familiar bell curve.
Its formal name is the normal distribution.
You may also hear it called a Gaussian distribution.

Naming the model

Now we can name the model:

$$\epsilon \sim \mathcal{N}(\mu, \sigma^2)$$

Read it as:

The error $\epsilon$ follows a normal distribution.

The normal model has two knobs.

The first knob is $\mu$.

That is the center of the bell curve.

In the basketball story, $\mu$ marks where the shot usually lands.
If $\mu$ is near 0, the player is usually scoring, not missing left or right.

The second knob is $\sigma^2$, the variance.

It controls how spread out the misses are.

We often talk about $\sigma$, the standard deviation, because it uses the same units as the error.

Small $\sigma$ means the player is consistent.
Large $\sigma$ means the shots spread out more.

---

The Normal Density Equation

The equation for the normal density curve is:

$$p(\epsilon) = \frac{1}{\sigma\sqrt{2\pi}} \exp\left( -\frac{(\epsilon-\mu)^2}{2\sigma^2} \right)$$

$\mu$ is the center of the error distribution.

$\sigma$ controls the spread.

Here, $\epsilon$ means one shot error:

$$\epsilon = \text{shot landing position} - \text{center of hoop}$$

Now look at the part that measures distance from the center:

$$(\epsilon - \mu)^2$$

For a well-aimed shot pattern, the average error should be near 0.

So we use the zero-centered Normal error model:

$$\epsilon \sim \mathcal{N}(0, \sigma^2)$$

That means $\mu = 0$.

So the distance term becomes:

$$(\epsilon - 0)^2 = \epsilon^2$$

And the Normal density becomes:

$$p(\epsilon) = \frac{1}{\sigma\sqrt{2\pi}} \exp\left( -\frac{\epsilon^2}{2\sigma^2} \right)$$

---

From Normal Errors to MSE

This section is the derivation.

If you only want the intuition, you can safely skip ahead to Interpreting MSE.

We have the Normal distribution.

Now we convert that distribution into a loss function.

The conversion uses maximum likelihood estimation (MLE).

We covered that bridge in the foundation article, so we will use it here without rebuilding every step.

Need the bridge first? From Probability Distributions to Loss Functions →

Here, we only ask:

When the errors follow a Normal distribution, what loss function appears?

For one error, the model asks: how plausible does this error look under the Normal curve?

$$p(\epsilon^{(1)})$$

Here, the superscript $(i)$ is just the example number.
It does not mean “raise to the power $i$.”
So $\epsilon^{(1)}$ means the first error, $\epsilon^{(2)}$ means the second error, and so on.

For the whole dataset, the model needs every error to look likely.

So it multiplies the probabilities together:

$$p(\epsilon^{(1)}) \times p(\epsilon^{(2)}) \times \ldots \times p(\epsilon^{(n)})$$

Instead of writing that long chain every time, we compress it with product notation:

$$L = \prod_{i=1}^{n} p(\epsilon^{(i)})$$

Apply the negative log:

$$-\log(L) = -\log\left( \prod_{i=1}^{n} p(\epsilon^{(i)}) \right)$$

Log turns the product into a sum:

$$-\log(L) = -\sum_{i=1}^{n} \log p(\epsilon^{(i)}) \tag{1}$$

Equation (1) is a sum over errors. So let’s simplify one term first:

$$-\log p(\epsilon^{(i)})$$

Once we know this piece, we can sum it over all $n$ shots.

For a zero-centered normal error:

$$p(\epsilon^{(i)}) = \frac{1}{\sigma\sqrt{2\pi}} \exp\left( -\frac{(\epsilon^{(i)})^2}{2\sigma^2} \right)$$

Now take the negative log:

$$-\log p(\epsilon^{(i)}) = -\log\left[ \frac{1}{\sigma\sqrt{2\pi}} \exp\left( -\frac{(\epsilon^{(i)})^2}{2\sigma^2} \right) \right]$$

Split the product inside the log:

$$\begin{aligned} -\log p(\epsilon^{(i)}) &= -\log\left(\frac{1}{\sigma\sqrt{2\pi}}\right) \\ &\qquad - \log\left[ \exp\left( -\frac{(\epsilon^{(i)})^2}{2\sigma^2} \right) \right] \end{aligned}$$

Clean up the first term:

$$-\log\left(\frac{1}{\sigma\sqrt{2\pi}}\right) = \log(\sigma\sqrt{2\pi})$$

Clean up the second term:

$$-\log\left[ \exp\left( -\frac{(\epsilon^{(i)})^2}{2\sigma^2} \right) \right] = \frac{(\epsilon^{(i)})^2}{2\sigma^2}$$

So one shot’s negative log likelihood becomes:

$$-\log p(\epsilon^{(i)}) = \log(\sigma\sqrt{2\pi}) + \frac{(\epsilon^{(i)})^2}{2\sigma^2}$$

Now return to Equation (1):

$$-\log(L) = -\sum_{i=1}^{n} \log p(\epsilon^{(i)})$$

We just simplified one repeated term:

$$-\log p(\epsilon^{(i)}) = \log(\sigma\sqrt{2\pi}) + \frac{(\epsilon^{(i)})^2}{2\sigma^2}$$

Now plug that back into Equation (1):

$$-\log(L) = \sum_{i=1}^{n} \left[ \log(\sigma\sqrt{2\pi}) + \frac{(\epsilon^{(i)})^2}{2\sigma^2} \right]$$

The first part is the same for every shot, so it adds up to $n$ copies:

$$-\log(L) = n\log(\sigma\sqrt{2\pi}) + \frac{1}{2\sigma^2} \sum_{i=1}^{n} (\epsilon^{(i)})^2$$

Now fill the $\epsilon^{(i)}$ using EGO: Error = Ground truth − Output

$$\epsilon^{(i)} = y^{(i)} - \hat{y}^{(i)}$$

So the full negative log likelihood becomes:

$$\begin{aligned} -\log(L) &= \underbrace{ n\log(\sigma\sqrt{2\pi}) }_{\substack{\text{constant}}} \\ &\qquad + \underbrace{ \frac{1}{2\sigma^2} }_{\substack{\text{fixed}\\\text{scale}}} \hspace{0.8em} \underbrace{ \sum_{i=1}^{n} \left( y^{(i)} - \hat{y}^{(i)} \right)^2 }_{\substack{\text{prediction}\\\text{errors}}} \end{aligned}$$

If we treat $\sigma$ as fixed, the first two pieces are dead weight for training.

They do not depend on the prediction $\hat{y}$.

So dropping them does not change which model wins.

The part the model minimizes is:

$$\sum_{i=1}^{n} \left( y^{(i)} - \hat{y}^{(i)} \right)^2$$

This is the sum of squared errors.

But the raw sum grows with dataset size.

A 1,000-shot dataset naturally creates a bigger penalty pile than a 10-shot dataset.

To make the score comparable across datasets and training batches, we average the pile.

That gives us the Mean Squared Error:

$$\text{MSE} = \frac{1}{n} \sum_{i=1}^{n} \left( y^{(i)} - \hat{y}^{(i)} \right)^2$$

So minimizing negative log likelihood is equivalent to minimizing MSE.

---

Interpreting MSE

Here is the formula we finally care about:

$$\text{MSE} = \frac{1}{n} \sum_{i=1}^{n} \left( y^{(i)} - \hat{y}^{(i)} \right)^2$$

Here, $(i)$ is the row number, not an exponent.

So $y^{(1)}$ means the first real value, and $\hat{y}^{(1)}$ means the first prediction.

The formula looks heavier than the question it is asking.

MSE is easiest to read as a big-miss amplifier.

It is asking:

How large is the average squared miss?

Read it with EGO:

Error = Ground truth − Output

$$\text{error}^{(i)} = y^{(i)} - \hat{y}^{(i)}$$

So MSE squares each error, piles the squared errors together, and averages the pile.

Let’s calculate that penalty.

For one example, ignore the average for a moment:

$$\text{loss} = \left( y - \hat{y} \right)^2$$

Using EGO:

$$\text{loss} = \text{error}^2$$

Now watch what the square does.

Tiny Miss: Error 1

Suppose the prediction misses by 1.

$$\text{loss} = 1^2 = 1$$

The model was close.

So the penalty stays small.

Medium Miss: Error 2

Now suppose the prediction misses by 2.

$$\text{loss} = 2^2 = 4$$

The error only doubled.

But the penalty became four times larger than the tiny miss.

That is the square starting to bite.

Wild Miss: Error 10

Now suppose the prediction misses by 10.

$$\text{loss} = 10^2 = 100$$

That is the whole personality of MSE.

It does not merely notice big misses.

It makes them expensive.

Average the Squared Errors

Now average the errors we just calculated:

$$\text{MSE} = \frac{1}{n} \sum_{i=1}^{n} \left( y^{(i)} - \hat{y}^{(i)} \right)^2$$

Now fill it with the three misses we just calculated:

$$\begin{aligned} \text{MSE} &= \frac{ \text{error}_1^2 + \text{error}_2^2 + \text{error}_3^2 }{3} \\ &= \frac{1^2 + 2^2 + 10^2}{3} \\ &= 35 \end{aligned}$$

That is the bill.

One wild miss made the average expensive.

The Bottom of the Bowl

So MSE does not simply punish being wrong.

It punishes being far away.

If the prediction lands close to the target, the loss stays low.

If the prediction drifts far away, the square builds a wall.

The optimizer’s job is to roll down that bowl until the prediction gets as close as possible to the target.

For one example, the bottom is where:

$$\hat{y} \approx y$$

For many examples, MSE asks for the model that makes the average squared miss as small as possible.

That matches the bell curve story.

Under the Normal distribution, small errors near the center are common.

Big errors live in the thin tails.

So when a prediction lands far away, MSE reacts strongly.

MSE hates big misses because the bell curve says big misses should be rare.

When MSE Works

MSE works when the bell curve story is believable.

Most misses are small.

Wild airballs are rare.

But if one weird shot dominates the average,

the square stops helping and starts bullying the model.

---

RMSE: Translating the Bill Back

MSE gives us the average squared miss.

That is useful because the square makes big misses expensive.

But it also creates one awkward problem.

The units get squared too.

If the model predicts house prices in dollars, MSE is measured in dollars squared.

If the model predicts temperature in degrees, MSE is measured in degrees squared.

That is not how humans talk.

So we take the square root.

$$\text{RMSE} = \sqrt{\text{MSE}}$$

Fully written:

$$\text{RMSE} = \sqrt{ \frac{1}{n} \sum_{i=1}^{n} \left( y^{(i)} - \hat{y}^{(i)} \right)^2 }$$

RMSE means Root Mean Squared Error.

It is still built from the same squared-error pile.

So big misses still matter.

But after the square root, the final number comes back in the original units.

Return to our three misses:

$$1,\ 2,\ 10$$

Their squared errors were:

$$1^2,\ 2^2,\ 10^2$$

So MSE was:

$$\text{MSE} = \frac{1 + 4 + 100}{3} = 35$$

Now take the square root:

$$\text{RMSE} = \sqrt{35} \approx 5.9$$

That means the model’s typical miss is about 5.9 units, measured back in the same units as the original target.

MSE is better for seeing the penalty machine.

RMSE is better for explaining the error to a human.

MSE is the squared bill.
RMSE is the bill translated back into real-world units.

---

Where Will You See MSE?

MSE shows up when a model predicts a number.

You will see it in linear regression, neural networks, and regression trees.

In linear regression, “least squares” means the model is minimizing squared errors.

In neural networks, MSE is common when the output is a price, temperature, coordinate, or other continuous value.

In regression trees, splits are often chosen by asking:

Which split reduces squared error the most?

The Scaling Trap

MSE squares raw errors, so scale can quietly take over.

If a model predicts both player salary and player height, salary errors may dominate the loss just because salary numbers are bigger.

That does not mean salary is the only thing that matters.

It means the bigger scale got the louder microphone.

For a deeper look at scaling during preprocessing, read:

Feature Scaling: Making Elephants and Bees Comparable

---

What Did MSE Actually Learn?

It learns the center, not the spread.

The Normal distribution has two knobs:

$\mu$ marks the center.
$\sigma$ controls the spread.

So when we train a model with MSE, what do we actually win?

Do we learn $\mu$?
Do we learn $\sigma$?

The Prize

Look back at the zero-centered error assumption:

$$\epsilon \sim \mathcal{N}(0, \sigma^2)$$

Now use EGO:

$$\epsilon = y - \hat{y}$$

So:

$$y - \hat{y} \sim \mathcal{N}(0, \sigma^2)$$

Move $\hat{y}$ to the other side:

$$y \sim \mathcal{N}(\hat{y}, \sigma^2)$$

But in regression, $\hat{y}$ comes from an input $x$.

The model predicts:

$$\hat{y} = f(x)$$

So more precisely:

$$y \mid x \sim \mathcal{N}( \underbrace{f(x)}_{\mu(x)}, \sigma^2 )$$

In a Normal distribution, the first slot is the center.

Here, the first slot is $f(x)$.

So the model’s prediction is holding the place of $\mu$.

For each input $x$, the model predicts the center of the bell curve.

The spread $\sigma^2$ is still there, but basic MSE treats it as fixed.

So yes:

MSE trains your model to estimate the conditional mean.

In plain English:

The model looks at the shot setup and asks:

Where does the ball usually land from here?

That usual landing point is $\hat{y}$.

The actual shot $y$ may still wiggle around it, but the model’s job is to find the center of the landing cluster.

Why Not Learn the Spread?

Why not fix the center and learn $\sigma$ instead?

Imagine telling a coach:

I do not know where this player keeps shooting, but I can tell you exactly how messy the misses are.

That is not the first thing we need.

The first job is to find where the player keeps shooting.

For every new input $x$, the model looks at the setup and asks:

Where is the center of this shot cluster?

That is what $\hat{y}$ does.

$$\hat{y}(x) \approx \mu(x)$$

Basic MSE focuses on finding that center.

It assumes the spread, $\sigma$, is fixed background noise.

What If We Want Both?

We can learn both.

We just need a richer model.

Instead of one output, the model would output two things:

1. the predicted center, $\hat{y}(x)$
2. the predicted spread, $\hat{\sigma}(x)$

Then we would keep the full Gaussian negative log-likelihood instead of simplifying it down to MSE.

That loss looks like this:

$$\text{Loss} = \log(\hat{\sigma}) + \frac{(y - \hat{y})^2}{2\hat{\sigma}^2}$$

Now the model can say:

Here is my best guess.

And also:

Here is how uncertain I am.

But standard MSE is the simpler version:

Find the center. Assume the width is fixed.

---

Before You Forget

Error

One shot’s miss. In EGO language: ground truth minus output. The hoop wanted one thing; the ball did another.

Error. One shot’s miss. In EGO language: ground truth minus output. The hoop wanted one thing; the ball did another.

---

Quiz

86% of people love quizzes after learning. Are you one of them?

★

Question 1 of 12 🏆 0 / 120 ⚡ Attempt 1 of 2

Question text

Next →