June 10, 2026 11 min read

Bernoulli Distribution to Binary Cross-Entropy Loss

Why Bernoulli yes/no outcomes lead to binary cross-entropy loss, using rain probabilities to explain two buckets, confident wrong guesses, and negative log penalties.

Stories by Sagar Kharel

Pre-requisite

This article assumes you are comfortable with the bridge between probability distributions and loss functions.

In particular, we will use these ideas:

likelihood
maximum likelihood
IID: “same setup, fresh wiggle”
negative log-likelihood

If those ideas feel familiar, keep going.

If they feel rusty, read this foundation article first:

Read the setup first From Probability Distributions to Loss Functions

Bernoulli and the Two-Bucket World

Imagine a weather forecaster on the evening news.

They look at the radar and say:

There is an 80% chance of rain tomorrow.

But tomorrow, you do not walk outside and get 80% wet.

The sky has only two endings:

It either rains, or it doesn’t.

That is the clue.

Bernoulli is for events with exactly two possible outcomes.

You can spot this same pattern everywhere.

Email filter: The system says, “I am 99% sure this email is spam.” But an email cannot be 99% spam. It either belongs in the inbox, or it belongs in the trash.

Medical test: The model says, “This looks highly likely to be positive.” But your body is not “highly likely” positive after reality arrives. The condition is present, or it is not.

Credit card swipe: The bank says, “This charge looks 10% fraudulent.” But a thief is not 10% stealing from you. It is either your purchase, or it is fraud.

Security login: The system says, “This login looks 40% suspicious.” But the login is not 40% attacker. It is either a real user, or it is an attack.

Machine failure: The sensor says, “There is a 70% chance this pump fails today.” But tomorrow, the pump is not 70% broken. It either failed, or it kept running.

Here is the pattern hiding inside all those examples.

Reality has only two buckets:

Rain or No rain
Spam or Not spam
Positive or Negative
Fraud or Not fraud

Before reality arrives, the model does not know which bucket will win.

So it pours probability between the two buckets.

Maybe 80% into Rain and 20% into No rain.

Maybe 10% into Fraud and 90% into Not fraud.

But reality does not arrive as a percentage.

You do not walk outside and get 80% wet.

The charge is not 10% fraud.

When the event happens, it lands in one bucket.

That is the Bernoulli world:

Two buckets.
A probability before.
A truth after.

The guess is a dial. Reality is a switch.

Bernoulli: To Bern or not Bern!

Bernoulli Visualized

Use the component below to watch the Bernoulli story become a distribution.

The left side shows the probability dial.

At 60%, the dial says:

P(\text{Rain}) = 0.60

and:

P(\text{No rain}) = 0.40

But the right side shows the important trick.

Bernoulli does not create a smooth curve.

It creates two bars.

One bar for 0.
One bar for 1.

In this chart:

0 means No rain.
1 means Rain.
The height of each bar is its probability.
The two bars always add up to 100%.

Probability dial 60%

P rain 60 percent. P no rain 40 percent.

BERNOULLI DISTRIBUTION

That is the Bernoulli shape.

Not a bell.

Not a tent.

Just two possible endings with probability split between them.

Bernoulli: To Bern or not Bern!

Naming the Model

We translate the yes/no outcome into numbers:

\text{Yes} = 1 \quad \quad \text{No} = 0

You can also think of this as success or failure.

In our weather story:

1 = \text{Rain}

0 = \text{No rain}

So when we write:

P(Y = 1)

we are asking: what is the probability that it will rain?

Bernoulli calls that probability $p$ :

P(Y = 1) = P(\text{Rain}) = p

And because there are only two buckets, the other bucket gets whatever is left:

P(Y = 0) = P(\text{No rain}) = 1 - p

So the model is:

Y \sim \text{Bernoulli}(p)

Read it as:

The outcome $Y$ follows a Bernoulli distribution with success probability $p$ .

That is why the Bernoulli distribution has two bars.

One bar for 0.

One bar for 1.

The height of each bar is the probability assigned to that bucket.

From Bernoulli to Binary Cross-Entropy

We have the Bernoulli distribution.

Now we convert that distribution into a loss function.

The conversion uses maximum likelihood estimation (MLE).

We covered that bridge in the foundation article, so we will use it here without rebuilding every step.

Need the bridge first? From Probability Distributions to Loss Functions

Here, we only ask:

When reality has only two buckets, what loss function appears?

For one example, Bernoulli says:

P(Y = y) = p^y(1-p)^{1-y}

This compact formula is a bucket picker.

If the truth is 1:

P(Y = 1) = p^1(1-p)^0 = p

If the truth is 0:

P(Y = 0) = p^0(1-p)^1 = 1-p

So the same formula asks:

How much probability did the model give to the bucket that actually happened?

Now let the model predict a probability for each row.

We write that prediction as:

\hat{p}^{(i)}

That means:

the model’s predicted probability for the 1 bucket on row $i$ .

So for row $i$ , the Bernoulli probability becomes:

(\hat{p}^{(i)})^{y^{(i)}} (1-\hat{p}^{(i)})^{1-y^{(i)}}

If the truth is 1, this becomes $\hat{p}^{(i)}$ .

If the truth is 0, this becomes $1-\hat{p}^{(i)}$ .

The formula wakes up the correct bucket and ignores the other one.

For the whole dataset, maximum likelihood multiplies those probabilities:

L = \prod_{i=1}^{n} (\hat{p}^{(i)})^{y^{(i)}} (1-\hat{p}^{(i)})^{1-y^{(i)}}

That product is the likelihood.

It asks:

How likely are all these observed yes/no outcomes under the model’s predicted probabilities?

A good model gives high probability to the bucket that happened on each row.

A bad model gives tiny probability to the truth.

Now we apply the negative log.

First:

-\log(L) = -\log \left[ \prod_{i=1}^{n} (\hat{p}^{(i)})^{y^{(i)}} (1-\hat{p}^{(i)})^{1-y^{(i)}} \right]

The log turns the product into a sum:

\begin{aligned} -\log(L) &= -\sum_{i=1}^{n} \log \left[ (\hat{p}^{(i)})^{y^{(i)}} (1-\hat{p}^{(i)})^{1-y^{(i)}} \right] \end{aligned}

Then the exponents come down:

\begin{aligned} -\log(L) &= -\sum_{i=1}^{n} \Big[ y^{(i)}\log(\hat{p}^{(i)}) \\ &\qquad\qquad + (1-y^{(i)})\log(1-\hat{p}^{(i)}) \Big] \end{aligned}

That is the negative log-likelihood for Bernoulli data.

Finally, average it across the dataset:

\begin{aligned} \text{BCE} &= -\frac{1}{n} \sum_{i=1}^{n} \Big[ y^{(i)}\log(\hat{p}^{(i)}) \\ &\qquad\qquad + (1-y^{(i)})\log(1-\hat{p}^{(i)}) \Big] \end{aligned}

And we get Binary Cross-Entropy.

That is the secret:

Binary cross-entropy is not an invented punishment.

It is what Bernoulli gives you after negative log-likelihood.

When reality has two buckets, BCE is the bill for putting too little probability on the bucket that actually happened.

Interpreting Binary Cross-Entropy

Here is the formula we finally care about:

\begin{aligned} \text{BCE} &= -\frac{1}{n} \sum_{i=1}^{n} \Big[ y^{(i)}\log(\hat{p}^{(i)}) \\ &\qquad\qquad + (1-y^{(i)})\log(1-\hat{p}^{(i)}) \Big] \end{aligned}

The formula looks heavier than the idea.

Binary cross-entropy asks:

How much probability did the model give to the bucket that actually happened?

If the truth is 1, the model should give a high probability to 1.

If the truth is 0, the model should give a high probability to 0.

BCE Punishes Confident Wrong Guesses

For one outcome, binary cross-entropy is:

\text{BCE} = -\left[ \underbrace{ y\log(\hat{p}) }_{\text{1 bucket}} + \underbrace{ (1-y)\log(1-\hat{p}) }_{\text{0 bucket}} \right]

Here, $\hat{p}$ means the model’s predicted probability for the $1$ bucket.

In our weather story:

1 = \text{Rain}

So:

\hat{p} = P(\text{Rain})

But BCE is not really obsessed with the label 1.

It is obsessed with the bucket that actually happened.

Call that number:

p_{\text{truth}}

If it rains, then:

p_{\text{truth}} = \hat{p}

If it does not rain, then:

p_{\text{truth}} = 1 - \hat{p}

So for one example, BCE becomes the same small question:

\text{loss} = -\log(p_{\text{truth}})

High probability on the truth means a small bill.

Tiny probability on the truth means a huge bill.

BCE is not just asking:

Was the model wrong?

It is asking:

How much probability did the model give to the truth?

Suppose it actually rains.

Then:

y = 1

So the $1$ bucket part wakes up, and BCE becomes:

\text{BCE} = -\log(\hat{p})

Now the loss depends on one thing:

How much probability did the model give to rain?

If the model said 90% chance of rain:

\text{BCE} = -\log(0.90) \approx 0.11

Small loss.

If the model said 10% chance of rain:

\text{BCE} = -\log(0.10) \approx 2.30

Bigger loss.

If the model said 1% chance of rain:

\text{BCE} = -\log(0.01) \approx 4.61

Very big loss.

Why?

Because giving rain only 1% means the model placed almost all its confidence in No rain.

It almost ruled out the thing that actually happened.

Tiny probability on the truth becomes huge loss after negative $\log$ .

High probability on the truth means small loss.

Tiny probability on the truth means large loss.

BCE is the loss for probability bets in a two-bucket world.

When BCE Gets Dangerous

BCE trusts the label.

That is powerful when labels are clean.

It is dangerous when labels lie.

Maybe a normal transaction gets marked as Fraud.

BCE does not know the bucket is wrong.

It treats the label as truth and punishes the model for disagreeing.

BCE can force the model to agree with bad labels.

BCE also needs care when one bucket is rare.

Suppose fraud happens only 1% of the time.

A lazy model can look good by pouring almost everything into Not fraud.

But that may miss the rare bucket we actually care about.

In that world, plain BCE may need help:

class weights,
better sampling,
better evaluation metrics.

Class weights are the bucket version of saying:

This rare bucket matters more than its size suggests.

One more practical note:

BCE expects the model output to behave like a probability.

If the model gives raw scores, those scores usually need to become probabilities first.

That is why binary classifiers often use a sigmoid before BCE.

Where Will You See BCE?

If MSE guesses the score, BCE guesses the coin flip.

Use BCE when the world only gives two endings:

Yes or No.
True or False.
Make or Miss.
Rain or No rain.

The model gives a probability.

Reality picks a bucket.

BCE grades the bet.

Logistic regression:
Despite the name, it is usually used for classification. It predicts a probability, and BCE grades that probability bet.

Neural networks:
When a binary classifier ends with one sigmoid output, it is making a Bernoulli-style bet. Spam or Not spam. Fraud or Not fraud.

Click, churn, and risk models:
Will the user click? Will the customer cancel? Is this transaction risky? The model gives a probability, and BCE sends the bill.

The Rare Bucket Trap

BCE needs care when one bucket is rare.

If fraud happens only 1% of the time, a lazy model can look good by pouring almost everything into Not fraud.

But that may miss the rare bucket we actually care about.

That is why BCE often needs class weights, better sampling, or better evaluation metrics.

Read more about this kind of rare-case problem in preprocessing:

The Winter Freeze

No scaling Trap

MSE and MAE care about physical units.

They measure the miss in dollars, minutes, inches, or pounds.

If one target is salary and another is height, the bigger scale can grab the microphone.

BCE does not have that same trap.

Because in BCE, the target is not a physical measurement.

It is a label:

0 \quad \text{or} \quad 1

No rain or Rain.
False or True.

BCE measures belief, not distance.

It only asks:

How much probability did the model assign to the truth?

You may still need to scale your inputs to keep training smooth.

But the target is already scaled by reality.

Before You Forget

Bernoulli distribution

The two-bucket rulebook: one outcome happens, the other does not.

Bernoulli distribution. The two-bucket rulebook: one outcome happens, the other does not.

Quiz

86% of people love quizzes after learning. Are you one of them?

★

Question 1 of 12 🏆 0 / 120 ⚡ Attempt 1 of 2

Question text