Model Evaluation: How to Trust a Model Before It Goes Live
Learn how validation, cross-validation, and a locked test set help choose a model without fooling yourself.
The Fraud
The bank has a problem.
Fraud is slipping through the cracks.
Some transactions are obvious.
Stolen cards.
Strange locations.
Impossible spending patterns.
Others are quieter.
They look normal until you compare them with thousands of other cases.
Meet Sherlock
To fight the fraud, the bank has already done the groundwork.
They already did data preprocessing:
- cleaned messy logs
- handled missing values
- shaped features
- split the data before Sherlock ever started training
Now the bank has built a fraud detective model.
They call it Sherlock.
Sherlock can now look at a transaction and make a call:
Fraud.
Not fraud.
Building Sherlock was only step one.
A trained model is not automatically a trusted model.
Now the team has to answer a harder question:
Is Sherlock actually any good?
That is what model evaluation is for.
Model evaluation tests a trained model on data it did not learn from, so we can estimate how it may behave on future cases.
For the bank, that means asking:
Does Sherlock catch fraud he has never seen before?
Does Sherlock avoid accusing too many honest customers?
Does Sherlock still look reliable when the case files change?
The goal is not to get a nice score on yesterday’s transactions.
The goal is to decide whether Sherlock deserves trust before real money is on the line.
The Data Setup
The bank starts with one original dataset: a large stack of historical transaction logs.
It includes:
- normal activity
- confirmed fraud
- false alarms
- messy edge cases
Before Sherlock can be evaluated, that one stack has to be split into separate roles.
Training Set
The training set is where Sherlock studies to find patterns:
- which clues usually point to fraud
- which clues usually point to normal behavior
- which combinations are suspicious
This is the team’s R&D lab.
Sherlock is allowed to learn, fail, and improve here.
Validation Set
The validation set is where the team checks Sherlock’s performance during development.
It helps the team choose between different Sherlock versions.
Maybe one Sherlock uses simple rules.
Maybe another Sherlock is more sensitive to suspicious behavior.
Maybe another Sherlock has different hyperparameter settings.
The team trains each version on the training set, then checks it on the validation set.
After seeing the validation results, the team can still refine Sherlock.
They can adjust settings.
They can try a different version.
They can train again.
The version that performs best on validation becomes the leading candidate.
Test Set
The test set is the sealed case file.
Once it is created, it stays locked away.
Sherlock does not learn from it.
The team does not tune against it.
Nobody uses it to pick the winner.
No training.
No tuning.
No peeking.
It stays untouched until the end.
The Golden Rule: Chain of Accountability
Think of the split like a chain of accountability.
Training is the team’s R&D lab.
Validation is the manager’s review set.
Testing is the executive’s final audit.
The team can learn in the R&D lab.
The manager can review progress and send the team back to improve Sherlock.
But the executive’s audit must stay sealed until the end.
How do we create the training, validation, and test splits fairly?
That depends on the problem:
- Simple random sampling shuffles the data and splits it into piles.
- Stratified sampling keeps rare classes represented in every split, which matters for fraud detection.
- Leakage checks make sure the split does not accidentally reveal tomorrow’s evidence today.
I covered those split decisions in data preprocessing
Estimating Generalization Performance
Earlier, the bank asked the question that starts model evaluation:
Is Sherlock actually any good?
Inside the R&D lab, the engineering team is busy trying to answer that question.
They can train Sherlock.
They can adjust Sherlock.
They can compare different Sherlock versions.
But the team is not trying to build a model that only looks good inside the lab.
They are trying to answer the harder question:
Will Sherlock work on cases outside the lab?
That is called generalization performance.
Generalization performance estimates how well a trained model will perform on new data it did not learn from.
For the bank, that means:
How well will Sherlock handle tomorrow’s transactions?
Before the executive’s final audit, the team needs a reliable way to estimate that answer.
That is why we use validation data.
Validation data is not used to train Sherlock directly.
It is the manager’s review set: a stack of cases kept outside the R&D lab so the team can see how Sherlock performs beyond the cases he studied.
The team can use it to check performance, compare Sherlock versions, and tune model settings.
But that freedom is exactly why validation is not the final test.
Once the team uses validation results to improve Sherlock, the validation set becomes part of development.
To run that reality check, the team uses two standard techniques:
- the holdout method
- cross-validation
The Holdout Method
The team takes the available development data and splits it into two piles:
- training set: Sherlock studies these cases to learn the rules
- validation set: Sherlock is checked on these cases to see if the rules hold up
It is the simplest way to estimate generalization performance.
And it gives the manager a review signal during development.
But the holdout method has a weakness.
It puts too much faith in one roll of the dice.
What if the validation set gets mostly obvious fraud cases?
Sherlock may score beautifully.
The team may think he is a genius.
But maybe the review set was just lucky.
What if the validation set gets the strangest edge cases?
Sherlock may look terrible.
The team may throw away a perfectly good model.
But maybe the review set was just unlucky.
With the holdout method, Sherlock’s performance estimate can be highly sensitive to how the data was split.
The team is not only judging Sherlock.
They are also trusting the luck of one partition.
To reduce that luck, the team needs a more robust review process.
That is where k-fold cross-validation enters.
K-Fold Cross-Validation
If the holdout method is one roll of the dice, k-fold cross-validation is a repeated review.
The team splits the development data into k equal parts.
Each part is called a fold.
Then the review process rotates.
In the first round:
- fold 1 becomes the validation fold
- the remaining folds become the training folds
- Sherlock trains on the training folds
- Sherlock is checked on fold 1
In the next round, fold 2 becomes the validation fold.
Sherlock trains again on the remaining folds.
Sherlock is checked again.
This continues until every fold has taken one turn as the manager’s review set.
At the end, the team has k validation scores.
The final cross-validation estimate is the average of those k scores.
One review can be blamed on luck.
An average across k reviews is harder to dismiss.
So the team is no longer asking:
Did Sherlock get lucky on one split?
They are asking:
Does Sherlock keep holding up as the review set changes?
Insight: Cross-validation does not erase luck. It makes luck harder to dominate the decision.
What Should k Be?
The value of k controls how many review rounds Sherlock has to survive.
A small k means fewer rounds.
That is faster, but each estimate can still depend on chunky splits.
A large k means more rounds.
That uses the data more thoroughly, but it costs more compute because Sherlock has to train again and again.
In practice, k = 10 is a common choice.
Ron Kohavi’s 1995 study found that ten-fold stratified cross-validation often gave a strong tradeoff for model selection on real-world datasets. See A Study of Cross-Validation and Bootstrap for Accuracy Estimation and Model Selection.
So when people say “k-fold cross-validation,” a common practical default is:
k = 10
Not because 10 is magic.
Because it is often a good balance between reliable review and reasonable cost.
Stratified K-Fold
Fraud cases are rare.
If the folds are created carelessly, one fold may get too few fraud cases.
That makes the review noisy.
Stratified k-fold keeps the fraud-to-normal ratio similar in every fold.
It is the cross-validation version of stratified sampling.
So each review round gives Sherlock a fairer mix of cases.
Scoring Sherlock Performance
Cross-validation gives the team a review process.
But each review still needs a score.
The team has to decide what “good” means before comparing Sherlock versions.
For fraud detection, the usual metrics tell different stories:
- Accuracy asks how often Sherlock is right overall
- Precision asks how often Sherlock is right when he says fraud
- Recall asks how many real fraud cases Sherlock catches
- F1 balances precision and recall
- MCC, or Matthews correlation coefficient, gives one balanced score when fraud is extremely rare. If Sherlock lazily calls everything “normal,” MCC exposes the trick.
We covered the full confusion matrix story in confusion matrix and metrics.
Here, the point is simple:
The scoring rule decides what kind of Sherlock the team will favor.
If they score by recall, they reward catching more fraud.
If they score by precision, they reward fewer false alarms.
If they score by F1 or MCC, they reward balance.
Only after choosing the scoring rule can the team compare Sherlock versions fairly.
Diagnosing and Tuning Sherlock
Once the team has a score, they may find out Sherlock is failing.
But a score only tells the team what happened.
It does not explain why it happened.
Maybe Sherlock is too simple.
Maybe Sherlock is memorizing old case files.
Maybe Sherlock needs more data.
Maybe Sherlock needs a different setting.
That is where diagnosis begins.
The team uses two visual tools:
- validation curves show what happens when the team changes one model setting
- learning curves show what happens when Sherlock gets more training data
These curves help the team decide whether Sherlock is underfitting, overfitting, or running out of useful signal.
We covered that full debugging loop in bias-variance tradeoff.
A score tells the team what happened.
The curves help explain why it happened.
Tuning is what the team tries next.
They try different hyperparameter settings.
One setting may make Sherlock simpler.
Another may make Sherlock more flexible.
Another may regularize him more strongly.
Each version is trained and scored using the same validation process.
With k-fold cross-validation, each setting gets reviewed across multiple folds.
The version with the strongest cross-validation score becomes the leading candidate.
But the test set is still locked.
The team can tune with validation.
They cannot tune with the executive’s final audit.
The Final Audit
Sherlock has survived the R&D lab.
The team picked a scoring rule.
They reviewed Sherlock with validation.
They used cross-validation to reduce split luck.
They diagnosed weak spots.
They tuned the hyperparameters.
Now the settings are locked.
The manager’s review is over.
It is time for the executive audit.
The sealed case file is opened, and Sherlock runs on the test set.
The final score goes to the executive as the go / no-go evidence.
Maybe that score is precision.
Maybe it is recall.
Maybe it is F1.
Maybe it is MCC.
This is the final estimate of Sherlock’s generalization performance.
Not a guarantee.
An estimate.
But if the test set stayed sealed until the end, it is the cleanest evidence the bank has.
If Sherlock Fails the Final Audit
What if the test score is bad?
The temptation is obvious.
Adjust Sherlock.
Try another setting.
Run the test again.
Do not do it.
If the team uses the test result to improve Sherlock, the test set stops being the executive’s final audit.
It becomes another manager’s review set.
The seal is broken.
The next score may look better, but it is no longer clean evidence.
If Sherlock fails the final audit, the honest answer is simple:
Sherlock is not ready.
The team may need better data, better features, a different model, or a new development cycle.
But they cannot tune their way to a trustworthy test score.
Model evaluation is not about helping Sherlock pass the audit.
It is about keeping the audit honest.
One Last Production Rule
When Sherlock goes live, the raw transaction data still needs the same preparation used during development.
Missing-value rules.
Category encodings.
Scaling statistics.
Feature transformations.
Those choices are not throwaway setup.
They are part of the model pipeline.
For example, if transaction amounts were scaled using the training set:
- preserve the training-set mean and standard deviation
- reuse them on validation data
- reuse them on test data
- reuse them in production
Sherlock should not recalculate those statistics from tomorrow’s transactions.
That would quietly change the evidence.
We covered this rule in data preprocessing.
The model and its preprocessing pipeline must ship together.
If Sherlock tries to process tomorrow’s clues using tomorrow’s averages, his logic will break.
He must always judge the future using the baseline he learned in the lab.
Rules from the Case File
Generalization performance
Not Sherlock memorizing yesterday’s fraud. It is the estimate of whether he can handle tomorrow’s transactions.
Generalization performance. Not Sherlock memorizing yesterday’s fraud. It is the estimate of whether he can handle tomorrow’s transactions.
Quiz
86% of people love quizzes after learning. Are you one of them?
Question text
Quiz complete