Wondering about training data vs test data? Training data teaches your model patterns; test data is held back to check it on examples it has never seen. A validation set sits between them for tuning.
If you are new to the wider topic, start with machine learning basics, then come back here for the details of splitting data.
| Item | Value |
|---|---|
| Independent data sets in best practice | Three (training, validation, test) |
| Common starting split (training / validation / test) | 80% / 10% / 10% |
| Example test accuracy in a cats-and-dogs demo | 87.5% |
| Test set performance checked before choosing a model | Never |
These figures come from a published guide to splitting ML data and an interactive explainer on train, test and validation sets. They can change, so check the sources before you rely on them.
The three data sets at a glance
In most supervised learning projects, best practice is to split your data into three independent sets: training, validation and test, as the MLU-Explain walkthrough describes. Each set answers a different question about your model.
- Training set: the data you feed the model so it can learn underlying patterns and relationships.
- Validation set: the data you use to understand performance across different model types and hyperparameter choices.
- Test set: the data you use to approximate the model’s unbiased accuracy in the wild.
Keeping them separate is the whole point. If one set had to answer all three questions, you could not tell a model that learned from one that merely memorised.
Training data vs test data: the core difference
The training set is where learning happens. The model uses it to find patterns, and the MLU-Explain walkthrough says it should be as representative as possible of the population you want to model. It also warns that any bias at this stage may be propagated downstream during inference.
The test set plays the opposite role. According to Kili Technology’s overview, it mirrors real-world data the model has never seen before, and no hyperparameters are adjusted during this final evaluation.
The practical difference shows up in what each set is allowed to change. A deep learning course lesson summarises it: weights are updated from the training set only, while the validation and test sets leave them alone.
| Data set | Updates model weights? | Main job |
|---|---|---|
| Training | Yes | Fits the model so it can learn patterns |
| Validation | No | Checks generalisation and guides tuning during development |
| Test | No | Gives a final check before deploying to production |
This comparison is based on the dataset summary table in a deep learning course lesson on train, test and validation sets, and the source may change, so check it before you rely on the details.
Where validation data fits in
Many people compare only two sets, but the validation set is what keeps the test set clean. It acts as an independent, unbiased dataset for comparing different algorithms trained on your training set.
Without it, you would tune on the same data you trained on. The MLU-Explain authors note that when the same dataset is used for both training and tuning, the model will overfit and won’t generalize well. If you want the background, read what is overfitting in machine learning for a fuller picture.
Comparing the two sets of results gives you a quick diagnostic. If results on the training data are very good but the validation results lag behind, the model is overfitting. If they are similarly good, you can be more confident it generalises.
Validation data can be overused too
Validation data is not immune to overfitting. The Daily Dose of Data Science post warns that repeatedly tuning on validation performance over many iterations risks indirect overfitting to the validation set.
Not every project needs a validation set. Kili’s overview says some experts consider models with no hyperparameters or tuning options to be fine without one, yet in most practical applications validation sets play a crucial role.
How to split and use your data
A common starting point is 80% training, 10% validation and 10% test, adjusted for dataset size and use case, according to Kili Technology’s guide. Treat it as a default to question, not a rule.
A workflow described by the Daily Dose of Data Science post keeps the roles apart:
- Set the test data aside and behave as if it does not exist.
- Analyse, transform and fit the model using the training set only.
- Evaluate on the validation set, improve the model, and repeat until you are satisfied.
- Retrain the chosen model and hyperparameters on the combined training and validation data.
- Evaluate once on the test data for a final, unbiased estimate.
When data is scarce: cross-validation
With little data, a single validation split can be noisy. The same post suggests k-fold cross-validation, where you split the data into k folds, commonly k=5 or k=10, and rotate which fold is used for validation.
The trade-off is cost: cross-validation is computationally more expensive because you train k models instead of one. It is still highly recommended when you do not have much data.
When data has a time order
If your data has a time component, random splits cause data leakage, since you might train on future data and validate on past data. The Daily Dose of Data Science post recommends chronological splits instead.
Test set mistakes that inflate your results
The cardinal rule comes from the MLU-Explain authors: never, under any circumstance, look at the test set’s performance before selecting a model. Peeking is a form of overfitting and will likely lead to unreliable performance expectations in production.
That rule covers more than model choice. The Daily Dose of Data Science post lists what a test set should not be used for:
- Model selection.
- Hyperparameter tuning.
- Feature engineering decisions.
- Any decision that influences the model.
A separate explainer covers what is feature engineering and why those choices count as modelling decisions that belong on the training side.
If validation and test results disagree
Do not panic. In the MLU-Explain demo, a simpler model scored higher on the test set than the model the validation set had chosen, and the authors say this might happen, yet it is not a bad thing. Test performance is a metric to assess future performance, not a number to optimise.
The same demo ends with an estimate that the model can tell cats from dogs with 87.5% accuracy on unseen data, as reported by MLU-Explain. That is what a test set is for: one honest number, read once.
If the model really does fail the test
If a model underperforms on the test set, the test set has been exposed, so any improvement made with knowledge of its results biases later evaluation. Likely causes to investigate include:
- Distribution shift between training and test data.
- Data leakage somewhere in your pipeline.
- A test set that is not representative.
Teams can also keep a separate holdout that is evaluated only at major milestones, such as before deployment.
Frequently asked questions
Do I always need a validation set?
Show answer
Not always. Kili Technology’s overview says some experts think models without hyperparameters or tuning options can skip it, but in most practical applications a validation set matters.
Can I tune my model on the test set?
Show answer
No. The test set is for a final, unbiased estimate of real-world performance, and model selection and hyperparameter tuning belong on the training and validation side.
What if I only have two subsets?
Show answer
Some publications call the validation set a test set when there are only two subsets, as Kili Technology notes. Check how a source uses the words before comparing its advice with other guidance.
Why is my test accuracy different from my validation accuracy?
Show answer
A gap can simply happen. The MLU-Explain authors say validation performance not exactly matching test performance is not a bad thing, though a large gap is worth checking for leakage or an unrepresentative test set.
