When we don't have enough data for a separate validation set, K-Fold CV solves the problem by reusing data. The training data is split into K equal-sized folds. In each iteration, one fold is held out as the validation set and the remaining K−1 folds are used for training. The final performance is the average across all K iterations.
Standard K-Fold may create folds where class proportions are unbalanced — one fold might have mostly one class. Stratified K-Fold ensures each fold has approximately the same percentage of samples from each class as the complete dataset. This is especially important for imbalanced classification problems.
LOOCV is K-Fold taken to its extreme: K = N (the number of samples). Each iteration uses a single sample as the validation set and all remaining N−1 samples for training.
Why is it almost unbiased? Each training set contains N−1 out of N samples — nearly the entire dataset. So the model trained in each fold is almost identical to a model trained on all the data. The performance estimate therefore closely approximates the true generalization error. It's "almost" rather than exactly unbiased because each fold is still missing one sample, introducing a tiny amount of bias.
The downside: it is computationally expensive — the model must be trained N times, and the estimate can have high variance since each validation set is just a single observation.
The simulation below demonstrates LOOCV on a classification task. In each iteration, the single held-out sample is predicted as either correct (✓) or incorrect (✗). The final LOOCV accuracy is the fraction of samples correctly classified.
For regression tasks, the logic is the same — but instead of correct/incorrect, each iteration produces a prediction error (e.g., squared error), and the final LOOCV score is the average error across all N iterations.
Each cross-validation strategy has trade-offs between computational cost, data usage, and reliability of the performance estimate. The right choice depends on your dataset size, class balance, and available compute.
| Aspect | K-Fold | Stratified K-Fold | LOOCV |
|---|