The Bias-Variance Tradeoff

Module 5 — Machine Learning for Time Series Forecasting
Created by Dr. Pedram Jahangiry | Enhanced with Claude
Website YouTube GitHub

Understanding Bias and Variance Through Repeated Sampling

The true relationship between x and y is a quadratic function (degree 2) plus random noise. You don't know this — you only observe noisy samples of 20 points. Each time you click "Draw Sample", a fresh sample of 20 points is generated and a polynomial of your chosen degree is fit to it. By drawing many samples, you can see how the fitted model behaves across repeated experiments.

Bias = how far the average prediction is from the truth (systematic error). Variance = how much predictions scatter across different samples (instability). A simple model (degree 0 or 1) is stable but consistently wrong — high bias. A complex model (degree 3 or 4) has extra parameters and changes more between samples — high variance.

Bias² — Systematic Error
Bias[f̂(x)] = E[f̂(x)] − f(x)
How far is the average prediction
across all samples from the truth?
Variance — Instability
Var[f̂(x)] = E[(f̂(x) − E[f̂(x)])²]
How much do individual predictions
vary around their average?
E[Test Error] = Bias² + Variance + σ²   (irreducible noise)
2
Draws: 0
Good Balance
Expected Test Error = Bias² + Variance + Irreducible Noise = —
Bias²
—
Draw samples to estimate
Variance
—
Draw samples to estimate
Irreducible Noise (σ²)
9.00
Cannot be reduced
Total Expected Error
—
Bias² + Variance + σ²
Fitted Models on Repeated Samples
Bias-Variance Tradeoff Curve