Essays

Calibration is the forecast quality nobody checks

A reliability diagram: three model curves around the diagonal

A model that says “70% chance of rain” should be right about seven times in ten when it says so. That property, calibration, is separate from accuracy: a model can rank cases perfectly and still be wildly over-confident.1

A quick check

Bin your predictions by stated probability, then compare each bin’s average prediction with the observed frequency. Plotted, a calibrated model hugs the diagonal.

A reliability diagram: three model curves around the diagonal
Predicted probability (x) against observed frequency (y) for three models. Blue sits above the diagonal (under-confident), orange below it (over-confident); teal, recalibrated, follows it.

Why it matters

  • Decisions use the probability, not the ranking: thresholds, budgets, staffing.
  • Over-confident models fail silently until an expensive miss.
  • Calibration drifts faster than accuracy when the population shifts.2
ModelAUCBrierECE
Baseline logistic0.7810.1620.041
Gradient boosted0.8240.1490.067
Boosted + isotonic0.8230.1410.012
Ranking barely moves after recalibration; the expected calibration error (ECE) drops by four fifths.

Recalibration is cheap: isotonic regression or Platt scaling on a held-out set takes a few lines and rarely costs accuracy. Checking it should be as routine as checking AUC.

  1. The distinction goes back to the verification of weather forecasts, where reliability diagrams have been standard for decades. ↩
  2. We saw this in a churn model last year: AUC moved by 0.01 while ECE tripled over two quarters. ↩