A model that says “70% chance of rain” should be right about seven times in ten when it says so. That property, calibration, is separate from accuracy: a model can rank cases perfectly and still be wildly over-confident.1
A quick check
Bin your predictions by stated probability, then compare each bin’s average prediction with the observed frequency. Plotted, a calibrated model hugs the diagonal.

Why it matters
- Decisions use the probability, not the ranking: thresholds, budgets, staffing.
- Over-confident models fail silently until an expensive miss.
- Calibration drifts faster than accuracy when the population shifts.2
| Model | AUC | Brier | ECE |
|---|---|---|---|
| Baseline logistic | 0.781 | 0.162 | 0.041 |
| Gradient boosted | 0.824 | 0.149 | 0.067 |
| Boosted + isotonic | 0.823 | 0.141 | 0.012 |
Recalibration is cheap: isotonic regression or Platt scaling on a held-out set takes a few lines and rarely costs accuracy. Checking it should be as routine as checking AUC.