Calibration and the Brier score answer different questions, so a good number on one does not vouch for the other. Calibration asks whether your 70% calls come true about 70% of the time, band by band. The Brier score grades every resolved yes-or-no call in one number, and it rewards two things at once: probabilities that match how often things happen, and probabilities that separate the cases that happen from the ones that do not. A low score can sit beside an overconfident band. A forecaster who gives every case the same probability can land on the calibration line and still score worse.
In the hypothetical quarter below, the differentiated forecasts score 0.205. A constant 70% forecast on the same outcomes scores 0.210 and matches the overall hit rate exactly. The lower score belongs to the forecaster whose 90% calls came true eight times in ten. Neither reading is wrong; each misses something the other catches.
What calibration means explains the wider idea. This piece starts after the yes-or-no outcomes are known: what does each reading show, and what does it leave out?
What does calibration show that the Brier score does not?
Calibration checks whether stated probabilities match observed frequencies, band by band. If calls made at 70% come true about 70% of the time across enough comparable cases, that band is close to calibrated. If 90% calls come true 80% of the time, the 90% band sits below the diagonal on a calibration curve: the stated confidence ran high in that record.
The Brier score compresses all resolved binary calls into one average squared error. It rewards probabilities close to the outcomes. It also rewards separation: giving higher probabilities to the cases that turn out to happen more often and lower ones to the cases that happen less often. A single point on the calibration line cannot show that. Chapter 8 of The Calibrated Mind draws the line plainly: the Brier score “grades your binary forecasts overall; it does not, by itself, isolate calibration.”
| Reading | The question it answers | What to keep beside it |
|---|---|---|
| Calibration curve or band table | Did each confidence band come true about as often as stated? | Each band’s mean stated probability, hit rate and count. |
| Brier average | How far were the probabilities from the binary outcomes overall? | The number and mix of resolved calls, plus the score of a prechosen reference on those same calls. |
How do you calculate a Brier score?
For one resolved yes-or-no call, write the forecast as a probability from 0 to 1 and the outcome as 1 if it happened or 0 if it did not. Square the difference: (probability - outcome)^2. Average those squared differences over the resolved binary rows. Lower is better on the same set of questions. A perfect row scores 0; a confidently wrong 100% call scores 1.
Chapter 8 gives a simple check. An 80% call that comes true scores (0.8 - 1)^2 = 0.04. If it fails, the row scores (0.8 - 0)^2 = 0.64, sixteen times as much. If your spreadsheet holds confidence as a whole percentage, use =POWER(confidence/100 - outcome, 2) on each resolved binary row and leave unresolved rows blank.
Saying 50% on every binary question scores 0.25 on every row, whatever happens. The book calls that an orientation mark, not a population average, and it is not a passing grade either. If the event usually happens far more or far less than half the time, a relevant historical rate is a fairer comparison than a coin flip.
Worked example: a hypothetical founder quarter
Every call, outcome and number in this example is hypothetical. A founder logs twenty comparable yes-or-no forecasts: will each qualified proposal close by its stated deadline? Each probability and resolution rule is written down before the deadline. Ten proposals receive 90%; ten receive 60%.
| Forecast band | Resolved calls | Closed by deadline | Hit rate | Squared-error sum | Band Brier average |
|---|---|---|---|---|---|
| 90% | 10 | 8 | 80% | 8 × 0.01 + 2 × 0.81 = 1.70 | 0.170 |
| 60% | 10 | 6 | 60% | 6 × 0.16 + 4 × 0.36 = 2.40 | 0.240 |
Add the row errors: 1.70 + 2.40 = 4.10. Divide by twenty resolved calls. The overall Brier score is 0.205. The 90% group closed more often than the 60% group, eight in ten against six in ten, so the record shows separation. But the 90% band landed at 80%, so its stated confidence ran high. The 60% band landed at 60%.
The score registered that overstatement too. Rerun the same outcomes with 80% on the first ten: 8 × 0.04 + 2 × 0.64 = 1.60, and (1.60 + 2.40) / 20 = 0.200. The excess confidence cost 0.005. That is a hindsight check, not a correction to adopt: ten calls are far too few to justify rescaling a band.
Now suppose earlier comparable proposals had led the founder to fix a 70% reference forecast before this quarter. Apply 70% to each of the same twenty proposals. Fourteen closed and six did not, so its Brier score is (14 × 0.09 + 6 × 0.49) / 20 = 0.210. Its single 70% band matches the quarter’s 14/20 hit rate exactly. Yet it scores slightly worse than 0.205, because the founder’s case-by-case calls separated the two groups and the flat reference could not. A match in one small sample does not show that the reference is reliably calibrated, and a 0.005 gap does not establish skill.
Line the three numbers up: 0.200 with separation and a top band that matches its hit rate, 0.205 with separation and an overstated top band, 0.210 with no separation at all. The Brier total moves with both qualities, so on its own it cannot tell you which one moved it. The band table can. That is why the two belong on the same page.
What should the Brier score be compared with?
Choose a reference forecast before the outcomes arrive, ideally a historical rate for genuinely similar questions, and score it on the same resolved calls. If you pick the 70% reference only after counting fourteen successes, you have described the result, not built a fair benchmark. A historical rate can also go stale when the underlying cases change, so keep the cohort and the rule for choosing the reference on the record.
Chapter 8 cites a comparison from a study of reliability diagrams by Dimitriadis, Gneiting and Jordan. It covers 86 daily probability-of-precipitation forecasts, each issued 24 hours ahead, for one location over one season: Niamey, Niger, July to September 2016. The world-leading 52-member ensemble of the European Centre for Medium-Range Weather Forecasts (ENS) scored 0.266. A reference forecast built from climatology (EPC) scored 0.234 and beat it. A statistically post-processed version of the same ensemble (EMOS) scored 0.232, and a purely data-driven statistical forecast the study calls Logistic scored best, at 0.206. The paper’s score decomposition, with the same uncertainty term (0.244) for all four, shows why the ensemble lost to the climatology reference: it separated wet days from dry ones better (discrimination 0.044 against 0.032), but its miscalibration penalty was three times as large (0.066 against 0.022). One Brier total shows who won. Calibration and separation show where the error came from. Precipitation forecasts for one location set no target for a founder’s proposals, which is why your own journal needs a reference from your own domain.
For your own journal, a lower score than a prechosen reference is a finding about those calls. To judge whether your forecasting practice is improving, compare periods with similar questions and keep gathering outcomes. A quarter of easier cases could lower the average without any improvement in your judgment.
How much can a thin calibration curve tell you?
Write 8 of 10 beside 80%. Ten calls in a band move sharply when one outcome changes. Chapter 3 of The Calibrated Mind sets the working rule: below about twenty comparable resolved forecasts, read the numbers as a record, not a verdict. Twenty is no guarantee either; the question mix and the way forecasts reached the journal still matter.
Set bands before reviewing outcomes and keep their boundaries fixed across reviews: 50–60%, 60–70%, 70–80%, 80–90% and 90–99%, with boundary values counting toward the higher band. For each band, compare the hit rate with the mean probability actually stated, not the band midpoint. A 70–80% band whose calls average 74% belongs against 74%. The reliability-diagram study shows that a traditional binned diagram can change shape with the choice of bins, and that sparsely populated bins carry large estimation uncertainty. Fixed bands keep your quarterly reviews comparable; they do not remove the noise in a band that holds a handful of calls.
Also check what entered the journal. If you recorded only the calls that felt safe to state, neither reading describes your decisions. Do not reconstruct a probability from memory after an outcome. Keep unresolved forecasts dated and out of both calculations until they resolve.
Do numeric ranges go into the Brier average?
No. “The launch reaches at least 100 paying accounts by Friday” is binary if you fixed what counts as a paying account and when to check. “The launch reaches 80 to 140 paying accounts by Friday” is a numeric range. Judge a range by coverage, the share of outcomes that land inside its bounds, and, when you want a score that also charges for width and for distance outside the bounds, by an interval score. Chapter 8 keeps these results apart from the binary Brier rows.
A Brier score is a squared error on a probability, bounded between 0 and 1. An interval score is in the units you forecast, such as accounts, and has no upper bound. Average the two and the result has no meaningful unit. Recasting every range as “inside or outside” also throws away how wide the interval was. The 90% test is a range exercise; keep its results in their own column.
What belongs in the review after the calculations?
Keep four things together: the Brier average for resolved binary calls, the band table with counts, the score of the prechosen reference on the same calls, and the frozen rationales with process notes. Each has a job. The score gives the overall error. The bands locate possible overstatement or understatement. The reference asks whether your judgment added anything on this set. The notes help investigate a pattern of misses.
Do not force a cause from one failed 70% call. A calibrated 70% forecast is supposed to fail about three times in ten. If several misses share an assumption, check it against what was known when you made the calls. If you find no broken assumption or missed signal, write “none found.” The five-line decision journal shows how to preserve your reasoning before a consequential choice; the prediction journal adds the probabilities and outcomes this review needs.
Try this today (20 minutes)
Take ten resolved yes-or-no calls from dated notes, with probabilities you wrote before the outcomes were known. If you do not have them, write ten testable calls now, each with a probability, a resolution date and a yes-or-no rule. Save the original numbers and run this test only when the outcomes arrive. A confidence number remembered after the fact cannot be scored as a forecast.
- Score the binary rows. Put 1 beside each event that happened and 0 beside each that did not. Calculate
=POWER(confidence/100 - outcome, 2)on each resolved row, then average the ten row scores. - Read the bands. Group the calls into the predeclared bands from chapter 8: 50–60%, 60–70%, 70–80%, 80–90% and 90–99%, counting boundary values toward the higher band. For each nonempty band, write the count, the mean stated confidence and the share that happened. Keep sparse bands visible and label them thin; do not turn ten calls into a diagnosis.
- Add a fair reference if you have one. Use a relevant historical rate that existed before these calls and score it on the same ten outcomes. If none was recorded, write “no prechosen reference” rather than choosing a flattering one now.
- Write the surprise. Put the Brier average beside the band readings. Which number surprised you? Open the original rationale for any repeated kind of miss, and record one question to test with the next set of comparable calls.
This first pass tells you what the two readings said about a small record. Keep the original rows and repeat the comparison as the bands fill.
Cite this:Calibration vs Brier score: what each measure misses.Len P. van der Hof. https://lenvanderhof.com/en/blog/calibrate-vs-brier/ ·