What is a calibration session, and what should it produce?

A calibration session should end with an output you can inspect, not just agreement: final ratings in HR, or probabilities in forecasting that you can check against what happens later. Both meetings put people’s judgments side by side, but they aim at different things. HR calibration aims to make ratings consistent across managers. Forecasting calibration, the meaning used in The Calibrated Mind, asks whether stated confidence holds up against what happens.

In either session, “we aligned” tells you too little. Keep each person’s first judgment, any revision, and the reason for it. Agreement can end a discussion without showing that anyone is right.

Which calibration session do you mean?

	HR calibration session	Forecasting calibration session
Who meets	Managers who supervise similar work, with a facilitator	A team facing a decision
What each person brings	Draft ratings for their own staff	A private probability for a defined future event
What comes out	Final ratings	A forecast register
What it aims for	A 3 means the same thing across teams	Over many forecasts, things called 70% likely happen about 70% of the time
What can go wrong	Ratings bunch toward the middle	Estimates drift toward the most confident voice

HR calibration produces final ratings

In performance management, managers put draft ratings side by side, discuss them, and adjust them. A 2014 practitioner article from SHRM, the US association for HR professionals, describes five steps:

1. Managers draft appraisals with proposed ratings.
2. Managers who supervise similar groups meet and display names and ratings for review.
3. Participants discuss the proposed rating for every employee.
4. Participants adjust the ratings.
5. Managers prepare the final appraisals.

The consultant Dick Grote, quoted in that article, describes the aim as neutralizing the effect of “tough graders” and “easy graders”. An aim is not evidence that the process achieves it.

The consistency problem is real. Salgado and Moscoso’s 2019 meta-analysis covered 224 independent samples. For overall job performance, their best estimates of observed interrater reliability were 0.61 for ratings collected for research and 0.45 for ratings collected for administrative use. In The Calibrated Mind, administrative use means raises, promotions, and terminations. (At 1.0, two supervisors’ ratings of the same people would line up perfectly; at 0, they would have no relation.) These figures measure agreement between supervisors. They do not measure whether the ratings are accurate, or what a calibration session changes.

One field study of a large multinational (Demeré, Sedatole and Woods, 2019) found that its calibration committees adjusted about a quarter of ratings. Downward changes were more frequent and larger than upward ones. Ratings became more consistent across supervisors and less lenient, but also bunched more toward the middle. The abstract of a 2026 review by Bol, Braga De Aguiar and Lill describes calibration as widely adopted, with its effectiveness and unintended consequences still debated.

None of the HR evidence cited here shows that calibration makes ratings more accurate. Consistent is not the same as accurate: a committee can bring every manager onto one standard and still misjudge the same people. As The Calibrated Mind warns, a room can converge by anchoring and become jointly wrong.

Four habits from the forecasting side fit an HR session. They are this page’s application of the book’s method to HR meetings, not a tested HR intervention, and this page is not HR, legal, or employment advice.

1. Write before you meet. Submit each draft rating with one line of evidence. Keep the first rating in the file.
2. Give every change a reason. Record which evidence moved the rating.
3. Ask for evidence, not reputation. Grote again: “The facilitator’s role is to make sure that ratings managers have data, not just favorable or unfavorable views.”
4. Run a noise audit once a year. Using the book’s method, give several managers the same anonymized case separately and compare their ratings. A wide spread shows disagreement about the same material. It does not tell you which rating is correct.

In forecasting, the same private write-down does more: it starts a record you can check against what happens.

What should a forecasting calibration session produce?

A forecast register. For each forecast, it records a question about one defined future event, its deadline, each person’s probability and reasons written before discussion, and a resolution rule: the evidence, agreed in advance, that decides whether the event happened. Revisions go beside the original estimate, never over it. The template is further down.

“Will the launch go well?” leaves the outcome open to argument about what “well” meant. “Will the customer sign the acceptance form by Friday 17:00?” gives you an event and a deadline. Add the time zone and state which evidence counts, so that someone who missed the meeting can resolve the forecast without negotiating.

Forecast calibration is the match between stated probabilities and observed frequencies: across many forecasts, events called 70% likely should happen about 70% of the time. A single success cannot establish that match, and neither can memory. You need a written record checked against outcomes. What calibration means explains the idea in more depth.

Why write the number down before anyone speaks?

An opening estimate can become an anchor. Imagine a founder opens the meeting with “this pilot is almost certain to work.” Colleagues may soften their doubts before they state a number. If you record only the estimates that follow, you capture where the conversation ended, not where each person started.

Unevenly shared information is a second problem. Lu, Yuan and McLeod’s 2012 meta-analysis covered 65 “hidden profile” studies (101 effects, 3,189 groups), in which members held some facts that others lacked. Groups mentioned shared information far more than unique information, a gap of about two standard deviations. Groups that had to pool scattered facts were eight times less likely to find the solution than groups given the full information.

That finding is a reason to collect each person’s evidence before discussion. It does not show that this page’s meeting format improves forecasts. Writing first preserves each initial judgment and gives information that only one person holds a place in the record.

Keep those first numbers after the discussion. If the plan needs one working number, record who chose it and how, beside the individual estimates.

A worked example (hypothetical)

In this hypothetical example, a small team is deciding whether to expand a customer pilot. Before the meeting, the facilitator writes one question: will the customer complete the agreed handover exercise and send written acceptance by Friday at 17:00, Amsterdam time?

The resolution rule comes next, before any numbers. It needs a completed exercise record and written acceptance from the named customer contact, or from a deputy authorized under the agreed process. A friendly call does not count. If either requirement misses the deadline, the answer is no.

Three people send their numbers privately: 40%, 65% and 80%. The facilitator reveals them together and asks each person for their strongest reason.

The person at 40% knows the customer contact is away on Friday. The person at 80% has heard that a deputy may sign. Now the useful question is clear: does the agreed process already authorize that deputy? Someone is assigned to check. That is a question of fact under the existing rule, not a chance to rewrite the rule once the result is in. Any revised probability is added with a time and a reason. The original 40%, 65% and 80% stay.

The team may still decide to prepare the expansion. Record that decision separately, for example in a five-line decision journal. The forecast states what the team expects; the decision states what it will do.

Copy this register

Field	What to write
Question	One event, one deadline, with time zone
First estimate	Name, probability, and time, written before discussion
Reasons	Main evidence, main uncertainty, what would change the number
Resolution rule	Which evidence counts; what happens if it is missing or disputed
Revisions	New number, time, and reason; the original stays
Owner	Who fetches the outcome; who settles disputes
Review	Date to check; later, the outcome and any action that followed

Keep the event deadline separate from the review date: moving Monday’s meeting does not move Friday’s deadline. If evidence is missing, the rule agreed beforehand decides what happens. “We cannot get the evidence yet” is not the same as “it did not happen”, and nobody gets to reinterpret an inconvenient result afterwards.

Where does this come from? CALIBRATE in one minute

CALIBRATE is Len P. van der Hof’s nine-step framework for holding judgments at the strength their evidence supports. It comes from The Calibrated Mind, available in English and Dutch. The session on this page is an editorial application of the book’s methods, not a validated instrument. You can use the register without the book.

1. Calibrate confidence: treat confidence as a measurement you can check, not a feeling.
2. Assess assumptions: name the hidden premises a judgment rests on.
3. Lean on probabilities: replace words like “likely” with numbers.
4. Investigate alternatives: test your view against the strongest competing explanation.
5. Build updating rules: decide in advance how much evidence should move you.
6. Record and review: keep a prediction record and check it against outcomes.
7. Act with appropriate uncertainty: act at a threshold instead of waiting for certainty.
8. Teach and extend: explain your reasoning to others so they can find the weak spots.
9. Evolve your model: notice your drift over time and correct it.

A forecasting session applies several of these steps: stated probabilities (1 and 3), updating rules set before results arrive (5), a preserved record (6), and reasoning others can examine (8).

The book describes a two-person version, the calibration conversation, and points larger groups to formal methods such as the IDEA protocol described by Hemming and others (2018): Investigate, Discuss, Estimate, Aggregate. In the book’s account, participants estimate alone, discuss, estimate alone again, and then combine the estimates by a rule rather than by consensus. The CALIBRATE framework page sets out the full method.

Signs the record is not yet useful

• The minutes say “confident”, “likely” and “aligned”, but contain no numbers and no resolution rules.
• The revised estimates overwrote the first ones, so nobody can see who updated or why.
• There is a review date but nobody named to fetch the evidence.
• One successful outcome is treated as proof that someone is well calibrated. Only many resolved forecasts can show that.
• In HR: the ratings now match across teams, and nobody can say what evidence moved them.

Set this up this week (15 minutes)

1. Pick one decision due this month and one assumption behind it that will be settled within weeks. (3 minutes)
2. Write the assumption as a forecast question with a deadline and time zone, and write the resolution rule underneath. (5 minutes)
3. Write down your own probability and reason privately. Send only the question and the rule to two colleagues, and ask each for a private number and one reason before any discussion. (2 minutes)
4. Create the register, set a review date, and name who will collect the outcome evidence. (5 minutes) Add your colleagues’ numbers when they arrive, then discuss the reasons together.

Running an HR session instead? Start with draft ratings, each with one line of evidence, and keep them in the file beside the final ratings and the reasons for any change. For practices you can run on your own, see calibration techniques for founders.

After the session, hand the register to someone who was not there. If they can name the question, the first estimates, the resolution rule, and who owns the next step, the session produced an output, not just agreement. At the review date, record the outcome from the agreed evidence. Repeated across many forecasts, that check is how you start to learn whether your confidence is calibrated.

Cite this:What is a calibration session, and what should it produce?Len P. van der Hof. https://lenvanderhof.com/en/blog/what-a-calibration-session-produces/ · Published 18 October 2026.