Evidence that fits your belief tells you little if you would see it just as often when the belief is false. The observation may be accurate, and useful for another purpose. On its own, it gives you little reason to change this particular belief.
Imagine a founder deciding whether to put more money into a new sales channel. A prospect sends a warm reply in the first week. That fits her belief that the channel will meet its 90-day revenue target. It also fits a channel that attracts interest and misses the target. The reply deserves a follow-up. It does not yet justify a higher forecast.
The distinction is between consistent with and discriminates. An observation is consistent with a claim when the two can coexist. It discriminates when one explanation makes it more likely than a relevant rival does. Something can fit your belief and still count against it, if the rival makes it more likely. The question is what each account predicted, not whether the sign feels encouraging.
Why is “consistent with” such a low bar?
Suppose inquiries rise after the founder opens the new channel. The increase is consistent with the channel working. It is also consistent with a seasonal rise in demand, or with a team that sent more outreach that month. Each account can accommodate the same result. If they all predicted an increase, the increase alone cannot tell you which one is right.
Start by naming the claim precisely. “The channel works” could mean it creates inquiries, produces signed orders, or brings in revenue that would not otherwise have arrived. Those claims call for different comparisons. For a forecast, “this channel will meet the stated revenue target by day 90” has a clear rival: it will miss. For a causal claim, “the channel caused the increase” needs a rival such as “demand rose across all channels in the same period.”
Then describe the observation without attaching your verdict. “Prospects seemed interested” is hard to check. “At least one prospect sent a positive reply by day seven” is clearer, provided you decided in advance what counts as positive and recorded every eligible reply. Now you can ask how often you would see that same observation if the target is met, and how often if it is missed.
The Evidence-Aware Life keeps apart two checks that are easy to merge. Chapter 2 asks what kind of support you have and whether it is rigorous, independent and relevant. Chapter 5 asks a further question once you have an observation: how expected is it under the claim, and under a relevant alternative? An accurate record of a sign that both accounts predict still discriminates poorly.
What makes evidence diagnostic?
Chapter 5 borrows a central question from Bayesian reasoning: how likely would this evidence be if the claim were true, compared with how likely it would be if the claim were false? When “false” is too broad to judge directly, name the main rival explanation and ask the question in two parts:
- How likely would I see this observation if my claim were true?
- How likely would I see the same observation if my claim were false, or if the main rival were true?
The likelihood ratio is the first answer divided by the second. A ratio near one means the observation gives little reason to favor either account. Above one, it favors the claim; below one, it favors the rival. The further the ratio sits from one, the stronger the pull. Neither answer is yet the probability that the claim is true after you have seen the observation.
For that you also need the starting odds: how plausible the rival outcomes were before the new sign appeared. A signal can be far more common among successes than among failures and still leave failure the likelier outcome, if failures were much more common to begin with. Chapter 5 makes this distinction, and warns that a formal-looking calculation without defensible inputs can add false precision.
Before calculating anything, check the observation itself. Was the measure defined before anyone saw the outcome? Does the record include the campaigns that missed, or only the ones someone remembered? Are the past cases comparable to this decision? A duplicated lead, a definition of “qualified” that drifted, or three reports repeating one event can make a signal look stronger than it is. Chapter 2’s checks for rigor, relevance and independence still apply.
Worked example: one sign fits, another discriminates
Consider a hypothetical archive of 1,000 comparable campaigns. Twenty met their revenue target by day 90; 980 missed it. The target and two first-week signs were defined before the campaigns began, and every campaign was recorded under the same rules, failures included.
The first sign is at least one warm reply by day seven. The second is at least ten qualified prospects requesting a second call by day seven. The two can overlap, so the columns below are separate counts, not categories to add together.
| Hypothetical outcome | Warm reply | Ten second-call requests | All campaigns |
|---|---|---|---|
| Target met | 19 | 18 | 20 |
| Target missed | 931 | 98 | 980 |
| Total | 950 | 116 | 1,000 |
The warm reply fits success but does not separate it from failure. It appeared in 19 of the 20 campaigns that met the target and in 931 of the 980 that missed: 95% in each group, so for this forecast its likelihood ratio is one. Among the 950 campaigns with a warm reply, 19 met the target: the same 2% as in the whole archive. The replies were real. They just could not tell the two groups apart.
The second sign behaves differently. It appeared in 18 of 20 successes (90%) and in 98 of 980 misses (10%). In this invented archive, the sign was nine times as common among campaigns that met the target as among those that missed. That gives the founder a reason to raise her forecast.
By how much? Across the whole archive, 20 of 1,000 campaigns succeeded: 2%. Among the 116 campaigns with ten second-call requests, 18 met the target, about 15.5%. A strong discriminator moved the estimate a long way, yet 98 of those 116 campaigns still missed. The starting odds did not vanish.
These numbers illustrate the comparison. They are not market rates or a forecast for a real company. The tidy result assumes an archive that includes the failures, uses one target and one window, and resembles the decision at hand. If the founder had picked the ten-request threshold after seeing which campaigns succeeded, or if “qualified” changed between campaigns, the fraction would look sturdier than its basis allows. Nor should she count overlapping signs as separate support: the second-call requests may come from the same prospects who sent warm replies.
The table also answers a forecasting question, not a causal one. In a suitable archive, a first-week sign can predict whether the target is met without showing that the channel caused the revenue. To investigate cause, the founder would need a comparison that gives alternatives such as shifting demand or extra outreach a chance to show themselves.
What if you cannot defend percentages?
If you have no comparable archive, do not invent conditional frequencies to make the update look exact. Chapter 5 suggests a smaller record: note which way the observation points, and why the size of the move is uncertain. A simple way to keep that record:
| Your best comparison | What to record |
|---|---|
| About as likely under the claim as under its rival | Fits both; little reason to move on this observation alone |
| More likely under the claim | Favors the claim; size uncertain |
| More likely under the rival | Counts against the claim; size uncertain |
| You cannot make the comparison yet | Diagnostic value unknown; name the missing cases or measure |
For the founder, a warm reply without comparable past campaigns reads: consistent with success; diagnostic value unknown. No reply after two days may be just as ordinary under both outcomes. A missed milestone later in the agreed sales cycle might count against the target, but only if that milestone is needed to reach the target and prospects have had time to get there. State those conditions before you interpret silence.
If a sign matters enough to change the investment, define the outcome and its resolution date. Chapter 3 asks for a forecast that another person could later resolve with the rule and date you wrote. A dated target keeps “success” from quietly changing meaning after the result arrives. The stakes then govern how much to commit and how much more to investigate. They do not change how likely the claim is.
How do you find a more useful observation?
Ask what you would expect to see if your belief were wrong. For the forecast, the founder can name a milestone that has to happen before the revenue target is within reach, set a review date, and record whether it happened. If the milestone really is required, missing it by that date is far more likely on the way to a miss than on the way to a hit. She should take that milestone from her own sales process, not borrow the invented threshold in this example.
The causal claim needs a different comparison. If inquiries rise across old and new channels while outreach stays the same, the new channel has a weaker claim to the whole increase. If inquiries rise only through the new channel, that is more informative, though other differences may remain. Decide what you will record before you look at the result. A useful comparison gives the rival a fair chance, instead of arranging the test so that every outcome can be called support.
One observation rarely settles a whole decision. A small, reversible trial may be reasonable on a rough directional update. A commitment that is hard to undo can call for better records or a narrower first step. The LEVEL framework, set out step by step in What is LEVEL?, ties the claim and its evidence to those stakes. Its Value and Expose steps are natural places to record this comparison and its gaps. LEVEL is the book’s mnemonic for these questions, not a validated scale or a promise of better outcomes.
How is this different from the other evidence checks?
Case study vs evidence asks what one documented case can show, and why it cannot tell you how often the outcome occurs across comparable settings. Familiar vs true asks whether repeated retellings have been mistaken for independent support. Those checks help you see what you actually hold and count it honestly. This one asks whether an observation, however well recorded, is more expected under one explanation than under another.
An evidence ledger records what was asserted, which source supports it, how strong that source is and what would retire the claim. A row is a good place to keep this comparison, but the two jobs differ. The ledger records evidence; this test asks whether a piece of evidence discriminates between rival explanations at all. An accurate row can still hold a sign that fits both accounts.
Try this today (15 minutes)
Choose one belief that is steering a decision you face now. Write it as a checkable outcome, with a date if you can, and state a relevant rival. Then list the three pieces of evidence you lean on most. For each, write how likely you would be to see it if the belief were true, and how likely if it were false.
Belief and date: _______________________________________
Relevant rival: _______________________________________
Evidence 1: ___________________________________________
Expected if true? __________ If false? _______________
Evidence 2: ___________________________________________
Expected if true? __________ If false? _______________
Evidence 3: ___________________________________________
Expected if true? __________ If false? _______________
One observation likely only if I were wrong (or far more likely then):
_______________________________________________________
Where and when I will check it: ________________________Keep as support only the pieces where you can explain a clear gap in favor of the belief. A piece that is more likely if you are wrong is evidence against you; record it that way. Write “unknown” where you cannot make the comparison, rather than forcing a number. Then fill in the last two lines: one observation you would expect only, or far more often, if you were wrong, and where and when you will look for it. Your belief may stay the same. Your reasons will be easier to test.
Cite this:Diagnostic evidence vs fitting evidence: what changes a belief?Len P. van der Hof. https://lenvanderhof.com/en/blog/diagnostic-evidence-vs-fitting-evidence/ ·