One documented case study can show that a result occurred in one setting; it cannot, by itself, estimate how often comparable companies see that result or whether yours will. One customer’s outcome is a lead for a decision, not a forecast.
In a hypothetical example, a founder is about to buy an automated onboarding tool. A vendor’s case study says a customer’s support tickets fell after installing it. The page names the customer, describes the rollout, and quotes a satisfied manager. Does that story justify a company-wide purchase?
The distinction is between possibility in one setting and frequency across comparable settings. A third question asks what caused the result. A fourth asks whether the conditions match your company. Each question needs its own support.
Is a case study evidence?
Yes. A case study is evidence about a particular case. Its strength depends on what you can inspect. A testimonial with no records is thin evidence. An account with dated outcomes, a clear sequence, and an honest account of other changes supports more precise claims about that setting. The label on the cover tells you little about the quality inside.
The Evidence-Aware Life places a reported experience on the second rung of its Evidence Ladder. The seven rungs are assertion, anecdote, authority, mechanism, study, convergence, and your own test. These are prompts for inspecting what kind of support sits underneath a claim, not automatic quality scores. Chapter 2 asks you to check rigor, independence, and relevance on every rung. A careful case account can be useful, but its detail alone does not give you a rate across companies. A case study is one rung on the ladder, not a verdict.
Keep four conclusions apart:
| Question | What one documented case can give you | What it cannot settle alone |
|---|---|---|
| Did it happen? | A record of the outcome in one place and period, subject to checking. | Whether the record is accurate and the outcome was measured consistently. |
| How did the work unfold? | The steps, constraints, and a possible mechanism. | Which step caused the outcome, or what would have happened without it. |
| How often does it happen across companies? | One company’s result, and possibly rates within that company. | A rate among comparable implementations elsewhere. |
| Would it work here? | Conditions to compare with your own. | Which conditions were necessary, or the result in a different team, customer group, or market. |
A case can contain more than an anecdote. If it includes a comparison or operational data, examine those pieces on their own terms. Still ask which conclusion each piece supports. Chapter 8 of the book makes this separation with an invented support-team case: the events are stipulated for the exercise, while rival explanations and the advice for another team remain open.
What can one case usefully show?
First, it can establish possibility in one setting and show the mechanism of the work. In the onboarding example, the useful details are which questions customers asked, where automation answered them, when a person stepped in, and which requests stayed unresolved. Those details show how the rollout unfolded step by step and tell you what to inspect in your own process. They do not tell you what would have happened without the change, or which conditions were necessary for the result.
Second, a verified case can serve as a counterexample to a literal universal claim. If someone says that an approach never works, one well-documented instance meeting the same definition of success challenges that sentence. It says nothing about whether the approach usually works. Be just as careful with the definition: fewer tickets need not mean that customers received better help.
Third, a credible account of harm can justify a prompt safety check before you know how common the harm is. If users in one case could not reach needed human support, you can inspect that path before expanding a rollout. The stakes can justify a precaution while the rate is unknown. They do not make the rate known.
The useful habit is to write the narrow sentence the case supports before writing the larger claim you hope to act on. The space between those sentences is your next investigation. How hard you check depends on what you plan to do with the case. Chapter 8 puts it plainly: “The use you intend to make of the story determines how much checking is worthwhile.”
Where does one success become a bad forecast?
A vendor’s featured customer becomes the average customer. The story may be accurate, and the customer may be pleased. But a case selected for publication is not a defined sample of everyone who tried the product. Ask how many comparable deployments there were, which customers were eligible for a case study, and what happened in the attempts that were not written up. Counting published successes without those attempts cannot give you a success rate.
A famous company’s history becomes your playbook. You can copy the visible tactic while lacking the company’s starting position, distribution, staff, or timing. The book’s introduction illustrates this with Daniel, a fictionalized composite founder who treats another founder’s channel success on a podcast as a forecast for his own business. The book names the trap: the founders who tried the same thing and failed never got a podcast. That is survivorship bias. Daniel’s story teaches the question; it is not evidence about a documented company. The reasoning carries over: if you hear chiefly from companies whose outcomes made good stories, you cannot infer how typical their tactics were.
One account becomes many voices. Chapter 2 warns against mistaking “one source in twenty coats” for independent convergence. Chapter 10 asks you to suppose that everyone in Daniel’s meeting has read the same vendor case study. Their agreement may reflect that reading accurately, but it adds no independent case. Two people who had also examined failed deployments would bring more evidence, even if they disagreed about what to do. The same applies when several blog posts or sales decks retell one customer’s result.
None of this requires accusing the vendor, founder, or team of dishonesty. It requires keeping the unseen attempts in view when you decide what one visible attempt can support.
Worked example: the ticket count fell
Suppose the vendor case gives the founder two onboarding cohorts. Every figure below is hypothetical. The before and after cohorts each had 30 days of follow-up, and tickets are counted over each customer’s first 30 days.
| Cohort | New customers | Onboarding tickets | Tickets per new customer |
|---|---|---|---|
| Before the tool | 400 | 120 | 0.30 |
| After the tool | 200 | 90 | 0.45 |
Total tickets fell from 120 to 90. That headline is arithmetically true. But half as many customers entered onboarding, and tickets per new customer rose from 0.30 to 0.45. Neither figure tells you whether service improved. More tickets could reflect more trouble, or easier access to help. Unresolved requests and the route to a person matter too.
The before-and-after comparison also cannot credit the tool with either change. The customer mix, help pages, staffing, or product may have changed at the same time. A more useful case would disclose those changes and show a comparison that helps distinguish plausible explanations. Even then, you would still need to ask whether that customer’s conditions resemble yours.
The founder should ask the vendor a different question about frequency: among all comparable customers who adopted the tool, what share improved on a defined onboarding outcome over the same period? That question needs the unsuccessful and unfinished attempts as well as the featured ones. It needs a fixed measure and time window. The startup-statistic check uses the same discipline: fill in who is counted, what event, and over what time.
If the vendor cannot supply that denominator, write rate unknown from this material. A collection of chosen stories does not fill the gap. The founder can still decide that a small test is worth doing, but should not put an invented success rate into the business case.
What evidence would justify your next move?
In chapter 10’s twenty-five-minute decision meeting, the team keeps two inputs apart: a vendor case study suggests that an onboarding change is plausible, while Daniel’s own support records identify the problem they hope to solve. Neither establishes that this particular change will work. Start the same way. Inspect your own support records first. Is onboarding actually producing avoidable tickets, or is the proposed tool solving a problem you have not measured? Define the eligible customers and the outcome before a pilot. In this example, the founder might track tickets per new customer, unresolved requests, and whether customers can still reach human help.
Next, limit the trial to one suitable onboarding path. Keep the existing route to a person. If a comparable group is feasible, use it; if not, record what else changed during the trial. Leave a short team decision record naming the action, the owner, the evidence gap, the stop condition if requests cannot be handled safely, and the review date. Do this before the first result arrives, when it is easier to set a fair standard.
Such a trial may reveal fit or a failure in this company. It will not estimate the tool’s market-wide success rate. A small before-and-after change will not, by itself, isolate cause either. Those limits do not make the trial useless. They tell the founder how much commitment its result can carry.
The book’s LEVEL framework gives the decision a short record:
- Locate: “The tool will reduce onboarding tickets per new customer here without leaving more requests unresolved.”
- Estimate: What would a full rollout cost if that claim is wrong? What would waiting or a limited trial cost?
- Value: Name the rung, one customer’s account, then check its records, selection, independence, and relevance alongside your own support data.
- Expose: The vendor’s denominator, rival explanations, and fit with your customers remain open.
- Lean: Ask for the missing record and, if safe and feasible, run a bounded trial before expanding.
The book offers LEVEL as a practical synthesis, not a validated guarantee of better outcomes. The five-step explanation sets out the full check. If several claims are travelling between the vendor page, budget memo, and rollout plan, keep a claim-level evidence ledger so the case does not acquire a broader conclusion in each retelling.
Try this today (15 minutes)
Choose one case study shaping a current decision. Keep the source open and write four lines:
What it shows: [Observed outcome, setting, time window, and inspectable record]
What it cannot show: [Frequency across comparable cases, cause, and fit with our setting]
Base-rate question: [Among which comparable attempts did this outcome occur, over what period?]
Cheapest safe test: [Bounded local check, owner, measure, stop rule, and review date]If the first line has no inspectable record, ask for one. If the third has no denominator, leave the rate unknown. If no safe test is available, seek better evidence before making a larger commitment. The case earns attention. The next move depends on what you can actually check.
Cite this:Case study vs evidence: what one success can tell you.Len P. van der Hof. https://lenvanderhof.com/en/blog/case-study-vs-evidence/ ·