--- title: "Model picking audit vs benchmark screenshot: what each shows | Len P. van der Hof" description: A benchmark screenshot shows one dated test result. A picking audit records why each production model was chosen, for which task, and which picks need review. image: "https://lenvanderhof.com/media/generated/blog-hero-picking-audit-vs-benchmark-screenshot.1f6fb2ad2eb1.wide.webp" --- [AI Systems](https://lenvanderhof.com/en/blog/category/ai-systems/) Research guide # Model picking audit vs benchmark screenshot: the rules your stack already runs on A chart can suggest a candidate. The audit shows why each model got its job, and which reasons nobody can defend. Len P. van der HofPublished 4 November 202610 min read The boxed newcomer draws the eye. The board shows where each tool belongs, and which places stand empty. Direct answer A benchmark screenshot shows how some models scored on someone else's test, with one metric, on one date. A model picking audit records every model your team runs: the task it serves, the real reason and date it was chosen, who owns that choice today, the evidence behind it, and each task's latency need, quality bar and data sensitivity. Unknown, inherited, demo-led and leaderboard-only reasons go to the top of the review queue. The audit finds the decisions that need evidence. It does not tell you which model should replace them. ## Key takeaways - A benchmark result measures its own test on its own date. Keep the date, metric and test conditions with any screenshot you save. - For every model in use, write the actual selection reason, the task, the date, today's owner and the evidence. Write 'unknown' when nobody knows. - For every task, write the latency need, quality bar and data sensitivity. Anything left unspecified is a gap your routing policy cannot fill. - Unknown, inherited, demo-led and leaderboard-only picks go to the top of the review queue. A flag asks for evidence, not an automatic model swap. - The book's score counts models accounted for: a reason that names task, owner and evidence, and no unspecified task need. A leaderboard position alone never clears that bar. **A benchmark screenshot tells you how some models scored on someone else’s test, on one date; a picking audit tells you why each model in your stack is there.** The screenshot can put a candidate on your list. It cannot tell you why an inherited endpoint still handles customer data, what a five-minute demo proved, or whether a task has a response-time limit at all. [The Model Portfolio](https://lenvanderhof.com/books/the-model-portfolio/) makes the picking audit the entry point to its method: list every model you run, write down the real reason each was chosen, and find the choices nobody can account for. In the book’s words, a leaderboard score is a model’s resume. A resume is not a hiring decision. ## What does a benchmark screenshot actually show? A leaderboard score reports how a model performed on a specific set of tasks, under specific conditions, graded by specific criteria, at the time the evaluation ran. That is real information, but not an answer to your question: how will this model perform on your tasks, in your system, under your load, judged by your quality bar? The book calls that distance the **benchmark-production gap**. Production work is rarely as clean as a benchmark: a customer chatbot that answers billing questions, handles returns and fields the odd technical complaint is three or four tasks behind one interface, each with its own reasoning needs, latency tolerance and quality threshold. A screenshot also shows one board, and boards crown different winners because they score different things. EvidenceQuestion it answersQuestion it leaves openBenchmark screenshotHow did these models score on this test, with this metric, on that date?Does the result transfer to our task, under our load and constraints?Picking auditWhy is this model in our stack, which task does it serve, and what evidence backed the choice?Does a test on representative examples still support that assignment? Dates and conditions matter because results move. Three dated cases from the book: - **Copilot Arena, reported in February 2025.** A VS Code extension built by university researchers showed developers paired code completions from ten models and recorded which one they took: more than 4.5 million suggestions and 11,604 votes from 1,642 users over roughly three months. Its ranking had a Spearman rank correlation of 0.1 or lower with most static coding benchmarks (0.62 with a chat preference arena’s coding board, 0.48 with its general board), and smaller models ranked higher on static benchmarks than in developers’ own editors ([Chi and colleagues](https://arxiv.org/abs/2502.09328)). One platform, one task type, one three-month window: it shows that rank does not carry across deployment contexts, not that in-editor rankings beat benchmarks. - **GSM1k, 2024.** Scale AI wrote a fresh grade-school maths test to mirror GSM8k. Several model families scored up to roughly eight percentage points lower on it, and each model’s drop tracked its tendency to reproduce GSM8k problems verbatim ([Zhang and colleagues](https://arxiv.org/abs/2405.00332)). That is a correlation: the same paper finds the affected families still generalize to new problems. - **SWE-bench Verified, 23 February 2026.** OpenAI announced it would stop reporting this coding benchmark. Its audit of a 27.6% subset of frequently failed tasks found flawed test cases in at least 59.4% of them, and every frontier model it tested showed signs of training-data contamination ([OpenAI](https://openai.com/index/why-we-no-longer-evaluate-swe-bench-verified/)). The 59.4% describes the audited subset, not the whole benchmark. None of this makes benchmarks worthless. A published score is a measurement with a shelf life, valid for its own test: keep the screenshot with its date, metric and conditions, and treat it as a reason to test a candidate on your own task, not to route traffic to it. ## What does the picking audit write down? The book’s protocol card has seven steps: 1. List every model running anywhere in your stack, including experimental integrations, vendor-supplied endpoints and tools individual team members use. 2. Write down why each was chosen. If nobody knows, write **unknown**. 3. Write down the tasks each model handles. 4. For each task, write down the **latency need, quality bar and data sensitivity**, marking any gap **unspecified**. 5. Find models whose reason is **unknown**, **inherited** or **demo results**: the highest-priority candidates for re-evaluation. 6. Find tasks with any need unspecified: gaps your routing policy cannot currently fill. 7. Find models missing from a formal registry. These are shadow models, and they need documenting now. The chapter’s 60–90 minute worksheet turns these into six parts: - **Part 1, model inventory:** model or endpoint, provider, tasks, approximate calls per day, date added and who added it. Flag any tool whose LLM calls you cannot confirm. - **Part 2, selection reason audit:** the actual reason, naming the task, today’s owner and the evidence, plus one of six categories: **Benchmark, Demo, Inherited, Cost, Vendor default, Unknown**. The book wants a description, not a justification. “It is a strong model” is a justification; “we reused the launch endpoint because it was already integrated” is a description you can check. - **Part 3, task need gaps:** latency need (**real-time** under 500 ms p95, **near-real-time** under five seconds p95, **batch** minutes or longer, or **unspecified**), quality bar, data sensitivity (**public, internal, confidential** or **unspecified**) and any privacy host constraint. - **Part 4, shadow model check:** tools the inventory missed, vendor products making LLM calls you never set up, and when every model was last formally reviewed. - **Part 5, priority flags:** your top three issues, most urgent first. That list is the audit’s work queue. - **Part 6, score the audit:** models accounted for, out of the Part 1 total. The book is plain that this is an inventory and hands out no verdicts. A poorly explained route is not proof of poor work; any move waits for a test on representative examples against a written quality bar. ## Which unwritten rule picked the model? Every team has a model selection process; most have not written it down. The book calls the real one the **implicit rulebook**: the heuristics that explain the choices actually made, seldom the ones a team would describe. Five surface when teams first run the audit: - **Flagship by default.** The newest frontier model starts every new feature. Deviating needs a justification; staying put needs none. - **Whatever the previous engineer used.** The adjacent feature gets the existing model because it is already integrated, and asking why feels like criticism. - **Best five-minute demo.** Better-looking output in three manual tests decides. That is a fluency signal, not a graded result ([Evaluate the reasoning, not the fluency](https://lenvanderhof.com/en/blog/evaluate-reasoning-not-fluency/)). - **Whatever the tool vendor supplies.** The first vendor SDK integrated made the routing decision. - **Cheapest that doesn’t obviously fail.** Cost is the only dimension, and “doesn’t obviously fail” the only quality gate. None is irrational given what the team knew at the time, the book notes, but all produce suboptimal outcomes at scale, because none accounts for task needs, latency class, privacy tier or cost at realistic traffic. Aggregate billing hides them: it answers *Can we afford the current bill?*, not *is our current routing policy cost-efficient?*, and the book calls it a lagging indicator of routing decisions made months earlier. The book shows the accumulation through Francesca and Amit, **composite characters rather than a measured company case**. Francesca’s inherited frontier endpoint, chosen on launch day for one task, now handles seven, from real-time autocomplete to a weekly research synthesis for sales. Before any route moves, Amit’s map shows four tasks with batch latency tolerance and low reasoning demand, and two that want speed more than depth. Those six become candidates to leave the frontier tier if parallel testing confirms quality parity; the research synthesis is the one task the book says genuinely rewarded deeper reasoning. Until then, the saving is a forecast waiting on evidence. ## What does a scored audit look like? **Hypothetical example.** A small product runs three model endpoints. Every date, owner, test size and threshold below is invented to show how the worksheet scores; no production result is reported. Model and taskActual selection reasonTask requirements and findingA: nightly support summariesThe operations lead, today’s owner, chose it on 12 September 2026 after a 50-ticket test met a written acceptance rule.Batch, done by 07:00. At least 45 of 50 held-out summaries must pass a fixed rubric. Internal data, approved host. Task, owner and evidence on record.B: instant billing repliesThe support lead, today’s owner, chose it on 19 September 2026 because it topped a public benchmark screenshot of unknown test date.Customer billing data, approved host, but no latency need or quality bar written down. Nothing links the rank to this task.C: contract-field extractionThe engineering lead, today’s owner, inherited it with a vendor SDK. Original date and evidence unknown.Batch, due by 09:00. Output format specified; quality bar and permitted host for confidential contracts are not. Part 6 counts **models accounted for**, not models judged good: the reason names the task, today’s owner and the evidence, and no task need is marked unspecified. **Demo**, **Inherited** and **Unknown** never clear that bar. **Benchmark**, **Cost** and **Vendor default** do not clear it on their own either, the book adds, because a leaderboard position, a low price and an SDK that shipped first are all reasons that skipped the task. A clears the bar. B fails twice: a screenshot rank is its only reason, and its task has no latency need or quality bar. C fails twice too: an inherited pick, with quality bar and host unspecified. The score is **one of three accounted for**. A’s small test supports its selection; it does not prove A is still the best model. The book sorts scores into four bands: **every model accounted for**, **all but two or three**, **half or fewer**, and **Part 1 itself is incomplete**. At half or fewer, the implicit rulebook is running your stack, and the book sends you to the ROUTE framework before you patch individual rows. ## What goes on the re-evaluation queue? Put **unknown**, **inherited** and **demo** reasons at the top, as the protocol card directs, and add **leaderboard-only** reasons: Part 6 already treats a leaderboard position alone as a reason that skipped the task. Cost-only and vendor-default picks need the same task-specific evidence. A flag asks for evidence; it does not order a swap. In the hypothetical, C comes first: identify the host behind the vendor SDK and whether confidential contracts may go there. Then B: write down the latency need and quality bar, recover the screenshot’s date and conditions if you can, and test on representative billing requests. A gets a later review date. That order is editorial judgement, not a measured risk ranking. The audit also shows which of the book’s four failure modes is most acute, and so where to start in its **ROUTE** discipline. Shadow models point to **Register**. Task mismatches, such as frontier capacity on batch jobs, point to **Objective typing**, which the book says closes the benchmark trap; B belongs here. An implicit rule nobody can defend, like C’s inherited default, points to **Utilize policy**. Cost opacity or latency misses point to **Track**; models never re-evaluated since they were added, to **Evolve**. [What is model routing?](https://lenvanderhof.com/en/blog/what-is-model-routing/) covers the written policy that follows, [Frontier, workhorse, or local](https://lenvanderhof.com/en/blog/frontier-vs-workhorse-vs-local/) the model classes, and [Is one AI model enough?](https://lenvanderhof.com/en/blog/one-model-is-not-a-portfolio/) when a single model works and when you need a portfolio. ## Try this today Give it twenty minutes for a first pass. 1. List the models and model-backed endpoints doing production work. Ask route owners about vendor features and team tools. If the list will not close, write **inventory incomplete** at the top. 2. Beside each, write the task, the **real reason it was chosen**, the date and today’s owner, or **unknown**. If a screenshot was the reason, add its test date, metric and conditions, or mark them unknown. 3. For each task, write the latency need, quality bar, data sensitivity and permitted host, or **unspecified**. 4. Put every row whose reason is **top of a leaderboard**, **unknown**, **inherited** or **demo** on the review list. Mark every unspecified need as a routing gap. 5. Count the models whose reason names task, owner and evidence and whose task has no unspecified need. Date the list and name an owner for the first gap. An incomplete inventory is your first finding; a score would pretend you know the denominator. If the list is complete, take the book’s action for this week: for the highest-volume model whose reason reads unknown, write the missing paragraph on what it handles, why it still deserves the route and who owns that answer, dated. If nobody can write it, you have your first re-evaluation candidate. No model needs to move before its task and evidence are clear. Cite this:Model picking audit vs benchmark screenshot: the rules your stack already runs on.Len P. van der Hof. [https://lenvanderhof.com/en/blog/picking-audit-vs-benchmark-screenshot/](https://lenvanderhof.com/en/blog/picking-audit-vs-benchmark-screenshot/) · Published 4 November 2026. ## Sources 1. [The Model Portfolio](https://lenvanderhof.com/books/the-model-portfolio/) 2. [Copilot Arena: A Platform for Code LLM Evaluation in the Wild (Chi et al., 2025)](https://arxiv.org/abs/2502.09328) · arXiv 3. [A Careful Examination of Large Language Model Performance on Grade School Arithmetic (Zhang et al., 2024)](https://arxiv.org/abs/2405.00332) · arXiv 4. [Why SWE-bench Verified no longer measures frontier coding capabilities (OpenAI, 2026)](https://openai.com/index/why-we-no-longer-evaluate-swe-bench-verified/) · OpenAI ## Further reading - [Is one AI model enough? When a single model works, and when you need a portfolio](https://lenvanderhof.com/en/blog/one-model-is-not-a-portfolio/) - [What is model routing? A policy, not a default](https://lenvanderhof.com/en/blog/what-is-model-routing/) - [Frontier, workhorse, or local: three classes, not a religious war](https://lenvanderhof.com/en/blog/frontier-vs-workhorse-vs-local/) - [Evaluate the reasoning, not the fluency](https://lenvanderhof.com/en/blog/evaluate-reasoning-not-fluency/) - [The Model Portfolio](https://lenvanderhof.com/books/the-model-portfolio/) About the author ## [Len P. van der Hof](https://lenvanderhof.com/en/authors/len-p-van-der-hof/) Entrepreneur, AI Innovator and Venture Builder Len P. van der Hof builds practical AI systems, digital ventures and evidence-informed tools for founders. ```json { "@context": "https://schema.org", "@graph": [ { "@type": "Person", "@id": "https://lenvanderhof.com/#person", "name": "Len P. van der Hof", "alternateName": [ "Len van der Hof", "L.P. van der Hof", "Leendert Pieter van der Hof" ], "honorificSuffix": "MSc", "url": "https://lenvanderhof.com/", "image": [ "https://lenvanderhof.com/photos/len-portrait-1.jpg", "https://lenvanderhof.com/photos/len-portrait-2.jpg", "https://lenvanderhof.com/photos/len-portrait-3.jpg", "https://lenvanderhof.com/photos/len-portrait-4.jpg", "https://lenvanderhof.com/photos/len-speaking.jpg", "https://lenvanderhof.com/photos/len-hero.jpg" ], "jobTitle": "Entrepreneur, AI Innovator and Venture Builder", "description": "Len P. van der Hof, MSc, is a Dutch entrepreneur and AI innovator in Zwijndrecht. He builds ReasonKit, MindSesh, Undominated.ai, books under his name, the fiction imprint LPH98.lifestyle, and technology ventures through LPH98.ventures. Eleven titles in Systems for the Strategic Self are available now, in English and Dutch.", "address": { "@type": "PostalAddress", "addressLocality": "Zwijndrecht", "addressCountry": "NL" }, "alumniOf": { "@type": "CollegeOrUniversity", "name": "Rotterdam School of Management, Erasmus University" }, "knowsAbout": [ "Artificial intelligence", "AI agents", "Agentic AI systems", "LLM routing", "SEO", "Generative engine optimization", "Answer engine optimization", "Venture building", "Founder performance", "Founder psychology", "Evidence-based decision-making" ], "sameAs": [ "https://www.linkedin.com/in/lenvanderhof/", "https://x.com/LenvanderHof", "https://www.youtube.com/channel/UCTG20buKqYYbitqqf7l3zJA", "https://www.instagram.com/Lenvanderhof/", "https://www.threads.com/@lenvanderhof", "https://github.com/Lenvanderhof", "https://huggingface.co/LPH98", "https://www.npmjs.com/~lenvanderhof", "https://www.goodreads.com/author/show/70983905.Len_P_van_der_Hof", "https://www.amazon.com/author/lenvanderhof", "https://www.bol.com/nl/nl/b/len-p-van-der-hof-msc/609879394/", "https://bsky.app/profile/lenvanderhof.com", "https://mastodon.social/@Lenvanderhof", "https://crates.io/users/Lenvanderhof", "https://cursor.com/@Lenvanderhof", "https://medium.com/@Lenvanderhof", "https://gitlab.com/Lenvanderhof", "https://hub.docker.com/u/lenvanderhof/", "https://dev.to/lenvanderhof", "https://www.facebook.com/Lenvanderhof", "https://soundcloud.com/Lenvanderhof" ], "affiliation": [ { "@id": "https://lenvanderhof.com/#publisher" }, { "@id": "https://lenvanderhof.com/#mindsesh" }, { "@id": "https://lenvanderhof.com/#lifestyle" } ] }, { "@type": "WebSite", "@id": "https://lenvanderhof.com/#website", "url": "https://lenvanderhof.com/", "name": "Len P. van der Hof", "description": "Len P. van der Hof, MSc, is a Dutch entrepreneur and AI innovator in Zwijndrecht. He builds ReasonKit, MindSesh, Undominated.ai, books under his name, the fiction imprint LPH98.lifestyle, and technology ventures through LPH98.ventures. Eleven titles in Systems for the Strategic Self are available now, in English and Dutch.", "inLanguage": [ "en", "nl" ], "publisher": { "@id": "https://lenvanderhof.com/#person" } }, { "@type": "Organization", "@id": "https://lenvanderhof.com/#publisher", "name": "LPH98.ventures", "url": "https://lph98.ventures", "founder": { "@id": "https://lenvanderhof.com/#person" } }, { "@type": "Organization", "@id": "https://lenvanderhof.com/#mindsesh", "name": "MindSesh", "url": "https://mindsesh.net", "founder": { "@id": "https://lenvanderhof.com/#person" } }, { "@type": "SoftwareApplication", "@id": "https://lenvanderhof.com/#reasonkit", "name": "ReasonKit", "url": "https://reasonkit.sh", "creator": { "@id": "https://lenvanderhof.com/#person" } }, { "@type": "SoftwareApplication", "@id": "https://lenvanderhof.com/#undominated", "name": "Undominated.ai", "url": "https://undominated.ai", "creator": { "@id": "https://lenvanderhof.com/#person" } }, { "@type": "Organization", "@id": "https://lenvanderhof.com/#lifestyle", "name": "LPH98.lifestyle", "url": "https://lph98.lifestyle", "founder": { "@id": "https://lenvanderhof.com/#person" } }, { "@type": "ImageObject", "@id": "https://lenvanderhof.com/en/blog/picking-audit-vs-benchmark-screenshot/#primaryimage", "url": "https://lenvanderhof.com/media/generated/blog-hero-picking-audit-vs-benchmark-screenshot.1f6fb2ad2eb1.wide.webp", "contentUrl": "https://lenvanderhof.com/media/generated/blog-hero-picking-audit-vs-benchmark-screenshot.1f6fb2ad2eb1.wide.webp", "representativeOfPage": true }, { "@type": "BreadcrumbList", "@id": "https://lenvanderhof.com/en/blog/picking-audit-vs-benchmark-screenshot/#breadcrumb", "itemListElement": [ { "@type": "ListItem", "position": 1, "name": "Home", "item": "https://lenvanderhof.com/" }, { "@type": "ListItem", "position": 2, "name": "Blog", "item": "https://lenvanderhof.com/en/blog/" }, { "@type": "ListItem", "position": 3, "name": "AI Systems", "item": "https://lenvanderhof.com/en/blog/category/ai-systems/" }, { "@type": "ListItem", "position": 4, "name": "Model picking audit vs benchmark screenshot: the rules your stack already runs on", "item": "https://lenvanderhof.com/en/blog/picking-audit-vs-benchmark-screenshot/" } ] }, { "@type": "WebPage", "@id": "https://lenvanderhof.com/en/blog/picking-audit-vs-benchmark-screenshot/#webpage", "url": "https://lenvanderhof.com/en/blog/picking-audit-vs-benchmark-screenshot/", "name": "Model picking audit vs benchmark screenshot: the rules your stack already runs on", "description": "A benchmark screenshot shows one dated test result. A picking audit records why each production model was chosen, for which task, and which picks need review.", "isPartOf": { "@id": "https://lenvanderhof.com/#website" }, "primaryImageOfPage": { "@id": "https://lenvanderhof.com/en/blog/picking-audit-vs-benchmark-screenshot/#primaryimage" }, "breadcrumb": { "@id": "https://lenvanderhof.com/en/blog/picking-audit-vs-benchmark-screenshot/#breadcrumb" }, "inLanguage": "en-GB" }, { "@type": "BlogPosting", "@id": "https://lenvanderhof.com/en/blog/picking-audit-vs-benchmark-screenshot/#article", "mainEntityOfPage": { "@id": "https://lenvanderhof.com/en/blog/picking-audit-vs-benchmark-screenshot/#webpage" }, "headline": "Model picking audit vs benchmark screenshot: the rules your stack already runs on", "description": "A benchmark screenshot shows one dated test result. A picking audit records why each production model was chosen, for which task, and which picks need review.", "datePublished": "2026-11-04T15:05:00.000Z", "author": { "@id": "https://lenvanderhof.com/#person" }, "publisher": { "@id": "https://lenvanderhof.com/#person" }, "image": [ "https://lenvanderhof.com/media/generated/blog-hero-picking-audit-vs-benchmark-screenshot.1f6fb2ad2eb1.square.webp", "https://lenvanderhof.com/media/generated/blog-hero-picking-audit-vs-benchmark-screenshot.1f6fb2ad2eb1.landscape.webp", "https://lenvanderhof.com/media/generated/blog-hero-picking-audit-vs-benchmark-screenshot.1f6fb2ad2eb1.wide.webp" ], "articleSection": "AI Systems", "inLanguage": "en-GB" } ] } ```