A thumbs-up records your reaction to one agent output; a scorecard tests that output against the role’s written standard and can change what the role is allowed to do. The distinction matters when the job repeats. One approval can tell you to keep a draft. It cannot, by itself, tell you whether the role has earned less supervision.
This is Evaluation in ROSTER, the operating method in The Agent Dream Team for running a specialist roster. Each role’s charter states its mandate, input contract, output contract, permitted tools, and escalation rule. The scorecard checks the output contract against named criteria, records exceptions, and gives the weekly review something concrete to act on.
What does a thumbs-up record?
It says you approved of something when you saw it. Perhaps the answer met the brief. Perhaps the prose sounded assured, or you had three minutes before a call. The same tap can stand for each of those judgments. It does not tell a later reviewer which source you checked, whether the question was answered, or which requirement still needed work.
A thumbs-up on Monday’s supplier brief and one on Friday’s may therefore look identical in the history even if you meant different things. A thumbs-down has the same problem. Unless you add a reason and define an approval workflow, neither reaction gives you a basis for comparing like outputs across weeks or changing a role’s authority.
Keep the reaction if it helps. It can flag a draft worth discussing or show a writer that this version suits your taste. Then ask what earned the reaction. “The source behind the pricing claim resolves” is usable feedback. “Looks good” is not a criterion you can repeat.
| Review | What remains in the record | Management use |
|---|---|---|
| Unannotated thumbs-up | One immediate reaction to one output | Reopen that item and ask why it was approved |
| Scorecard | Output, date, role-specific criteria, levels, gate result and reason | Compare like work across reviews and decide on a named permission |
What belongs on an agent scorecard?
Start with the output contract in the agent charter: what this role delivers and in what form. A Research Specialist needs criteria for research output, not for how polished its prose sounds. The book’s starting design uses three to five criteria, with described levels 3: Excellent, 2: Acceptable, 1: Needs revision. These are the author’s operating choices, not tested cutoffs.
Write levels a second reviewer could apply to the same output. “Good sources” leaves the hard decision to the reviewer. “Every material claim has a link to a source that actually supports it” tells them what to inspect. Grade the deliverable, not the length of the agent’s working log or how hard it appeared to try.
Here is a hypothetical Research Specialist. Its contract calls for one supplier brief per assigned question: a direct answer, a source list behind factual claims, material unknowns and a next step. The sample decision question is: “Should we invite this supplier into a limited pilot?” Its first rubric could be:
| Criterion | 3: Excellent | 2: Acceptable | 1: Needs revision |
|---|---|---|---|
| Evidence traceability | Every material claim has a direct link to a source that supports it | Every claim is supported, but one citation needs a format repair | A material claim is unsupported or its cited source does not support it |
| Fit to the decision | Answers the pilot question and explains the main trade-off | Answers it but leaves one trade-off thin | Answers another question |
| Uncertainty | Names material unknowns and explains how each affects the decision | Names the unknowns without their effect | Presents a material unknown as settled |
| Contract completeness | Answer, sources, unknowns and next step are present and usable | All four are present; one needs a format repair | A required part is absent |
Now mark a separate pass-or-fail release gate: every material factual claim must have verifiable support before this brief informs a supplier decision. Check the gate before reading the average. A 1 on evidence traceability in this rubric fails it, but a high score on the other three criteria cannot clear it. A 2 may also require a citation repair before handoff. The average is a summary of the rubric, not a probability that the brief is true.
Keep the rubric version with each score. If a defect exposes a vague level, sharpen it and start a new version; otherwise next month’s numbers may refer to a different standard. A change to the charter itself changes the job being graded. The book resets the role’s earned autonomy to L2 or below after such a change until the new configuration has earned its own record.
A worked example: what do five briefs show?
All five briefs and scores below are hypothetical. Score each at delivery in the order shown above. The source gate is a separate decision.
| Brief | Evidence / fit / uncertainty / completeness | Average | Source gate | Review note |
|---|---|---|---|---|
| A | 3 / 3 / 3 / 3 | 3.00 | Pass | Ready for founder review |
| B | 3 / 2 / 3 / 3 | 2.75 | Pass | Pilot trade-off needs one clearer line |
| C | 1 / 3 / 1 / 3 | 2.00 | Blocked | States that current insurance is confirmed; no supporting record |
| D | 3 / 3 / 2 / 3 | 2.75 | Pass | Names an unknown without its effect on the pilot |
| E | 3 / 2 / 3 / 3 | 2.75 | Pass | Pilot trade-off lacks a decision condition |
The five averages sum to 13.25, so the week’s average is 2.65 out of 3. That number hides brief C. Its unsupported insurance claim blocks the brief from use in the decision until a person verifies or removes it. A tidy synthesis cannot compensate for that gap. In the book’s ladder, a gate failure also prevents promotion in that review window. For a role at L2, one gate-relevant failure calls for a move to L1 at the weekly review: the role can still draft, but the founder checks every output before handoff.
The score identifies a problem, not its cause. Inspect C’s source list and the input packet. Was a source omitted, did it fail to support the claim, or did the original request imply an answer it could not substantiate? Record what you find. Otherwise a model change, trigger fix and rubric rewrite all look equally plausible from the number alone.
Why keep the objective score beside the scorecard?
The Agent Dream Team uses two numbers. The 0.0–1.0 objective score asks whether the role met the target you set before the week’s work. The 1–3 output score asks how well each delivery met its standing contract. Read them side by side at the weekly review; never average them into one grade.
Suppose Monday’s target was simply “deliver five briefs by Friday.” All five arrive on time, so the count-and-deadline target scores 1.0. Their output-quality average remains 2.65, and C remains blocked. A mandatory factual check would sit outside the objective fraction too: a failed mandatory condition blocks delivery even when the count scores 1.0. An excellent brief can also miss a stretch target. Write the target before the work arrives, or Friday’s grade becomes a description of what happened instead of a test of what you asked for.
When does a score change a permission?
The book’s Agent Trust Ladder gives each role a level of delegated autonomy. L1 is draft-only, with every output reviewed. At L2 (Suggest-with-confidence), the role may pass work that meets conditions you can check from outside the model, and it flags the rest. The book’s examples of such flag conditions: fewer than three sources, a client the role has not seen before, an output format the charter does not name. The model’s own confidence never counts. Higher levels allow more routine work inside the charter. The level belongs to the role, not the model running it. Swap the underlying model, the prompt or the tools, and the role runs under restricted review until the new setup has requalified.
Write the permission as a sentence. For the hypothetical specialist: “At L2, hand an internal supplier brief to the Draft Editor only if the evidence gate passes, the pilot question is answered and this output averages at least 2.5. Otherwise hold it for my review.” That per-output threshold is part of this example. In the book, 2.5 is the author’s own floor for a weekly average at L2 and above, not a validated cutoff for individual briefs. The Friday record should state one of five decisions: hold, promote, demote, flag for charter review, or escalate to manual override, with a reason and a date to look again.
The book requires four consecutive qualifying review cycles and enough scored work for a promotion; a gate failure or one week below the current floor can demote a role. In the five-brief example, brief C costs the role its L2 handoff permission for now. Even if the other four briefs were excellent, that week’s gate failure cannot be erased by their average. If you rewrite the charter, reassess the permission against the changed job.
The ladder never waives the human final gate. Public, financial, legally binding and irreversible actions are capped at L3 in the book, and the founder confirms them before execution. A role may work at L4 on internal drafts and still need approval to publish, contact a client or commit money. The approval timeout test covers what happens when the named approver does not answer. Silence never becomes approval.
How does a scorecard become useless?
First, score only when something breaks. Then you have a failure and no ordinary work to compare it with. Score each delivery, including quiet ones, and review the pattern on a fixed day. The book puts that pass on Friday, at ten to fifteen minutes for a three-role roster. A quick mid-task check can redirect work, but it cannot replace the score on the finished output.
Second, score faithfully and never decide. A log grows while every role keeps the same authority. In the book’s words, the scorecard becomes a diary. At the weekly review, write one decision for each role and the condition for revisiting it. If evidence traceability keeps scoring 1, inspect the input, source access and charter before telling the role to “do better.” If the role no longer fits, retiring a production agent is a separate decision under Rotation.
Sometimes the rubric passes work you still would not send. Hold it at the human gate, name the defect, and decide whether to add a criterion or escalation rule. Keep that exception visible beside the score. A silent change of mind teaches next week’s reviewer nothing.
This is an operating design, not a proven intervention. Chapter 8 cites a 2019 review of human acute-care teamwork as a correlational analogy. It does not report a trial of agent scorecards. Five good scores and the book’s chosen thresholds cannot establish that the next output will be reliable.
Try this today (30 minutes)
Pick one role and collect its last five outputs of the same kind. Open its output contract. If it has none, write the expected deliverable before you grade past work, and mark that standard as provisional.
- Write three to five criteria from that deliverable. Define levels 1, 2 and 3 for each. Write any release gate as a separate pass-or-fail condition.
- Score all five outputs, including the ordinary ones. Keep a link, date, rubric version, criterion scores, gate result and one exception note in each row. If you cannot explain a level, rewrite the level before using it to change permissions.
- Write one permission the role has or could earn. State the score and number of review cycles required to gain it, the score or gate failure that withdraws it, and the actions that still need your approval. Put the next review date beside the decision.
Ask a second person to read the five rows. Can they see why each score was given and why the permission did or did not change? One batch may expose a gate failure; it cannot satisfy a four-cycle promotion rule. If the answer depends on how you felt when you clicked, the scorecard still needs work.
Cite this:An agent scorecard turns output review into a permission decision.Len P. van der Hof. https://lenvanderhof.com/en/blog/agent-scorecard-vs-thumbs-up/ ·