--- title: "ML for Agent Builders — Auto-Training, Fine-Tuning, and Eval Loops for LLM-Powered Systems | Len P. van der Hof" description: TRAIN helps agent builders decide when owned ML, fine-tuning, retrieval, evaluation, and drift monitoring are worth the operational cost. image: "https://lenvanderhof.com/books/covers/ml-for-agent-builders-en-front.jpg?v=20260819-hitll-nobg" --- ![ML for Agent Builders — front cover](https://lenvanderhof.com/books/covers/ml-for-agent-builders-en-front.jpg?v=20260819-hitll-nobg) ![ML for Agent Builders — back cover](https://lenvanderhof.com/books/covers/ml-for-agent-builders-en-back.jpg?v=20260819-hitll-nobg) No. 12 · Manuscript complete · AI & agents · TRAIN # ML for Agent Builders Auto-Training, Fine-Tuning, and Eval Loops for LLM-Powered Systems Your agents call models all day. Time to own a few. Agents lean on models constantly, but the decision to own one stays murky. ML for Agent Builders is a build loop for the narrow models agents depend on: clear calls on fine-tune versus RAG, labeling, and eval drift, so you win on cost, latency, and control instead of renting everything. Look inside↓Notify me at launch→← All books pages 410 chapters 15 hours of reading ± 5 editions EN · NL Production status 1. Design 2. Drafting 3. Manuscript 4. Production 5. Launched The editions ## Kindle, Paperback and Hardcover Production shots of the editions. Not for sale yet — this page is the public working version. - ![ML for Agent Builders, kindle](https://lenvanderhof.com/books/editions/ml-for-agent-builders/en-kindle-2cb026376ebc.jpg) ### Kindle The digital edition, ready in any Kindle app. - ![ML for Agent Builders, paperback](https://lenvanderhof.com/books/editions/ml-for-agent-builders/en-paperback-eae034a9b55f.jpg) ### Paperback The working copy. Mark it up. Take it into the meeting. - ![ML for Agent Builders, hardcover](https://lenvanderhof.com/books/editions/ml-for-agent-builders/en-hardcover-5a39c4243392.jpg) ### Hardcover The shelf edition. Dark full-art jacket, built to stay in the room. The book ## Auto-Training, Fine-Tuning, and Eval Loops for LLM-Powered Systems A support agent hands every message to a frontier model to pick a queue. The call costs money, adds latency, and depends on a vendor who can change pricing or models. You can write Python and wire an API. You have not run a full training pipeline, and the "fine-tune it or add RAG" argument can end without a question anyone can test. Chapter 9 invents a quieter failure to show what an aggregate can hide. Validation accuracy looks steady near 0.89 for eight weeks while three billing disputes sit in the wrong queue for four days. The model architecture had not changed. Nor had the training data. The product had. An aggregate can pass while a class fails, and a final macro number is not evidence about the class whose failure costs the most. An owned model beside your LLM is a candidate, not an improvement, until a comparison on the same messages and acceptance criteria says otherwise. The general-purpose model can remain the right choice. Write down why. TRAIN is the author's five-discipline loop, and each leaves artifacts: What you learn ## What this book puts in your hands - The TRAIN protocol: Target metric, Represent data, Automate, Inspect, Narrow - Decide fine-tune vs RAG, control labeling and eval drift - Narrow models that win on cost, latency and control The framework ## TRAIN, step by step Target metric, Represent data, Automate, Inspect, Narrow 1. 1 ### Target metric 2. 2 ### Represent data 3. 3 ### Automate 4. 4 ### Inspect 5. 5 ### Narrow Figure 7.1: High, middle and low confidence proposals all pass human-owned From the book ## Figure 7.1: High, middle and low confidence proposals all pass human-owned ML for Agent Builders, printed page: Figure 7.1: High, middle and low confidence proposals all pass human-owned Inspect more pages↓ ## Look inside Pages from the print edition. - [The contract is a living document; any change to the label taxonomy⤢](https://lenvanderhof.com/books/previews/ml-for-agent-builders/en/approved-page-101-p101-68f19ddf385e8110e8521d45-59538e0798e87e2e0aae63e8.webp) - [Step 1: Name the candidate and its core task⤢](https://lenvanderhof.com/books/previews/ml-for-agent-builders/en/approved-page-136-p136-68f19ddf385e8110e8521d45-1e7d2af35f20b43fc04662fe.webp) - [Practice: Synthetic Data QA Checklist 6 13 135⤢](https://lenvanderhof.com/books/previews/ml-for-agent-builders/en/approved-page-157-p157-68f19ddf385e8110e8521d45-539dfceacd6019c3f7882af7.webp) - [Figure 7.1: High, middle and low confidence proposals all pass human-owned⤢](https://lenvanderhof.com/books/previews/ml-for-agent-builders/en/approved-page-166-p166-68f19ddf385e8110e8521d45-dfd7aa29455ee97cadc0e36a.webp) - [Figure 8.1: Immutable offline artifacts lead to evaluation; only authorized⤢](https://lenvanderhof.com/books/previews/ml-for-agent-builders/en/approved-page-187-p187-68f19ddf385e8110e8521d45-f08fad408223f276596046c8.webp) - [A flagged or sampled row cannot be closed by batch acceptance, so⤢](https://lenvanderhof.com/books/previews/ml-for-agent-builders/en/approved-page-218-p218-68f19ddf385e8110e8521d45-051ee3381f33fb3f64c7a846.webp) - [Confirm the approved response, including abstention or block where⤢](https://lenvanderhof.com/books/previews/ml-for-agent-builders/en/approved-page-253-p253-68f19ddf385e8110e8521d45-9772dcdfddf35e62c6c6f27c.webp) - [Practice: First owned-model scan 13⤢](https://lenvanderhof.com/books/previews/ml-for-agent-builders/en/approved-page-35-p035-68f19ddf385e8110e8521d45-3892412416cdfa7024918c3e.webp) · The contents ## Chapter by chapter 15 chapters Every chapter of ML for Agent Builders with its printed epigraph, what you can do afterwards, and the moment it is built for. 1. IntroductionIntroduction: Models your agents ownBefore you train a model, name the task that would justify maintaining it. 2. Chapter 1Build vs Buy vs PromptThe decision you keep postponing is already being made: by default, by inertia, by whoever wrote the architecture doc. 3. Chapter 2TRAIN OverviewA lab experiment runs until you get tired of it. A product loop runs until the product does not need it anymore. 4. Chapter 3Target the MetricA model judged without a metric is a compass needle with no north. It spins, eventually settles, and points nowhere in particular. 5. Chapter 4Represent the DataBefore you train, write down what each row means, where it came from and which decisions it may inform. 6. Chapter 5Fine-Tune vs RAGAsk which constraint needs relief, then compare what each path costs to satisfy it. 7. Chapter 6Synthetic Data with GuardrailsGenerated examples are candidates, not independent evidence that their labels are right. 8. Chapter 7Agent-Assisted LabelingA confident proposal is still a proposal. Decide who may turn it into a training label. 9. Chapter 8Automate the PipelineA training script you run by hand is a manual. A pipeline you run on a trigger is infrastructure. 10. Chapter 9Eval Harnesses for Agent MLAutomation without regression testing speeds up failure and lends it the confidence of a working pipeline. 11. Chapter 10Deploy Beside the LLM RouterA model that only exists in a notebook is not yet part of the agent system. Deployment is the decision that turns an experiment into infrastructure. 12. Chapter 11Inspect DriftEvery model is a photograph of the world at the moment you trained it. The question is whether the world has moved since. 13. Chapter 12Narrow Scope DisciplineWrite the model's boundary before asking what else it could do. 14. Chapter 13Case Studies, Routing, Rerank, ModerationThree hypothetical builds make the decisions concrete. Their results are not evidence for yours. 15. ConclusionConclusion: TRAIN on the calendarA checkpoint ends with a decision, an owner and a date, even when the decision is not to train. Who it is for ## Who this book was written for Three walkthroughs follow the same structure, from task to serving decision: an intent router, a cross-encoder reranker, a moderation filter. You start with an architectural preference and finish with the comparison on record. TRAIN is the author's operating proposal, not a validated intervention, and its scenarios supply no measured advantage for owned models. A defensible no-build decision is a successful use of this book. ### The reader it was written for The agent-stack engineer-founder. Seed to Series A, 1–20 people, shipping LLM agents but hitting walls on classifiers, rerankers, and small fine-tunes. Can prompt and wire APIs; does not have a full ML org. Needs owned narrow models without becoming a data scientist. ### Also a fit for The product engineer on an agent team. Responsible for eval quality, labeling throughput, and deployment next to the router — reports to a technical lead who still says "just fine-tune it." ### What you will use it on - Decide build vs buy vs prompt for the next agent dependency - Stand up a golden set and regression eval before production - Use agents to accelerate labeling without shipping ungrounded data - Deploy classifiers/rerankers beside the LLM router - Monitor drift without a dedicated ML platform team ### Probably not for you if - PhD ML researchers optimizing foundation training - Readers who only want prompt tips - Clinical or regulated ML without a safety gate Editions ## Editions and specifications Edition Formats Chapters Pages Reading time ISBN (paperback) English ML for Agent Builders In production 15 410 ± 5 hours — Dutch ML voor agent-bouwers In production 15 428 ± 5 hours — Both editions are written natively. The Dutch text is not a machine translation of the English. · Trim size: 6x9″ Frequently asked ## What readers usually want to know ### What is ML for Agent Builders about? TRAIN helps agent builders decide when owned ML, fine-tuning, retrieval, evaluation, and drift monitoring are worth the operational cost. The subtitle is: Auto-Training, Fine-Tuning, and Eval Loops for LLM-Powered Systems. ### What is the TRAIN framework? TRAIN: Target metric, Represent data, Automate, Inspect and Narrow. Target metric, Represent data, Automate, Inspect, Narrow ### Is there a Dutch edition? Yes. The Dutch edition is ML voor agent-bouwers, written as a native edition rather than a machine translation. It moves through the same production line. ### How long is ML for Agent Builders? This edition runs 15 chapters, 410 pages in print and roughly 5 hours of reading. ### Who is ML for Agent Builders for? TRAIN is the author's operating proposal, not a validated intervention, and its scenarios supply no measured advantage for owned models. A defensible no-build decision is a successful use of this book. The production system ## How this book was made Every title moves through the same gated production line: sourced research, a claim-level evidence ledger, structural review, fact-checking, red-team critique, and a bilingual final edit. AI agents do specialist work inside those gates; judgment, voice, and accountability stay human. - Claims enter an evidence ledger with a source and a confidence grade before they reach the page - English and Dutch are two native editions, not a translation of one another - Every chapter clears readability, rhythm, and style gates before it is typeset Read the system in The Agentic Author→Production filesFront PDFBack PDFFull cover PDFKindle front JPG The series ## More in this theme AI Agents for Startup StrategyThe Agentic CodebaseThe Multi-Agent OrganizationThe RAG EngineerThe Model PortfolioThe Agent Dream Team