A model deprecation turns into a scramble when the replacement, the regression check and the switch date are still unnamed on the day the notice arrives. The provider sets the outside deadline. Your deprecation protocol sets the rest: every route that depends on the model, including fallback slots; the tests each replacement must pass; where traffic goes if it fails; and your own move date, ahead of the provider’s.
That is the distinction between a planned retirement and a surprise sunset. In a planned retirement, you know what takes over and what must pass before traffic moves. In a surprise sunset, the deadline arrives while you are still finding dependencies or guessing whether the replacement works.
Chapter 11 of The Model Portfolio treats a vendor notice as an off-cycle trigger, worth a targeted review without waiting for the quarterly calendar, and runs the retirement through six phases. The provider controls its date. The chapter’s advice is to draft the first two phases for the model with your oldest review date before any notice exists, so a real notice starts the clock with two phases done.
What should I do when an AI model I rely on is deprecated?
Open a targeted review immediately, scoped to that model’s routes and dependencies; unrelated model experiments wait for the normal calendar. Record the exact model version, the notice, the retirement deadline and one migration owner. Identify every affected route: each assignment of a particular kind of work to a model. Include fallback chains, where the model takes work only when another model fails.
The chapter sets two response targets: an impact assessment within two business days of the notice, and a replacement decision within the first third of the window the provider actually gave. These are operating targets, not guarantees that a migration can fit. In a hypothetical 30-day window, the decision belongs within the first ten days.
Distinguish choosing a candidate from approving it for production. Write the remaining tests, their owner and their deadline into the decision record.
If the agent’s whole role is ending rather than its model, use the production-agent retirement checklist instead: it covers the triggers, keys and recurring costs that need closing.
What is the difference between deprecation and sunset?
Keep the provider’s notice beside your own registry status: the notice records what the provider is changing, your registry what your system permits. A provider’s “deprecated” may still be fully routable, so record its term next to yours.
The chapter uses three internal states:
| State | Meaning | Permitted use |
|---|---|---|
| Active | Operating under its current mandate | The routes authorised by that mandate |
| Deprecated | Demoted from its current role | Only the uses allowed by the demotion reason |
| Sunset | Retired and recorded as inactive | No traffic |
The reason matters. Under the chapter’s rules, a model demoted for cost or supersession may remain a fallback while its replacement is validated. A model demoted for quality failure leaves every fallback chain. It may stay in shadow evaluation, where its outputs are assessed without serving users.
If its host is no longer permitted to process a route’s data, the model leaves that route’s fallback chain and its shadow evaluation in the same step. Shadow copies of live requests still carry the restricted data. Continue evaluation only with material the host may receive, such as appropriately classified synthetic or redacted examples.
An ordinary demotion can stop at a restricted role. A retirement notice opens the full protocol.
What belongs in the deprecation protocol?
Use the chapter’s six phases in order. Each leaves a record the next phase uses.
1. Trigger
Record what opened the protocol and when: a retirement notice, confirmed quality failure, cost escalation the mandate cannot justify, a validated cheaper replacement or a change in data eligibility. The trigger type sets the clock for everything after it.
Write the outside deadline and your internal migration date separately. The window is the deadline minus a stated buffer for failed tests and rework.
2. Impact assessment
List primary routes, fallback chains, cascades that escalate to the model, agent mandates, batch jobs and evaluation suites that use it as a comparator. Give each dependency an owner and a replacement action.
The chapter’s working rule: a registry query should answer this within an afternoon, and if it cannot, fixing the registry comes first. Then check the answer against live routing configuration and callers, because a forgotten fallback or overnight job can still send the model work.
3. Communication
Tell affected owners before traffic moves: engineering for the routing change, product for any change users will notice, finance for the cost-baseline reset and compliance for the privacy and data-processing review. The escalation path in the retiring model’s mandate names the model that absorbs its failures. Follow it to the routes that inherit the change, and tell their owners too.
One person may hold several roles in a small company. Record who was told and when.
4. Rollback
Before migration opens, define what sends traffic back: the measure, threshold, observation window and destination that takes over. A confirmed hard failure needs immediate action.
For cost-driven replacement or supersession, the book keeps the predecessor deprecated, not sunset, until the replacement has held at full traffic through a 30-day elevated-monitoring window. That is its working convention, not a universally sufficient detection window.
A predecessor that failed quality or lost data eligibility cannot serve that recovery role. Nor can an endpoint the provider removes before monitoring ends. In those cases, chapter 11 requires a second compliant host, proven before the migration window opens. Check that it stays available for the whole monitoring period.
5. Migration window
Move traffic route by route, in stages, with each route’s regression gate as the condition for the next step. Re-validate prompts for the replacement on every route instead of copying them: the chapter treats prompts as coupled to the model they were engineered against.
Attach the test results and rollout dates. Book the chapter’s one-week safety checkpoint when approving the change. If a full comparison must wait until the route holds at full traffic, record that later review date separately.
6. Tombstone record
Before marking the model sunset, remove every active routing reference, including fallbacks; update evaluation suites that use it as a comparator; and record the retirement date and reason. Stop shadow calls too: sunset means no traffic.
Archive the replacement, the actual migration cost and duration against estimates, and lessons learned. Preserve the status history so a past routing log can still be interpreted.
How does the replacement qualify for each route?
Chapter 11 requires six clearances before promotion, and every one of them blocks. Check data eligibility before sending test requests, then record the evidence for each clearance.
| Clearance | What must be established |
|---|---|
| Task quality | On the same items from the route’s task suite, the confidence interval of the paired difference stays within the quality margin the route registered in advance. |
| Cost | Cost per request is compared at the same stated workload and volume. Any increase is justified by a measured quality gain on that route. |
| Regression | The route’s registered checks pass on held-out live requests, including its zero-tolerance criteria. |
| Latency | The route’s response-time budget holds at the percentile it specifies, such as P95. |
| Data eligibility | The replacement’s host is permitted to process the route’s data. |
| Routing compatibility | The replacement works within its escalation and fallback chains, and any required redesign is tested and approved. |
These checks apply per assignment: passing extraction tests does not qualify a model for a summary job’s fallback slot, and a good offline score says nothing about response time under load.
A worked example: the forgotten fallback
Consider a hypothetical support-software company. Model A extracts fields from incoming tickets. It is also the fallback for a nightly job that summarises tickets. Its provider gives an imagined 30-day retirement window. A, B and C are placeholders, not real models or a vendor schedule.
The engineering owner finds both dependencies on the first day. The summary job normally runs on another model, so its successful nightly runs have never tested a replacement for A’s fallback role.
Model B is the candidate. Model C runs on a separate eligible host and, for this example, has already qualified for both tasks at the required load. The team still exercises recovery to C before migration. A will disappear during the monitoring period, so the plan cannot depend on returning to A.
For summaries, the team writes a hard rule before testing: every unresolved complaint must remain identifiable and marked unresolved. One confirmed omission or false closure blocks promotion.
B passes the extraction checks but produces this hypothetical summary failure:
| Item | Content |
|---|---|
| Test input | T-41: refund requested, unresolved. T-42: login restored, resolved. |
| B’s summary | Login restored for T-42. |
| Required finding | T-41 still has an unresolved refund request. |
| Decision | Fail. B omitted T-41 and cannot take the summary fallback slot. |
The team revises the summary prompt and keeps this case as a regression check. Passing the repaired example alone will not qualify B; it must also pass the separate held-out tests and the remaining route checks.
The illustrative schedule makes that decision explicit:
| When | Required record or action |
|---|---|
| First two business days | Dependencies, owners, outside deadline and communication record. |
| By day 10 | B named as candidate; C named for recovery; B’s summary failure recorded as blocking. |
| By day 15 | All six clearances and recovery tests complete for each proposed assignment. Any route B cannot qualify for goes to already qualified C. |
| Days 16–21 | Staged migration on the approved routes, with A removed from primary and fallback routing by the end of day 21. |
| During and after rollout | Run the booked checkpoint, complete A’s tombstone record and continue elevated monitoring with C available. |
The final nine days are a buffer, not permission to waive a gate. If B violates the hard summary rule during rollout, stop expanding B’s traffic and send subsequent affected work to C.
This example assumes a qualified standby. Without one, the team has an unresolved continuity problem, and writing earlier dates does not solve it.
What should be ready before the next notice?
Use the chapter’s quarterly review to compare your measured replacement lead time with each provider’s notice history. The chapter’s target is a replacement cycle of at most half the provider’s historical notice period; a cycle longer than the notice is its red flag. Past that flag, cut the work a replacement requires or keep a validated standby ready.
Read each provider’s published policy, including its exceptions, and record when you read it, who reads it next and which announcement triggers an earlier re-read.
Check what primary and fallback share: two models from the same provider go down together in a vendor-level incident. A second provider adds evaluation and operating work, so add one where an outage or retirement would hurt most. Is one AI model enough? covers that wider portfolio choice; this page asks only whether the named alternative can take the affected route.
The protocol makes preparation inspectable. The evidence reviewed here does not establish fewer outages, lower costs or a universally sufficient migration window.
Try this today
List every model your product calls by name and version, including fallbacks and batch jobs. For each, write the replacement you would use if it disappeared in 30 days and whether you have tested it.
Then choose one affected route and run a small readiness drill:
- Write the row. Record the current model and host, route, owner, proposed replacement, recovery destination, test evidence and test date. Treat an empty evidence field as untested.
- Set one hard check. State the required result before running the candidate. For the example above, unresolved ticket IDs and their status must survive the summary. Confirm the host may receive the test data.
- Replay two cases. In a test environment, use one ordinary case and one known failure case, with permitted data. Save the inputs, prompt version, outputs and pass/fail result.
- Exercise recovery. With the retiring endpoint unavailable in the test setup, trigger the proposed recovery path. Check the output and the logs: which model and host actually handled the request?
- Record the gap. Mark each check passed, failed or not run, and give the first missing check an owner and a date.
Two cases show what happened on those cases, not route quality or production readiness, so keep that evidence apart from the promotion decision. Untested means you have a surprise waiting.
Cite this:Deprecation protocol vs surprise sunset: retire AI models on your schedule.Len P. van der Hof. https://lenvanderhof.com/en/blog/model-deprecation-vs-surprise-sunset/ ·