What it means to be fault tolerant in the AI age

In June 2026, Notion pulled every Anthropic model out of its picker. The notice was careful: the models weren't down. They were degraded — "experiencing degraded performance, which is causing a higher rate of failures." So Notion disabled them across the board and rerouted to other providers. Most users barely noticed. But read the notice again and sit with what it admits: the only way they caught it was that failures had already reached users. The provider was returning 200s the whole time.

That's the failure mode nothing in your stack was built to catch. And it's worth being precise about why.

Failure used to be loud

Every fault-tolerance primitive you trust was designed around failure that announces itself. A node crashes. A request returns 500. A timeout fires. A health check goes red. Retries, replication, circuit breakers, failover — all of it assumes something stopped, and that you can see it stopped. We have spent forty years getting very good at recovering from loud failure.

The Notion incident is the clean counterexample. The provider never went down. The API returned 200s, the responses were well-formed JSON, the health checks stayed green. The fault wasn't on the wire. It was in the output — refusals, truncations, degraded answers wearing the costume of a successful call.

In the AI age, a fault is no longer "did it error?" It's "did it work?" The unit of failure moved up a layer — from the transport to the outcome — and almost no one's tooling moved with it.

Three layers of fault, and which one you're missing

It helps to separate the kinds of failure an AI workload can have:

• Transport faults — node down, 500, timeout. Loud. Solved. Your load balancer already handles these.
• Semantic faults — a 200 that's well-formed but wrong, refused, truncated, or degraded. Silent. Mostly unhandled. This is the Notion case.
• Economic faults — the recovery itself isn't free. Rerouting changes your per-token cost; a degraded model burns spend on retries that can't succeed. Invisible unless you attribute it.

Notion's stack could see the first layer. It caught the second only through customer complaints. And its lever for recovery — disable the whole provider, for everyone — ignored that the actual blast radius was a slice, and ignored the third layer entirely.

None of that is a knock on Notion. It's what the available tools let you do. Your queue knows about jobs, not outcomes. Your tracer knows a call happened, not whether it was the right call. Your framework knows the function returned, not whether it returned something that worked. Each layer is correct about the question it tracks. The question that matters — did this item, for this customer, get what it was sent to do? — falls in the seams.

What AI-age fault tolerance actually requires

Four things, none of which a status code can give you:

1. Detect at the outcome layer. Score ran vs. worked per item — a refusal, an empty retrieval, a null field, a truncation — not per HTTP status.
2. Scope by tenant and item. A degraded model doesn't take down a server. It quietly poisons a subset of outputs across many customers. The blast radius is semantic, not infrastructural.
3. Recover by reprocessing the slice. You can't retry the same call against the same degraded model. You re-route to a fallback and re-run only the affected items.
4. Make failover a decision, not a reflex. "Degraded" is a judgment. Pause and notify; let an operator make the scoped call. Notion made a blunt global one because that was the only call available.

What it looks like with the layer in place

We built a small, runnable version of the Notion incident — a support-ticket triage workload running across four tenants. It's in the examples repo and runs offline, no API key. Here's the whole story in three moves.

The workload is the code you'd already have. The only Papayya-specific lines are the decorator and wrapping the model call in a step:

from papayya import agent

@agent(name="triage-ticket")
def triage(run, ticket_id, text, partition_key):
    classify = run.llm_step("classify", call_my_model)
    resp = classify(text, ticket_id)
    return run.complete({"ticket_id": ticket_id, "label": resp["content"]})

call_my_model is your function, calling your provider with your key — Papayya is not a gateway and never touches the call. partition_key is the tenant. That's the adoption cost: two lines around code you already wrote.

Then the primary model degrades — a slice of calls come back as 200s with stop_reason="refusal" and no content. The workload code does not change, and nobody writes a refusal check. Papayya still records each one as degraded — ran, but didn't work — because the step inspects the response shape for you. What you get isn't "the provider is flaky." It's the blast radius, per tenant:

  acme      2/5 worked, 3 degraded  <- degraded slice
  globex    4/4 worked, 0 degraded
  initech   2/4 worked, 2 degraded  <- degraded slice
  umbrella  1/4 worked, 3 degraded  <- degraded slice

Eight of seventeen items ran and didn't work — and you know exactly which eight, and whose.

Then you recover the slice. Re-route to a healthy fallback — you change which model your function calls — and reprocess only the eight degraded items. The nine that already worked are never re-run. Blast radius and recovery are the same precise slice, instead of "disable everything and re-run the batch."

The line worth underlining

Here's the part that is the thesis. Papayya ships a replay for runs that genuinely failed — crashed, raised, timed out. The degraded items in this demo can't use it, because they didn't fail. They completed. The model returned, the function returned, the run was marked done. It just didn't work.

That's the whole problem in one sentence: degradation isn't failure, so every tool built for failure looks right past it. What makes scoped recovery possible at all is that the layer recorded the per-item outcome, not just the status.

That's what we mean by the workload layer. Bring your model, your keys, your framework — keep all of it. What you add is a layer that knows your run is a workload: that items belong to tenants, that outcomes are accountable, and that a 200 which didn't work is a failure you can see, scope, and reprocess — before your users are the ones who report it.

That's what fault tolerance means in the AI age. The team that catches the degradation per tenant, and reprocesses the slice, has it. The team that finds out from an angry user does not.