# Your task queue did its job. The answer was still wrong.

> A queue's contract is the exit code. A model call can return a well-formed wrong answer and exit 0. What a queue gives you, where it stops, and what we haven't built yet.

2026-10-19 · Kingsley Torlowei · [https://getpapayya.com/blog/your-task-queue-did-its-job](https://getpapayya.com/blog/your-task-queue-did-its-job)

---

import LoopVideo from "@/components/LoopVideo";

You already have a task queue. Celery, Sidekiq, BullMQ, SQS with a pool of workers behind it. It's probably good. It has been running your background jobs for years, and most nights nothing pages anyone.

So the question for an AI pipeline isn't whether you need a queue. You have one. It's whether the queue can answer the question an AI pipeline actually asks.

## What a queue gets right

Be fair to the thing first. A decent task queue gives you:

- **A task you define once and enqueue many times.** `@app.task`, then `.delay(args)`.
- **Workers you can add.** Throughput is the size of the pool.
- **Retries.** A dropped socket or a provider blip is tried again without anyone looking.
- **Timeouts, rate limits and concurrency caps**, so one runaway job can't take the pool with it.
- **Schedules.** Run this every night at two.
- **A dead-letter queue** for the jobs that ran out of retries.
- **A UI** that shows what is queued, running, done and dead.

That's a real list, and it took the industry fifteen years to make it boring. Anything that replaces a queue has to do all of it.

## Where a queue stops

Here is one night of a ticket-tagging job. Each ticket goes to a model, which returns a tag. The queue runs a thousand of them.

<LoopVideo
  className="my-10 md:-mx-28"
  src="/video/hero-conveyor.mp4"
  poster="/video/hero-conveyor-poster.jpg"
  width={1920}
  height={560}
  label="A task queue: eight tickets pass through a worker and land in a done tray, each stamped exit 0. Ticket 8357 returned an empty object and was stamped exit 0 like the others. The same eight tickets run again with a grade step after the worker, and ticket 8357 is marked degraded."
/>

Ticket 8357 goes into the worker, the model call returns `{}`, the function returns `{}`, and the process exits 0. The queue acks it. Into the done column it goes.

The queue did nothing wrong. Its contract is: did the function return without raising? It did. Whether `{}` is a sensible tag for a support ticket is not a question a queue can ask, because a queue never sees what came back. It sees an exit code.

In a normal background job, that's fine. A function that resizes an image either raises or produced an image. A model call is different. It can return a perfectly well-formed wrong answer, and nothing raises: an empty object, a `null`, a tag the schema doesn't allow, an answer to a different question. Those are the failures an AI pipeline actually has, and every one of them exits 0.

So you find out the way everyone finds out: a customer notices, a week later.

## Then you have to fix it

Say you found it. Ticket 8357, tagged with nothing. Now what does the queue give you?

**A retry, if you still have the arguments.** The queue held the job's arguments until it acked, then let go of them. Unless you saved the input somewhere yourself, the thing you need to reproduce the failure is gone.

**The job, and nothing smaller.** For a ticket that's fine: one ticket, one job. But AI work is rarely one call. A 150-page document is one job with 150 pages inside it. When page 73 comes back blank, the smallest thing the queue can retry is the document, so all 150 pages run again. That's 300 page executions to fix one, and the 149 pages that were fine get written downstream a second time.

**No way to know if the retry worked.** It exits 0 again. That told you nothing the first time.

## The same queue, graded

Papayya does the list at the top. A task is a function with `@agent` on it. Work goes in as a run of items. Workers lease from a pool. Retries are on by default: five attempts, waiting 1, 2, 4 and 8 seconds. Timeouts, per-tenant concurrency and rate limits are on the decorator, and schedules, a dead-letter inbox and a dashboard are there too:

```python
import papayya
from papayya import agent

@agent(name="tag_tickets", concurrency_per_key=4, rate_limit="60/min")
def tag_tickets(run, ticket: dict) -> dict:
    reply = run.step("model", classify, item_id=ticket["id"])(ticket["body"])
    if not reply.get("tag"):
        papayya.mark_degraded("model returned no tag")
    return reply
```

```bash
papayya runs submit --agent $AGENT_ID --file tickets.jsonl
```

The difference is what happens after the exit code. Every item gets a verdict, independent of whether it ran: `completed` and `degraded` are two columns, not one word. Ticket 8357 still exits 0. It also lands in the work list as degraded, with the reason, before any customer sees it.

The item keeps the input that produced it, so reproducing it is a lookup, not an archaeology project. Each `run.step` is checkpointed when it returns, so re-driving the item reuses every step that already worked and re-runs only the one that broke. That's page 73 alone, not the document: one page execution instead of three hundred. On a measured 40-page document, re-driving from page 17 took [60.3 seconds against 100.6 from scratch](/blog/your-agent-isnt-slow-because-of-the-cpu).

And after a fix, you can check it before you trust it. Pull the items that went wrong, verify your fix against them locally, then release the re-drive to production. [Recovery](https://docs.getpapayya.com/concepts/recovery) walks through it.

## What we haven't built

Some queue features we don't have yet. Said plainly, so you don't build on them:

- **Delayed jobs.** There's no "run this in ten minutes". Schedules are cron.
- **Priorities.** Items lease oldest-first, with a fair share per account so one tenant's backlog can't starve the rest. There's no marking one item as urgent.
- **Chaining and fan-out.** There's no built-in "extract every page, then merge". Today that fan-out is your code: submit the pages as items, then merge their results.
- **Completion callbacks.** Nothing calls your URL when a run finishes. You poll, or watch the dashboard.

And one honest difference in weight. Celery is `pip install` and a Redis. Papayya is a control plane, a worker pool and Postgres, and it runs in your cloud. Your inputs never leave it. That's why it's more than a pip install, and why the first install is one we do with you.

## The line

A task queue answers one question: did it run?

For an AI pipeline that's the easy one. Papayya answers the second: did it work? And when it didn't, it re-runs the one that broke, not the thousand that didn't.