Engineering guide

FastAPI BackgroundTasks vs Durable Worker Queues

Choose the smallest safe execution model for email, webhooks, file processing, AI jobs, and other work that continues after an HTTP response.

Codez Win engineering teamReviewed: 2026-08-25

Copy the working template

Act as a senior Python platform engineer. Review the FastAPI post-response task below and decide whether it belongs in BackgroundTasks, an in-process asyncio task, a database-backed worker, or a broker-based queue. Return: (1) the task failure contract, (2) loss and duplicate impact, (3) duration and resource risks, (4) the recommended execution model and why, (5) a durable job schema if required, (6) idempotency and retry rules, (7) shutdown and deployment behavior, (8) logs, metrics, and operator actions, and (9) the smallest safe migration from the current implementation. Do not recommend distributed infrastructure unless the reliability contract requires it.

Task context:
[Paste the route, task code, expected duration, side effects, traffic, deployment model, and failure symptoms]

A practical starting sequence

  1. Classify the task by loss tolerance, duration, and side effects.
  2. Use in-process tasks only when best-effort execution is acceptable.
  3. Move durable work behind a persisted job record and idempotent worker.
  4. Define retry, observability, and deployment behavior before increasing volume.

Start with the failure contract

Decide what should happen if the API process exits one millisecond after returning the response. Cache warming or an analytics hint may be disposable. Payments, document conversion, imports, paid AI generation, and external synchronization usually are not. Write the tolerated loss rate, maximum duration, duplicate impact, and completion evidence before selecting a library.

What BackgroundTasks guarantees

FastAPI BackgroundTasks runs callable work after the response inside the same application process. It needs no broker, but it does not create durable state, survive restarts, coordinate retries across replicas, or provide an operational queue. Deployments, autoscaling, memory kills, and machine failures can interrupt work after the client received success.

Keep in-process work bounded

Use it for short, repeatable or disposable work that does not hold scarce resources. Pass identifiers instead of large request objects, create fresh database sessions, set network timeouts, and record exceptions. Keep CPU-heavy media work and long model calls out of the web process because they compete with requests and complicate shutdown.

Create a durable acceptance boundary

Persist a job or inbox row before acknowledging work that must complete. Store a stable ID, task type, validated input reference, status, attempts, next-attempt time, timestamps, and last error. Commit it with the business state that requires the task. Return 202 Accepted with a job ID when completion happens later.

Retry selectively and idempotently

Retry timeouts, connection resets, rate limits, and temporary server errors with exponential backoff and jitter. Do not retry invalid input or missing permissions unchanged. Enforce a business idempotency key at the database or destination so a crash after the side effect cannot produce a duplicate charge, email, or publication.

Plan deploys and shutdowns

Workers should stop claiming jobs, finish or release the current lease, and exit within a known grace period. Leases or visibility timeouts let another worker recover abandoned work. Version payloads because queued jobs may outlive the application version that created them, and test a deployment while work is running.

Operate the queue visibly

Track accepted, running, succeeded, retrying, and terminally failed jobs. Measure queue age, completion latency, attempts, expired leases, dead letters, and failures by dependency. Carry the job ID through logs and provide a safe replay action that keeps the original idempotency key.

Migrate in small steps

Define a task function with serializable input, add one job table, and separate acceptance from execution. Start with a database-backed worker or managed queue, then add retries and dead-letter handling around real failures. Keep BackgroundTasks for explicitly best-effort work instead of distributing every small action.

How this guide was prepared

This guide turns production engineering practice into a repeatable decision process. Examples are checked for explicit inputs, observable outcomes, failure handling, and reversible actions. Validate the steps against your own traffic, data model, permissions, and recovery objectives.

Read our editorial and review standards

Frequently asked questions

Can FastAPI BackgroundTasks retry failed work?

Not durably by itself. A retry loop inside the function still disappears when the process exits.

Should queued work return 200 or 202?

Use 202 Accepted when work is durably accepted but completes later, and return a job identifier.

Do I always need Celery and Redis?

No. A database-backed worker or managed queue may be sufficient; durability matters more than the framework name.

How do I prevent duplicate execution?

Assume duplicate delivery and enforce a stable business idempotency key at the side-effect boundary.

Reusable resources

Prompts and workflows for this problem