prompt
RabbitMQ Backlog Diagnosis and Retry Strategy Review
Turn RabbitMQ queue growth into an evidence-based containment and retry plan without creating a retry storm.
Engineering guide
Choose the smallest safe execution model for email, webhooks, file processing, AI jobs, and other work that continues after an HTTP response.
Act as a senior Python platform engineer. Review the FastAPI post-response task below and decide whether it belongs in BackgroundTasks, an in-process asyncio task, a database-backed worker, or a broker-based queue. Return: (1) the task failure contract, (2) loss and duplicate impact, (3) duration and resource risks, (4) the recommended execution model and why, (5) a durable job schema if required, (6) idempotency and retry rules, (7) shutdown and deployment behavior, (8) logs, metrics, and operator actions, and (9) the smallest safe migration from the current implementation. Do not recommend distributed infrastructure unless the reliability contract requires it. Task context: [Paste the route, task code, expected duration, side effects, traffic, deployment model, and failure symptoms]
Decide what should happen if the API process exits one millisecond after returning the response. Cache warming or an analytics hint may be disposable. Payments, document conversion, imports, paid AI generation, and external synchronization usually are not. Write the tolerated loss rate, maximum duration, duplicate impact, and completion evidence before selecting a library.
FastAPI BackgroundTasks runs callable work after the response inside the same application process. It needs no broker, but it does not create durable state, survive restarts, coordinate retries across replicas, or provide an operational queue. Deployments, autoscaling, memory kills, and machine failures can interrupt work after the client received success.
Use it for short, repeatable or disposable work that does not hold scarce resources. Pass identifiers instead of large request objects, create fresh database sessions, set network timeouts, and record exceptions. Keep CPU-heavy media work and long model calls out of the web process because they compete with requests and complicate shutdown.
Persist a job or inbox row before acknowledging work that must complete. Store a stable ID, task type, validated input reference, status, attempts, next-attempt time, timestamps, and last error. Commit it with the business state that requires the task. Return 202 Accepted with a job ID when completion happens later.
Retry timeouts, connection resets, rate limits, and temporary server errors with exponential backoff and jitter. Do not retry invalid input or missing permissions unchanged. Enforce a business idempotency key at the database or destination so a crash after the side effect cannot produce a duplicate charge, email, or publication.
Workers should stop claiming jobs, finish or release the current lease, and exit within a known grace period. Leases or visibility timeouts let another worker recover abandoned work. Version payloads because queued jobs may outlive the application version that created them, and test a deployment while work is running.
Track accepted, running, succeeded, retrying, and terminally failed jobs. Measure queue age, completion latency, attempts, expired leases, dead letters, and failures by dependency. Carry the job ID through logs and provide a safe replay action that keeps the original idempotency key.
Define a task function with serializable input, add one job table, and separate acceptance from execution. Start with a database-backed worker or managed queue, then add retries and dead-letter handling around real failures. Keep BackgroundTasks for explicitly best-effort work instead of distributing every small action.
This guide turns production engineering practice into a repeatable decision process. Examples are checked for explicit inputs, observable outcomes, failure handling, and reversible actions. Validate the steps against your own traffic, data model, permissions, and recovery objectives.
Read our editorial and review standardsNot durably by itself. A retry loop inside the function still disappears when the process exits.
Use 202 Accepted when work is durably accepted but completes later, and return a job identifier.
No. A database-backed worker or managed queue may be sufficient; durability matters more than the framework name.
Assume duplicate delivery and enforce a stable business idempotency key at the side-effect boundary.
Reusable resources
prompt
Turn RabbitMQ queue growth into an evidence-based containment and retry plan without creating a retry storm.
prompt
Convert a Spring Boot exception sample into a precise investigation plan instead of a generic explanation.
workflow
Turn an alert into an evidence-based incident update, containment plan, and root-cause hypothesis.