Background Task Processing with Celery & Redis: Keeping Web Apps Fast

When a user clicks "Submit" on a web application, they expect an immediate response in under 200 milliseconds.

If your backend attempts to send transactional emails, generate a 10-page PDF report, and sync records to HubSpot inside the HTTP request-response cycle, the user will experience 5-second loading spinners or timeout errors.

Asynchronous background processing with Celery and Redis is how professional engineering teams keep web applications fast.

There is a second reason beyond speed, and it is the one that matters more in production: background tasks can be retried. A network blip during a CRM sync inside a request becomes a user-visible failure; the same blip inside a task becomes a retry that nobody notices. Moving work into a queue is as much about durability as it is about latency.

The Request-Response vs Background Queue Flow

User Click ──► Django API ──► Pushes task to Redis Queue ──► Returns HTTP 200 (50ms)
                                       │
                                       ▼
                       Celery Worker (Runs task in background)
                       • Sends Email
                       • Generates PDF Report
                       • Syncs CRM Data

3 Essential Use Cases for Celery in SaaS

1. Transactional Email & SMS: Never wait for SendGrid or Twilio API response inside the user's web request.
2. Third-Party API Sync & Retries: Automatically retry failed webhook deliveries using Celery's exponential backoff (autoretry_for=(RequestException,), retry_backoff=True).
3. Scheduled Cron Jobs (Celery Beat): Automate daily subscription renewal checks, weekly metric digests, and database cleanup routines.

What Belongs in a Queue (And What Doesn't)

Not everything slow should be asynchronous. The test is whether the user needs the result to continue:

That last row is worth a note: imports and exports should report progress, which means a task ID, a status field, and a polling or WebSocket endpoint. A queue without progress feedback just moves the spinner somewhere else.

The Failure Modes Nobody Warns You About

Celery is easy to start with and subtle to run well. These are the six issues we see most often:

1. Tasks that run twice. By default, a worker that dies after receiving a task may lose it; with acks_late=True it will be redelivered, which means every task must be idempotent. Use a natural key — invoice ID, order number, webhook event ID — and check whether the work was already done before doing it.

2. Losing tasks when Redis is shared with the cache. This is the mistake we most often find in codebases. Ruby-style Redis usage defaults to an eviction policy (allkeys-lru), and if your broker shares the same instance as your cache, a memory spike can evict queued tasks. Run the broker on its own Redis instance with maxmemory-policy noeviction, and enable AOF persistence. If delivery guarantees are critical, RabbitMQ is a fair choice for the broker; Redis is excellent as a cache and good enough as a broker.

3. Retry storms. autoretry_for=(RequestException,) with exponential backoff is right, but a third-party outage can still queue thousands of retries. Add retry_jitter, cap max_retries, and consider a circuit breaker so the queue does not compound the outage.

4. Poison messages. A task that fails deterministically will retry forever and occupy a worker. Send it to a dead-letter queue after N attempts, alert, and expose a way for a human to re-run it after a fix.

5. Long tasks blocking short ones. A worker processing a 40-minute export cannot send password reset emails. Route by queue: email, reports, imports, webhooks, each with its own worker and concurrency settings. Set soft_time_limit so a runaway task cannot hold a slot indefinitely.

6. Memory growth. Long-running workers leak — especially when rendering PDFs or loading large ORMs. Set max_tasks_per_child (e.g. 200) so workers recycle.

Configuration That Actually Matters

Two more practices that pay off repeatedly: do not store results you never read (ignore_result=True), and never change a task's argument signature without a versioning plan — during a rolling deploy, old workers and new messages coexist, and a renamed argument manifests as a TypeError in production only.

Scheduling with Celery Beat

Scheduled work is where correctness quietly matters:

• Run exactly one beat scheduler. Two schedulers means every daily job runs twice. Use a single instance with a lock, or django-celery-beat with the database scheduler.
• Be explicit about timezones. A "daily 8am report" must be 8am in the customer's timezone, not the server's. Store the customer's timezone and compute the schedule from it, and be careful with DST transitions.
• Make scheduled tasks safe to run twice or to skip. Monthly invoicing should key on the billing period, not on "did the cron fire".
• Use a fan-out pattern for per-tenant work: one lightweight task that enumerates tenants and dispatches a task per tenant, so one failing customer does not block the rest.

A Word on Choosing Celery at All

Celery is the default, not the only option, and for a small application it can be more machinery than the problem needs. The alternatives worth knowing:

• Database-backed queues (Django-Q2, or a simple jobs table polled by a cron): no Redis, no extra container, and transactions you already understand. Perfectly adequate for a few thousand tasks a day.
• RQ: simpler than Celery, Redis-based, good documentation, fewer features around scheduling and routing.
• Cloud queues (SQS, Cloud Tasks, Azure Queue): managed, durable, and priced per request — worth considering if you would rather not operate a broker at all.
• Cron plus a management command: still the right answer for a nightly cleanup that has no retry requirements.

We default to Celery because routing, retries, backoff, scheduling and monitoring are all well-trodden paths, and engineers joining a project already know it. If a client's workload is genuinely small, we will say so and skip the extra infrastructure.

Testing and Debugging Tasks

Tasks deserve tests as much as views do, and they are easier to test than most teams assume: call the task function directly (synchronously) with the arguments it would receive, and assert on the resulting database state and outbound calls. Keep a small number of tests that exercise the queue itself — enqueue through the API, then run the worker inline with CELERY_TASK_ALWAYS_EAGER=True — so you catch serialisation problems, which are the ones that only appear in production.

When debugging a queue that has stopped moving, work the problem in this order:

1. Is the worker alive? Check heartbeats and container restarts before anything else.
2. Is the queue draining? A flat line with work pending means workers are either dead or blocked on one slow task. Look at task start times, not just failures.
3. Is it one task or all of them? A single failing task type points at a dependency; everything stalling points at the broker.
4. Did the deploy change a signature? Argument renames during a rolling deploy produce TypeError storms that look like data problems.
5. Is Redis out of memory? With the wrong eviction policy, tasks disappear without a trace and no error is logged anywhere.

Write that list down somewhere the on-call engineer can find it. Most queue incidents are diagnosed in minutes with a checklist and in hours without one.

Monitoring: The Part That Separates Hobby from Production

A queue that nobody watches will fail silently. Minimum viable observability:

• Queue depth per queue, alerted when it grows for more than a few minutes — the earliest sign of a stuck worker or a slow dependency.
• Task duration percentiles (p50/p95/p99) per task name, so the slow report is visible before it becomes a timeout.
• Failure rate and retry rate by task, with the exception type recorded.
• Worker heartbeat, so a dead container is noticed before the backlog builds.
• Correlation IDs passed into tasks and logged, so "this customer's invoice email never arrived" can be traced from the API request to the worker that handled it.

Flower, Prometheus with the Celery exporter, or an APM tool all work; the specific tool matters far less than having the alert.

The Sizing and Cost Conversation

Background workers are cheaper than most teams assume, but they are not free: a separate container, a Redis instance with persistence, and monitoring. For a B2B SaaS in the low thousands of users, a single 2-vCPU worker container and a small managed Redis (≈$15–$50/month) typically suffices, with concurrency tuned to the workload. The cost that is easy to underestimate is the second Redis instance once you separate broker and cache — a small price for not dropping tasks.

Related reading: Automated PDF generation is the classic queue workload; real-time WebSockets covers the other direction of async work; and DevOps and CI/CD covers deploying workers alongside the app. Build high-performance web systems with sub-100ms response times. Partner with KEHEM IT.

Requests timing out on slow jobs? Describe the workload and we will scope the background processing.