PyPIInfraupdated 05 Aug 2026

celery

Item: celery
Rating: 4
Author: Keyoor

Celery is a distributed task queue for Python: you decorate functions as tasks, call them with .delay(), and separate worker processes pick them up through a message broker (RabbitMQ and Redis are the feature-complete transports). It handles retries, periodic scheduling via celery beat, task workflows (chains, groups, chords), and scaling across machines, which is why it has been the default background-job system in the Django world for well over a decade.

celery · v5.6.3 · 13.9M/wk · ★ 28,761 · pushed 04 Aug 2026 · celery/celery · docs · reviewed by Keyoor

Verdict

Still the heavyweight standard for Python background processing; the operational overhead is the fee for its maturity, workflows, and scaling. Pick it when you need those features, and pick RQ, Huey, or arq when you just need work off the request path.

API stability4/5The task and canvas APIs have been steady across the 5.x line since 2020; the 4-to-5 CLI and config-naming changes still pollute old answers, and 5.7 will drop Python 3.9.

Docs4/5docs.celeryq.dev is deep: tutorials, a full configuration reference, and operations guides; the depth cuts both ways, since finding the one setting you need takes real digging.

Maintenance4/5Pushed the day of this review with 5.6.x current, but it is a self-described minimal-funding project with nearly 800 open issues and PRs and no Windows support.

Ecosystem4/5First-class Django and Flask paths, Flower for monitoring, experimental brokers beyond RabbitMQ/Redis, and protocol clients in Node, Go, and Rust listed by the project itself.

Use it if

You need background jobs that survive restarts and scale across multiple worker machines
You already run RabbitMQ or Redis and want mature retry, routing, and rate-limit controls on top
You need periodic jobs, task chains, or fan-out/fan-in workflows (groups and chords) beyond what a cron entry covers
You run Django or Flask, where the integration paths are well worn and heavily documented

Skip it if

Your stack is asyncio-first: Celery workers are prefork process based, and native async task execution is not part of the story; arq fits an event-loop codebase better
You want simple: Celery means a broker, usually a result backend, worker processes, and a large configuration surface; RQ or Huey covers plain job queues with far less machinery
You deploy on Windows: the project explicitly does not support it and asks you not to file Windows issues
You have one server and modest job volume: a cron job or a database-backed queue avoids operating a broker at all

Setup reality

pip install celery is the easy part. The real setup is running a broker (RabbitMQ or Redis), adding a result backend if you ever read return values, keeping worker processes alive under systemd or a supervisor, and running celery beat as yet another process if you schedule anything. Configuration naming changed to lowercase forms in the 4.x era, so old Stack Overflow answers mix two styles. Local development means a broker plus a worker running next to your app, and reproducing bugs that only appear under the prefork pool is a skill of its own.

Patterns

Define an app and a taskdefine-task

from celery import Celery

app = Celery("tasks", broker="redis://localhost:6379/0")

@app.task
def add(x, y):
    return x + y

Calling add(2, 2) runs it inline like a normal function; only add.delay(2, 2) actually sends it to the queue.

Start a workerrun-worker

celery -A tasks worker --loglevel=INFO --concurrency=4

In the 5.x CLI, -A comes before the worker subcommand; the old 'celery worker -A tasks' order from 4.x tutorials fails.

Wait for a task resultget-result

app = Celery("tasks", broker="redis://localhost:6379/0",
             backend="redis://localhost:6379/1")

result = add.delay(4, 4)
print(result.ready())
print(result.get(timeout=10))

get() needs a configured result backend, and calling it inside another task is a documented deadlock trap.

Retry automatically on specific exceptionsauto-retry

import requests

@app.task(autoretry_for=(requests.RequestException,),
          retry_backoff=True, max_retries=5)
def fetch(url):
    resp = requests.get(url, timeout=10)
    resp.raise_for_status()
    return resp.text

retry_backoff gives exponential delays with jitter applied by default, so retries do not stampede a recovering service.

Retry with control over the delaymanual-retry

@app.task(bind=True, max_retries=3)
def process(self, item_id):
    try:
        do_work(item_id)
    except TemporaryError as exc:
        raise self.retry(exc=exc, countdown=60)

bind=True makes self the task instance; self.retry raises to re-queue, so no code after it will run.

Schedule recurring jobs with beatperiodic-tasks

from celery.schedules import crontab

app.conf.beat_schedule = {
    "cleanup-every-hour": {
        "task": "tasks.cleanup",
        "schedule": crontab(minute=0),
    },
}
# run alongside workers:
#   celery -A tasks beat

beat is a separate scheduler process; run exactly one instance or every job fires once per beat you started.

Compose tasks with chain, group, and chordtask-workflows

from celery import chain, group, chord

chain(fetch.s(url), parse.s(), store.s()).delay()

callback = summarize.s()
chord(group(fetch.s(u) for u in urls))(callback)

.s() builds a signature (task plus partial args); chords require a result backend because the callback waits on every group member.

Send tasks to dedicated queuesroute-queues

app.conf.task_routes = {"tasks.heavy_*": {"queue": "heavy"}}

# dedicated worker for that queue:
#   celery -A tasks worker -Q heavy --concurrency=2

Workers consume only the queues named with -Q (default queue: celery), so a routed task with no matching worker waits forever.

Kill tasks that run too longtime-limits

from celery.exceptions import SoftTimeLimitExceeded

@app.task(soft_time_limit=60, time_limit=90)
def crunch(data_id):
    try:
        heavy_computation(data_id)
    except SoftTimeLimitExceeded:
        cleanup(data_id)

The soft limit raises inside your task so you can clean up; the hard limit kills the worker child process outright.

Stop storing results you never readskip-results

@app.task(ignore_result=True)
def send_email(to, subject, body):
    smtp_send(to, subject, body)

# or globally:
app.conf.task_ignore_result = True

Fire-and-forget tasks that store results silently fill your Redis backend; ignore_result is the cheapest Celery optimization there is.

Alternatives

Package	Registry	Pick it when
rq	PyPI	You are on Redis anyway and want a job queue you can understand end to end in an afternoon.
dramatiq	PyPI	You want Celery-style features with fewer knobs and reliability-focused defaults.
huey	PyPI	You want a tiny queue with periodic tasks for small apps, including SQLite-backed setups.
arq	PyPI	Your codebase is asyncio and you want async task execution on Redis instead of prefork workers.