标签：#celery

The first time we deployed Celery to production on a client project, we thought we had done everything right. We had workers running, tasks queuing, and Redis as the broker. Six weeks later, the task queue was backed up with 40,000 unprocessed jobs, the workers had silently died, nobody knew, and a batch of client invoices had not been generated for two weeks. That was four years ago. Since then we have deployed Celery on dozens of projects and we have learned what actually goes wrong — not in development, where everything works, but in production, where things fail in ways you do not anticipate. This post covers the configuration and operational patterns we now use on every Celery deployment. Why tasks fail silently (and how to stop it) The most dangerous thing about Celery is how quietly it can fail. A worker process dies, the task queue fills up, and your application keeps accepting work and sending it to a queue that nobody is processing. No exception is raised. No alert fires. Users notice eventually, or you notice when a daily report does not arrive. The fix has two parts: monitoring and task acknowledgement configuration. Task acknowledgement By default, Celery acknowledges a task (removes it from the queue) as soon as a worker picks it up, before the task runs. If the worker dies mid-task, the task is lost. # celery.py app = Celery ( ' myproject ' ) app . conf . update ( # Only acknowledge after the task completes successfully task_acks_late = True , # If a worker dies, reject the task back to the queue task_reject_on_worker_lost = True , # Limit memory — workers that leak memory will restart cleanly worker_max_memory_per_child = 200_000 , # 200MB in KB # Limit tasks per child process to prevent long-running workers # from accumulating state worker_max_tasks_per_child = 1000 , ) With task_acks_late=True , a task that is picked up by a dying worker will be requeued and picked up by another worker. The task might run twice (more on that shortly), but it will n