1.7K
1.6K
+ 1
280

What is Celery?

Celery is an asynchronous task queue/job queue based on distributed message passing. It is focused on real-time operation, but supports scheduling as well.
Celery is a tool in the Message Queue category of a tech stack.
Celery is an open source tool with 23.2K GitHub stars and 4.6K GitHub forks. Here’s a link to Celery's open source repository on GitHub

Who uses Celery?

Companies
476 companies reportedly use Celery in their tech stacks, including Udemy, Robinhood, and Accenture.

Developers
982 developers on StackShare have stated that they use Celery.
Pros of Celery
99
Task queue
63
Python integration
40
Django integration
30
Scheduled Task
19
Publish/subsribe
8
Various backend broker
6
Easy to use
5
Great community
5
Workflow
4
Free
1
Dynamic
Decisions about Celery

Here are some stack decisions, common use cases and reviews by companies and developers who chose Celery in their tech stack.

Guilherme Silva
Back End Developer at Company Hero · | 4 upvotes · 23.8K views
Needs advice
on
GolangGolang
and
KafkaKafka

I'm analyzing companies with stacks similar to my company, as we are in a process of breaking the monolith for microservices. I noticed that your stack is very similar to ours, Python, Django, Celery, and so on. Analyzing the technology you use I could see the use of Go and Kafka which made me think that you also went through a similar process. So here's my question I would like to know what were some of the reasons why you adopted GO in your ecosystem?? scalability, performance?? We are looking into the possibility of starting to use GO but for that, I wanted to know why use GO instead of Python?? or why to use both together?? It may be something trivial but all experience and opinion are important to us?? Thanks.

See more
Pulkit Sapra
Needs advice
on
Amazon SQSAmazon SQSKubernetesKubernetes
and
RabbitMQRabbitMQ

Hi! I am creating a scraping system in Django, which involves long running tasks between 1 minute & 1 Day. As I am new to Message Brokers and Task Queues, I need advice on which architecture to use for my system. ( Amazon SQS, RabbitMQ, or Celery). The system should be autoscalable using Kubernetes(K8) based on the number of pending tasks in the queue.

See more
Needs advice
on
CeleryCelery
and
RabbitMQRabbitMQ

I am just a beginner at these two technologies.

Problem statement: I am getting lakh of users from the sequel server for whom I need to create caches in MongoDB by making different REST API requests.

Here these users can be treated as messages. Each REST API request is a task.

I am confused about whether I should go for RabbitMQ alone or Celery.

If I have to go with RabbitMQ, I prefer to use python with Pika module. But the challenge with Pika is, it is not thread-safe. So I am not finding a way to execute a lakh of API requests in parallel using multiple threads using Pika.

If I have to go with Celery, I don't know how I can achieve better scalability in executing these API requests in parallel.

See more
Greg Smethells
CTO and Software Architect at Medstrat · | 3 upvotes · 111.2K views
Needs advice
on
AppOpticsAppOptics
and
DatadogDatadog

We use AppOptics. I am curious what are the current leaders for APM for small companies (50 employees) that use Python, MariaDB, RabbitMQ, and Google Cloud Storage. We run both Celery and Gunicorn services. We are considering Datadog or some other deep code profiling tool that can spot I/O, DB, or other response time/request rate issues

See more
Michael Mota

Automations are what makes a CRM powerful. With Celery and RabbitMQ we've been able to make powerful automations that truly works for our clients. Such as for example, automatic daily reports, reminders for their activities, important notifications regarding their client activities and actions on the website and more.

We use Celery basically for everything that needs to be scheduled for the future, and using RabbitMQ as our Queue-broker is amazing since it fully integrates with Django and Celery storing on our database results of the tasks done so we can see if anything fails immediately.

See more

Blog Posts

GitHubPythonNode.js+47
54
72266
JavaScriptGitHubPython+42
53
21767
GitHubPythonSlack+25
7
3146
GitHubPythonDocker+24
13
16999

Celery Alternatives & Comparisons

What are some alternatives to Celery?
RabbitMQ
RabbitMQ gives your applications a common platform to send and receive messages, and your messages a safe place to live until received.
Kafka
Kafka is a distributed, partitioned, replicated commit log service. It provides the functionality of a messaging system, but with a unique design.
Airflow
Use Airflow to author workflows as directed acyclic graphs (DAGs) of tasks. The Airflow scheduler executes your tasks on an array of workers while following the specified dependencies. Rich command lines utilities makes performing complex surgeries on DAGs a snap. The rich user interface makes it easy to visualize pipelines running in production, monitor progress and troubleshoot issues when needed.
Cucumber
Cucumber is a tool that supports Behaviour-Driven Development (BDD) - a software development process that aims to enhance software quality and reduce maintenance costs.
Amazon SQS
Transmit any volume of data, at any level of throughput, without losing messages or requiring other services to be always available. With SQS, you can offload the administrative burden of operating and scaling a highly available messaging cluster, while paying a low price for only what you use.
See all alternatives

Celery's Followers
1600 developers follow Celery to keep up with related blogs and decisions.