Need advice about which tool to choose?Ask the StackShare community!

RabbitMQ

17.8K
15.6K
+ 1
521
Redis

50.9K
38.5K
+ 1
3.9K
Add tool

RabbitMQ vs Redis: What are the differences?

Developers describe RabbitMQ as "A messaging broker - an intermediary for messaging". RabbitMQ gives your applications a common platform to send and receive messages, and your messages a safe place to live until received. On the other hand, Redis is detailed as "An in-memory database that persists on disk". Redis is an open source, BSD licensed, advanced key-value store. It is often referred to as a data structure server since keys can contain strings, hashes, lists, sets and sorted sets.

RabbitMQ and Redis are primarily classified as "Message Queue" and "In-Memory Databases" tools respectively.

"It's fast and it works with good metrics/monitoring" is the primary reason why developers consider RabbitMQ over the competitors, whereas "Performance" was stated as the key factor in picking Redis.

RabbitMQ and Redis are both open source tools. Redis with 37.4K GitHub stars and 14.4K forks on GitHub appears to be more popular than RabbitMQ with 5.95K GitHub stars and 1.78K GitHub forks.

According to the StackShare community, Redis has a broader approval, being mentioned in 3261 company stacks & 1782 developers stacks; compared to RabbitMQ, which is listed in 940 company stacks and 548 developer stacks.

Advice on RabbitMQ and Redis
Pulkit Sapra
Needs advice
on
Amazon SQSAmazon SQSKubernetesKubernetes
and
RabbitMQRabbitMQ

Hi! I am creating a scraping system in Django, which involves long running tasks between 1 minute & 1 Day. As I am new to Message Brokers and Task Queues, I need advice on which architecture to use for my system. ( Amazon SQS, RabbitMQ, or Celery). The system should be autoscalable using Kubernetes(K8) based on the number of pending tasks in the queue.

See more
Replies (1)
Anis Zehani
Recommends
KafkaKafka

Hello, i highly recommend Apache Kafka, to me it's the best. You can deploy it in cluster mode inside K8S, thus you can have a Highly available system (also auto scalable).

Good luck

See more
Needs advice
on
CeleryCelery
and
RabbitMQRabbitMQ

I am just a beginner at these two technologies.

Problem statement: I am getting lakh of users from the sequel server for whom I need to create caches in MongoDB by making different REST API requests.

Here these users can be treated as messages. Each REST API request is a task.

I am confused about whether I should go for RabbitMQ alone or Celery.

If I have to go with RabbitMQ, I prefer to use python with Pika module. But the challenge with Pika is, it is not thread-safe. So I am not finding a way to execute a lakh of API requests in parallel using multiple threads using Pika.

If I have to go with Celery, I don't know how I can achieve better scalability in executing these API requests in parallel.

See more
Replies (1)
Recommends
rqrqRedisRedis

For large amounts of small tasks and caches I have had good luck with Redis and RQ. I have not personally used celery but I am fairly sure it would scale well, and I have not used RabbitMQ for anything besides communication between services. If you prefer python my suggestions should feel comfortable.

Sorry I do not have a more information

See more
Meili Triantafyllidi
Software engineer at Digital Science · | 6 upvotes · 294.9K views
Needs advice
on
Amazon SQSAmazon SQSRabbitMQRabbitMQ
and
ZeroMQZeroMQ

Hi, we are in a ZMQ set up in a push/pull pattern, and we currently start to have more traffic and cases that the service is unavailable or stuck. We want to: * Not loose messages in services outages * Safely restart service without losing messages (ZeroMQ seems to need to close the socket in the receiver before restart manually)

Do you have experience with this setup with ZeroMQ? Would you suggest RabbitMQ or Amazon SQS (we are in AWS setup) instead? Something else?

Thank you for your time

See more
Replies (2)
Shishir Pandey
Recommends
RabbitMQRabbitMQ

ZeroMQ is fast but you need to build build reliability yourself. There are a number of patterns described in the zeromq guide. I have used RabbitMQ before which gives lot of functionality out of the box, you can probably use the worker queues example from the tutorial, it can also persists messages in the queue.

I haven't used Amazon SQS before. Another tool you could use is Kafka.

See more
Kevin Deyne
Principal Software Engineer at Accurate Background · | 5 upvotes · 97.1K views
Recommends
RabbitMQRabbitMQ

Both would do the trick, but there are some nuances. We work with both.

From the sound of it, your main focus is "not losing messages". In that case, I would go with RabbitMQ with a high availability policy (ha-mode=all) and a main/retry/error queue pattern.

Push messages to an exchange, which sends them to the main queue. If an error occurs, push the errored out message to the retry exchange, which forwards it to the retry queue. Give the retry queue a x-message-ttl and set the main exchange as a dead-letter-exchange. If your message has been retried several times, push it to the error exchange, where the message can remain until someone has time to look at it.

This is a very useful and resilient pattern that allows you to never lose messages. With the high availability policy, you make sure that if one of your rabbitmq nodes dies, another can take over and messages are already mirrored to it.

This is not really possible with SQS, because SQS is a lot more focused on throughput and scaling. Combined with SNS it can do interesting things like deduplication of messages and such. That said, one thing core to its design is that messages have a maximum retention time. The idea is that a message that has stayed in an SQS queue for a while serves no more purpose after a while, so it gets removed - so as to not block up any listener resources for a long time. You can also set up a DLQ here, but these similarly do not hold onto messages forever. Since you seem to depend on messages surviving at all cost, I would suggest that the scaling/throughput benefit of SQS does not outweigh the difference in approach to messages there.

See more
André Almeida
Technology Manager at GS1 Portugal - Codipor · | 5 upvotes · 273K views
Needs advice
on
Azure Service BusAzure Service Bus
and
RabbitMQRabbitMQ

Hello dear developers, our company is starting a new project for a new Web App, and we are currently designing the Architecture (we will be using .NET Core). We want to embark on something new, so we are thinking about migrating from a monolithic perspective to a microservices perspective. We wish to containerize those microservices and make them independent from each other. Is it the best way for microservices to communicate with each other via ESB, or is there a new way of doing this? Maybe complementing with an API Gateway? Can you recommend something else different than the two tools I provided?

We want something good for Cost/Benefit; performance should be high too (but not the primary constraint).

Thank you very much in advance :)

See more
Replies (2)

A Pro of Azure Service Bus is reliability and persistence: you can send message when receiver is offline; receiver can read it when it back online. A Cons is costs and message size. You can consider also SignalR

See more
Recommends

There are many different messaging frameworks available for IPC use. It's not really a question of how "new" the technology is, but what you need it to do. Azure Service Bus can be a great service to use, but it can also take a lot of effort to administrate and maintain that can make it costly to use unless you need the more advanced features it offers for routing, sequencing, delivery, etc. I would recommend checking out this link to get a basic idea of different messaging architectures. These only cover Azure services, but there are many other solutions that use similar architectural models.

https://docs.microsoft.com/en-us/azure/event-grid/compare-messaging-services

See more
Needs advice
on
KafkaKafkaRabbitMQRabbitMQ
and
RedisRedis

We are going to develop a microservices-based application. It consists of AngularJS, ASP.NET Core, and MSSQL.

We have 3 types of microservices. Emailservice, Filemanagementservice, Filevalidationservice

I am a beginner in microservices. But I have read about RabbitMQ, but come to know that there are Redis and Kafka also in the market. So, I want to know which is best.

See more
Replies (4)
Maheedhar Aluri
Recommends
KafkaKafka

Kafka is an Enterprise Messaging Framework whereas Redis is an Enterprise Cache Broker, in-memory database and high performance database.Both are having their own advantages, but they are different in usage and implementation. Now if you are creating microservices check the user consumption volumes, its generating logs, scalability, systems to be integrated and so on. I feel for your scenario initially you can go with KAFKA bu as the throughput, consumption and other factors are scaling then gradually you can add Redis accordingly.

See more
Recommends
Angular 2Angular 2

I first recommend that you choose Angular over AngularJS if you are starting something new. AngularJs is no longer getting enhancements, but perhaps you meant Angular. Regarding microservices, I recommend considering microservices when you have different development teams for each service that may want to use different programming languages and backend data stores. If it is all the same team, same code language, and same data store I would not use microservices. I might use a message queue, in which case RabbitMQ is a good one. But you may also be able to simply write your own in which you write a record in a table in MSSQL and one of your services reads the record from the table and processes it. The most challenging part of doing it yourself is writing a service that does a good job of reading the queue without reading the same message multiple times or missing a message; and that is where RabbitMQ can help.

See more
Recommends
NATSNATS

We found that the CNCF landscape is a good advisor when working going into the cloud / microservices space: https://landscape.cncf.io/fullscreen=yes. When choosing a technology one important criteria to me is if it is cloud native or not. Neither Redis, RabbitMQ nor Kafka is cloud native. The try to adapt but will be replaced eventually with technologies that are cloud native.

We have gone with NATS and have never looked back. We haven't spend a single minute on server maintainance in the last year and the setup of a cluster is way too easy. With the new features NATS incorporates now (and the ones still on the roadmap) it is already and will be sooo much mure than Redis, RabbitMQ and Kafka are. It can replace service discovery, load balancing, global multiclusters and failover, etc, etc.

Your thought might be: But I don't need all of that! Well, at the same time it is much more leightweight than Redis, RabbitMQ and especially Kafka.

See more
Amit Mor
Software Architect at Payoneer · | 3 upvotes · 523.3K views
Recommends
KafkaKafka

I think something is missing here and you should consider answering it to yourself. You are building a couple of services. Why are you considering event-sourcing architecture using Message Brokers such as the above? Won't a simple REST service based arch suffice? Read about CQRS and the problems it entails (state vs command impedance for example). Do you need Pub/Sub or Push/Pull? Is queuing of messages enough or would you need querying or filtering of messages before consumption? Also, someone would have to manage these brokers (unless using managed, cloud provider based solution), automate their deployment, someone would need to take care of backups, clustering if needed, disaster recovery, etc. I have a good past experience in terms of manageability/devops of the above options with Kafka and Redis, not so much with RabbitMQ. Both are very performant. But also note that Redis is not a pure message broker (at time of writing) but more of a general purpose in-memory key-value store. Kafka nowadays is much more than a distributed message broker. Long story short. In my taste, you should go with a minialistic approach and try to avoid either of them if you can, especially if your architecture does not fall nicely into event sourcing. If not I'd examine Kafka. If you need more capabilities than I'd consider Redis and use it for all sorts of other things such as a cache.

See more
Needs advice
on
RabbitMQRabbitMQ
and
RedisRedis

Hello there, We're developing a team chat application which would consist of direct (one-to-one) conversations and channel (group) conversations. I'm not the developer (of course), but my team suggested to go with Redis.

I've seen tech stacks of BIG team chat applications like Slack and Flock...but they haven't used RabbitMQ and used Redis instead.

A quick question, what's a good choice to go with for RabbitMQ or Redis for a message queue system in our case?

See more
Replies (2)
Recommends
RabbitMQRabbitMQ

This is of course determined by the needs of your application. It is important how many of your estimated instant users in your application will be. Also, the features of the application will affect the architecture of the application. For example, if the message data would be processed on the server, I would prefer a distributed server solution such as akka actor with the rabbitmq cluster. I would definitely use Redis. Both technologies are incomparable lanes. Redis is a database and its purpose is to process data from a different memory with the memory used by the code running on the server. Rabbit is a messaging queue system. It contributes to the architecture in a different dimension. Performance and stability are keywords.

See more
Andres Paredes
Lead Senior Software Engineer at InTouch Technology · | 3 upvotes · 15.9K views
Recommends
RabbitMQRabbitMQ

Each tool supports different use cases. RabbitMQ is a middleware peace supporting message driven reqs like you are trying to accomplish and Redis, on the other hand, allows to store data if performance, cache is important. If we are taking about a message queue system approach you could use RabbitMQ, Amazon SNS/SQS or Apache Kafka

See more
Needs advice
on
KafkaKafka
and
RabbitMQRabbitMQ

Our backend application is sending some external messages to a third party application at the end of each backend (CRUD) API call (from UI) and these external messages take too much extra time (message building, processing, then sent to the third party and log success/failure), UI application has no concern to these extra third party messages.

So currently we are sending these third party messages by creating a new child thread at end of each REST API call so UI application doesn't wait for these extra third party API calls.

I want to integrate Apache Kafka for these extra third party API calls, so I can also retry on failover third party API calls in a queue(currently third party messages are sending from multiple threads at the same time which uses too much processing and resources) and logging, etc.

Question 1: Is this a use case of a message broker?

Question 2: If it is then Kafka vs RabitMQ which is the better?

See more
Replies (4)
Tarun Batra
Senior Software Developer at Okta · | 7 upvotes · 476.3K views
Recommends
RabbitMQRabbitMQ

RabbitMQ is great for queuing and retrying. You can send the requests to your backend which will further queue these requests in RabbitMQ (or Kafka, too). The consumer on the other end can take care of processing . For a detailed analysis, check this blog about choosing between Kafka and RabbitMQ.

See more
Trevor Rydalch
Software Engineer at InsideSales.com · | 6 upvotes · 476.2K views
Recommends
RabbitMQRabbitMQ

Well, first off, it's good practice to do as little non-UI work on the foreground thread as possible, regardless of whether the requests take a long time. You don't want the UI thread blocked.

This sounds like a good use case for RabbitMQ. Primarily because you don't need each message processed by more than one consumer. If you wanted to process a single message more than once (say for different purposes), then Apache Kafka would be a much better fit as you can have multiple consumer groups consuming from the same topics independently.

Have your API publish messages containing the data necessary for the third-party request to a Rabbit queue and have consumers reading off there. If it fails, you can either retry immediately, or publish to a deadletter queue where you can reprocess them whenever you want (shovel them back into the regular queue).

See more
Guillaume Maka
Full Stack Web Developer · | 2 upvotes · 475.6K views
Recommends
RabbitMQRabbitMQ

As far as I understand, Kafka is a like a persisted event state manager where you can plugin various source of data and transform/query them as event via a stream API. Regarding your use case I will consider using RabbitMQ if your intent is to implement service inter-communication kind of thing. RabbitMQ is a good choice for one-one publisher/subscriber (or consumer) and I think you can also have multiple consumers by configuring a fanout exchange. RabbitMQ provide also message retries, message cancellation, durable queue, message requeue, message ACK....

See more
Recommends
RabbitMQRabbitMQ

In my opinion RabbitMQ fits better in your case because you don’t have order in queue. You can process your messages in any order. You don’t need to store the data what you sent. Kafka is a persistent storage like the blockchain. RabbitMQ is a message broker. Kafka is not a good solution for the system with confirmations of the messages delivery.

See more
Needs advice
on
KafkaKafkaRabbitMQRabbitMQ
and
RedisRedis

Hello! [Client sends live video frames -> Server computes and responds the result] Web clients send video frames from their webcam then on the back we need to run them through some algorithm and send the result back as a response. Since everything will need to work in a live mode, we want something fast and also suitable for our case (as everyone needs). Currently, we are considering RabbitMQ for the purpose, but recently I have noticed that there is Redis and Kafka too. Could you please help us choose among them or anything more suitable beyond these guys. I think something similar to our product would be people using their webcam to get Snapchat masks on their faces, and the calculated face points are responded on from the server, then the client-side draw the mask on the user's face. I hope this helps. Thank you!

See more
Replies (3)
Jordi Martínez
Senior software architect at Bootloader · | 3 upvotes · 426.4K views
Recommends
KafkaKafka

For your use case, the tool that fits more is definitely Kafka. RabbitMQ was not invented to handle data streams, but messages. Plenty of them, of course, but individual messages. Redis is an in-memory database, which is what makes it so fast. Redis recently included features to handle data stream, but it cannot best Kafka on this, or at least not yet. Kafka is not also super fast, it also provides lots of features to help create software to handle those streams.

See more
Recommends
RabbitMQRabbitMQ

For this kind of use case I would recommend either RabbitMQ or Kafka depending on the needs for scaling, redundancy and how you want to design it.

Kafka's true value comes into play when you need to distribute the streaming load over lot's of resources. If you were passing the video frames directly into the queue then you'd probably want to go with Kafka however if you can just pass a pointer to the frames then RabbitMQ should be fine and will be much simpler to run.

Bear in mind too that Kafka is a persistent log, not just a message bus so any data you feed into it is kept available until it expires (which is configurable). This can be useful if you have multiple clients reading from the queue with their own lifecycle but in your case it doesn't sound like that would be necessary. You could also use a RabbitMQ fanout exchange if you need that in the future.

See more
Recommends
RabbitMQRabbitMQ

I've used all of them and Kafka is hard to set up and maintain. Mostly is a Java dinosaur that you can set up and. I've used it with Storm but that is another big dinosaur. Redis is mostly for caching. The queue mechanism is not very scalable for multiple processors. Depending on the speed you need to implement on the reliability I would use RabbitMQ. You can store the frames(if they are too big) somewhere else and just have a link to them. Moving data through any of these will increase cost of transportation. With Rabbit, you can always have multiple consumers and check for redundancy. Hope it clears out your thoughts!

See more
Decisions about RabbitMQ and Redis
Mickael Alliel
DevOps Engineer at Rookout · | 4 upvotes · 319.9K views

In addition to being a lot cheaper, Google Cloud Pub/Sub allowed us to not worry about maintaining any more infrastructure that needed.

We moved from a self-hosted RabbitMQ over to CloudAMQP and decided that since we use GCP anyway, why not try their managed PubSub?

It is one of the better decisions that we made, and we can just focus about building more important stuff!

See more
Colin Chartier
Founder at Distributed Containers Inc. · | 4 upvotes · 27.6K views

We needed a centralized "job" processor for our CI runs, but continuously had issues with transactions across services:

INSERT INTO ci_jobs(...) VALUES (...) RETURNING id

redis-cli LPUSH $id

wasn't good enough, since a temporary inability to connect to redis would kill the run in a strange way.

Instead, I used postgres itself as the job server with PUBLISH / SUBSCRIBE and an atomic claiming mechanism using FOR UPDATE SKIP LOCKED using Postgres.

See the blog post below for more details:

See more
Get Advice from developers at your company using StackShare Enterprise. Sign up for StackShare Enterprise.
Learn More
Pros of RabbitMQ
Pros of Redis
  • 232
    It's fast and it works with good metrics/monitoring
  • 79
    Ease of configuration
  • 58
    I like the admin interface
  • 50
    Easy to set-up and start with
  • 20
    Durable
  • 18
    Intuitive work through python
  • 18
    Standard protocols
  • 10
    Written primarily in Erlang
  • 8
    Simply superb
  • 6
    Completeness of messaging patterns
  • 3
    Scales to 1 million messages per second
  • 3
    Reliable
  • 2
    Distributed
  • 2
    Supports AMQP
  • 2
    Better than most traditional queue based message broker
  • 1
    Inubit Integration
  • 1
    Delayed messages
  • 1
    Supports MQTT
  • 1
    Runs on Open Telecom Platform
  • 1
    High performance
  • 1
    Reliability
  • 1
    Clusterable
  • 1
    Clear documentation with different scripting language
  • 1
    Great ui
  • 1
    Better routing system
  • 880
    Performance
  • 537
    Super fast
  • 510
    Ease of use
  • 441
    In-memory cache
  • 321
    Advanced key-value cache
  • 190
    Open source
  • 179
    Easy to deploy
  • 163
    Stable
  • 152
    Free
  • 120
    Fast
  • 40
    High-Performance
  • 39
    High Availability
  • 34
    Data Structures
  • 32
    Very Scalable
  • 23
    Replication
  • 20
    Pub/Sub
  • 20
    Great community
  • 17
    "NoSQL" key-value data store
  • 14
    Hashes
  • 12
    Sets
  • 10
    Sorted Sets
  • 9
    Lists
  • 8
    BSD licensed
  • 8
    NoSQL
  • 7
    Integrates super easy with Sidekiq for Rails background
  • 7
    Async replication
  • 7
    Bitmaps
  • 6
    Keys with a limited time-to-live
  • 6
    Open Source
  • 5
    Strings
  • 5
    Lua scripting
  • 4
    Hyperloglogs
  • 4
    Awesomeness for Free!
  • 3
    Transactions
  • 3
    Runs server side LUA
  • 3
    outstanding performance
  • 3
    Networked
  • 3
    LRU eviction of keys
  • 3
    Written in ANSI C
  • 3
    Feature Rich
  • 2
    Performance & ease of use
  • 2
    Data structure server
  • 1
    Simple
  • 1
    Channels concept
  • 1
    Scalable
  • 1
    Temporarily kept on disk
  • 1
    Dont save data if no subscribers are found
  • 1
    Automatic failover
  • 1
    Easy to use
  • 1
    Existing Laravel Integration
  • 1
    Object [key/value] size each 500 MB

Sign up to add or upvote prosMake informed product decisions

Cons of RabbitMQ
Cons of Redis
  • 9
    Too complicated cluster/HA config and management
  • 6
    Needs Erlang runtime. Need ops good with Erlang runtime
  • 5
    Configuration must be done first, not by your code
  • 4
    Slow
  • 14
    Cannot query objects directly
  • 2
    No secondary indexes for non-numeric data types
  • 1
    No WAL

Sign up to add or upvote consMake informed product decisions

- No public GitHub repository available -

What is RabbitMQ?

RabbitMQ gives your applications a common platform to send and receive messages, and your messages a safe place to live until received.

What is Redis?

Redis is an open source (BSD licensed), in-memory data structure store, used as a database, cache, and message broker. Redis provides data structures such as strings, hashes, lists, sets, sorted sets with range queries, bitmaps, hyperloglogs, geospatial indexes, and streams.

Need advice about which tool to choose?Ask the StackShare community!

Jobs that mention RabbitMQ and Redis as a desired skillset
What companies use RabbitMQ?
What companies use Redis?
See which teams inside your own company are using RabbitMQ or Redis.
Sign up for StackShare EnterpriseLearn More

Sign up to get full access to all the companiesMake informed product decisions

What tools integrate with RabbitMQ?
What tools integrate with Redis?

Sign up to get full access to all the tool integrationsMake informed product decisions

Blog Posts

Nov 20 2019 at 3:38AM

OneSignal

PostgreSQLRedisRuby+8
8
4234
Jun 6 2019 at 5:11PM

AppSignal

RedisRubyKafka+9
15
1268
GitHubDockerReact+17
35
33329
What are some alternatives to RabbitMQ and Redis?
Kafka
Kafka is a distributed, partitioned, replicated commit log service. It provides the functionality of a messaging system, but with a unique design.
ActiveMQ
Apache ActiveMQ is fast, supports many Cross Language Clients and Protocols, comes with easy to use Enterprise Integration Patterns and many advanced features while fully supporting JMS 1.1 and J2EE 1.4. Apache ActiveMQ is released under the Apache 2.0 License.
ZeroMQ
The 0MQ lightweight messaging kernel is a library which extends the standard socket interfaces with features traditionally provided by specialised messaging middleware products. 0MQ sockets provide an abstraction of asynchronous message queues, multiple messaging patterns, message filtering (subscriptions), seamless access to multiple transport protocols and more.
Amazon SNS
Amazon Simple Notification Service makes it simple and cost-effective to push to mobile devices such as iPhone, iPad, Android, Kindle Fire, and internet connected smart devices, as well as pushing to other distributed services. Besides pushing cloud notifications directly to mobile devices, SNS can also deliver notifications by SMS text message or email, to Simple Queue Service (SQS) queues, or to any HTTP endpoint.
Gearman
Gearman allows you to do work in parallel, to load balance processing, and to call functions between languages. It can be used in a variety of applications, from high-availability web sites to the transport of database replication events.
See all alternatives