Needs advice
on
KafkaKafkaRabbitMQRabbitMQ
and
RedisRedis

We are currently moving to a microservice architecture and are debating about the different options there are to handle communication between services. We are currently considering Kafka, Redis or RabbitMQ as a message broker. As RabbitMQ is a little bit older, we thought that it may be outdated. Is that true? Can RabbitMQ hold up to more modern tools like Redis and Kafka?

READ LESS
4 upvotes·183.1K views
Replies (4)
Senior Software Developer at Okta·
Recommends
on
Kafka
RabbitMQ
in

We have faced the same question some time ago. Before I begin, DO NOT use Redis as a message broker. It is fast and easy to set up in the beginning but it does not scale. It is not made to be reliable in scale and that is mentioned in the official docs. This analysis of our problems with Redis may help you.

We have used Kafka and RabbitMQ both in scale. We concluded that RabbitMQ is a really good general purpose message broker (for our case) and Kafka is really fast but limited in features. That’s the trade off that we understood from using it. In-fact I blogged about the trade offs between Kafka and RabbitMQ to document it. I hope it helps you in choosing the best pub-sub layer for your use case.

READ MORE
Tag: message-queue (tarunbatra.com)
10 upvotes·145K views

It depends on your requirements like number of messages to be processed per second, real time messages vs delayed, number of servers available for your cluster, whether you need streaming, etc.. Kafka works for most use cases. Not related to answer but would like to add no matter whatever broker you chose, for connecting to the broker always go for the library provided by the broker rather than Spring kafka or Spring AMQP. If you use Spring, then you will be stuck with specific Spring versions. In case you find bugs in spring then difficult because you will have to upgrade entire application to use a later Spring core version. In general, use as minimum libraries as possible to get rid of nuisance of upgrading them when they are outdated or bugs are found with them.

READ MORE
5 upvotes·3 comments·145.8K views
kaffarell
kaffarell
·
December 1st 2021 at 12:16PM

Thanks for the insight! A fast message broker would be important, persistency isn't. We also plan to deploy the message broker as a docker container to our cluster. I read somewhere online that kafka is not meant to be deployed as a container... Is that true? (What also confused me is that there isn't a official docker image for kafka).

·
Reply
Makarand Joshi
Makarand Joshi
·
December 2nd 2021 at 11:03AM

Radis is bit different compared to Rabbit MQand Kafka so use Redis only if its for non critical message flow. Between Rabbit MQ and Kafka , our experience as been for large message processing application Rabbit becomes really unstable and even have encountered corrupt data so we switched to Kafka which is more reliable

·
Reply
Bipul Agarwal
Bipul Agarwal
·
December 16th 2021 at 8:37PM

Hi Makarand, how easy was your journey from RabbitMQ to Kafka? Is it okay to ask if you have had any specific challenges as RabbitMQ sends to consumers while in Kafka Consumer needs to read. + Message structure is a bit different too?

Also, It would be nice to know if you migrated to Managed Kafka service or self hosted? (I am trying to understand how tough would it be to manage our own Kafka as we are almost finalising going with Kafka) :)

Thanks

·
Reply
View all (4)
Avatar of Tarun Batra

Tarun Batra

Senior Software Developer at Okta