We are currently moving to a microservice architecture and are debating about the different options there are to handle communication between services. We are currently considering Kafka, Redis or RabbitMQ as a message broker. As RabbitMQ is a little bit older, we thought that it may be outdated. Is that true? Can RabbitMQ hold up to more modern tools like Redis and Kafka?
I would not recommend RabbitMQ. It does not scale well.
I would recommend Redis if you want non-durable, fast, distributed, and scalable messaging. Note it has almost no guarantees around at least once delivery and similar so you will have to handle that.
I would recommend Kafka if you want the whole deal with a bit more bloat. It is durable, fast, distributed and scalable. It has few downsides other than a learning curve and higher cost.
I currently use Redis as my message queue and plan to upgrade to Kafka in Q1
We have faced the same question some time ago. Before I begin, DO NOT use Redis as a message broker. It is fast and easy to set up in the beginning but it does not scale. It is not made to be reliable in scale and that is mentioned in the official docs. This analysis of our problems with Redis may help you.
We have used Kafka and RabbitMQ both in scale. We concluded that RabbitMQ is a really good general purpose message broker (for our case) and Kafka is really fast but limited in features. That’s the trade off that we understood from using it. In-fact I blogged about the trade offs between Kafka and RabbitMQ to document it. I hope it helps you in choosing the best pub-sub layer for your use case.
