Alternatives to Kafka logo

Alternatives to Kafka

ActiveMQ, RabbitMQ, Amazon Kinesis, Apache Spark, and Akka are the most popular alternatives and competitors to Kafka.
23.5K
607

What is Kafka and what are its top alternatives?

Apache Kafka is a distributed event streaming platform that is widely used for building real-time data pipelines and streaming applications. Its key features include high-throughput, fault tolerance, scalability, and support for real-time data processing. However, one limitation of Kafka is its complexity in setup and configuration, especially for beginners.

  1. RabbitMQ: RabbitMQ is a highly reliable and scalable message broker that supports multiple messaging protocols. Compared to Kafka, RabbitMQ is easier to set up and manage, but it may not be as performant for high-throughput scenarios.
  2. Apache Pulsar: Apache Pulsar is another distributed messaging and event streaming platform that offers multi-tenancy, strong consistency, and geo-replication. It provides better isolation and resource utilization compared to Kafka, but may have a steeper learning curve.
  3. NATS: NATS is a lightweight and high-performance messaging system built for cloud-native applications. It excels in simplicity and low latency, making it a good alternative to Kafka for real-time messaging use cases.
  4. Amazon Kinesis: Amazon Kinesis is a fully managed service for real-time data streaming and processing. It offers seamless integration with other AWS services and is suitable for users looking for a cloud-native alternative to Kafka.
  5. Redis Streams: Redis Streams is a feature of Redis that allows for the processing of log data in real-time. It is known for its simplicity and high-performance, but may not offer the same level of durability and scalability as Kafka.
  6. Google Cloud Pub/Sub: Google Cloud Pub/Sub is a managed messaging service that enables asynchronous communication between applications. It provides low latency and global scalability, making it a viable alternative to Kafka for Google Cloud users.
  7. Apache Flink: Apache Flink is a distributed stream processing framework that offers support for event-time processing, exactly-once semantics, and stateful computations. It can be used in conjunction with a messaging system like RabbitMQ or Pulsar as an alternative to Kafka for stream processing.
  8. ActiveMQ: Apache ActiveMQ is a popular open-source message broker that supports JMS and MQTT protocols. It provides features such as message persistence, clustering, and high availability, making it a reliable alternative to Kafka for messaging-oriented applications.
  9. Apache RocketMQ: Apache RocketMQ is a distributed messaging and streaming platform that offers low latency, high throughput, and strong consistency. It is suitable for scenarios where Kafka's durability and fault tolerance are not strictly required.
  10. Azure Event Hubs: Azure Event Hubs is a cloud-based event ingestion service that can scale elastically based on workload. It is optimized for processing large volumes of data in real-time, making it a competitive alternative to Kafka for Azure users.

Top Alternatives to Kafka

  • ActiveMQ
    ActiveMQ

    Apache ActiveMQ is fast, supports many Cross Language Clients and Protocols, comes with easy to use Enterprise Integration Patterns and many advanced features while fully supporting JMS 1.1 and J2EE 1.4. Apache ActiveMQ is released under the Apache 2.0 License. ...

  • RabbitMQ
    RabbitMQ

    RabbitMQ gives your applications a common platform to send and receive messages, and your messages a safe place to live until received. ...

  • Amazon Kinesis
    Amazon Kinesis

    Amazon Kinesis can collect and process hundreds of gigabytes of data per second from hundreds of thousands of sources, allowing you to easily write applications that process information in real-time, from sources such as web site click-streams, marketing and financial information, manufacturing instrumentation and social media, and operational logs and metering data. ...

  • Apache Spark
    Apache Spark

    Spark is a fast and general processing engine compatible with Hadoop data. It can run in Hadoop clusters through YARN or Spark's standalone mode, and it can process data in HDFS, HBase, Cassandra, Hive, and any Hadoop InputFormat. It is designed to perform both batch processing (similar to MapReduce) and new workloads like streaming, interactive queries, and machine learning. ...

  • Akka
    Akka

    Akka is a toolkit and runtime for building highly concurrent, distributed, and resilient message-driven applications on the JVM. ...

  • Apache Storm
    Apache Storm

    Apache Storm is a free and open source distributed realtime computation system. Storm makes it easy to reliably process unbounded streams of data, doing for realtime processing what Hadoop did for batch processing. Storm has many use cases: realtime analytics, online machine learning, continuous computation, distributed RPC, ETL, and more. Storm is fast: a benchmark clocked it at over a million tuples processed per second per node. It is scalable, fault-tolerant, guarantees your data will be processed, and is easy to set up and operate. ...

  • Apache Flink
    Apache Flink

    Apache Flink is an open source system for fast and versatile data analytics in clusters. Flink supports batch and streaming analytics, in one system. Analytical programs can be written in concise and elegant APIs in Java and Scala. ...

  • Redis
    Redis

    Redis is an open source (BSD licensed), in-memory data structure store, used as a database, cache, and message broker. Redis provides data structures such as strings, hashes, lists, sets, sorted sets with range queries, bitmaps, hyperloglogs, geospatial indexes, and streams. ...

Kafka alternatives & related posts

ActiveMQ logo

ActiveMQ

610
1.3K
77
A message broker written in Java together with a full JMS client
610
1.3K
+ 1
77
PROS OF ACTIVEMQ
  • 18
    Easy to use
  • 14
    Open source
  • 13
    Efficient
  • 10
    JMS compliant
  • 6
    High Availability
  • 5
    Scalable
  • 3
    Distributed Network of brokers
  • 3
    Persistence
  • 3
    Support XA (distributed transactions)
  • 1
    Docker delievery
  • 1
    Highly configurable
  • 0
    RabbitMQ
CONS OF ACTIVEMQ
  • 1
    ONLY Vertically Scalable
  • 1
    Support
  • 1
    Low resilience to exceptions and interruptions
  • 1
    Difficult to scale

related ActiveMQ posts

I want to choose Message Queue with the following features - Highly Available, Distributed, Scalable, Monitoring. I have RabbitMQ, ActiveMQ, Kafka and Apache RocketMQ in mind. But I am confused which one to choose.

See more
Naushad Warsi
software developer at klingelnberg · | 1 upvote · 785.1K views
Shared insights
on
ActiveMQActiveMQRabbitMQRabbitMQ

I use ActiveMQ because RabbitMQ have stopped giving the support for AMQP 1.0 or above version and the earlier version of AMQP doesn't give the functionality to support OAuth.

If OAuth is not required and we can go with AMQP 0.9 then i still recommend rabbitMq.

See more
RabbitMQ logo

RabbitMQ

21.3K
18.7K
557
Open source multiprotocol messaging broker
21.3K
18.7K
+ 1
557
PROS OF RABBITMQ
  • 235
    It's fast and it works with good metrics/monitoring
  • 80
    Ease of configuration
  • 60
    I like the admin interface
  • 52
    Easy to set-up and start with
  • 22
    Durable
  • 19
    Standard protocols
  • 19
    Intuitive work through python
  • 11
    Written primarily in Erlang
  • 9
    Simply superb
  • 7
    Completeness of messaging patterns
  • 4
    Reliable
  • 4
    Scales to 1 million messages per second
  • 3
    Better than most traditional queue based message broker
  • 3
    Distributed
  • 3
    Supports MQTT
  • 3
    Supports AMQP
  • 2
    Clear documentation with different scripting language
  • 2
    Better routing system
  • 2
    Inubit Integration
  • 2
    Great ui
  • 2
    High performance
  • 2
    Reliability
  • 2
    Open-source
  • 2
    Runs on Open Telecom Platform
  • 2
    Clusterable
  • 2
    Delayed messages
  • 1
    Supports Streams
  • 1
    Supports STOMP
  • 1
    Supports JMS
CONS OF RABBITMQ
  • 9
    Too complicated cluster/HA config and management
  • 6
    Needs Erlang runtime. Need ops good with Erlang runtime
  • 5
    Configuration must be done first, not by your code
  • 4
    Slow

related RabbitMQ posts

James Cunningham
Operations Engineer at Sentry · | 18 upvotes · 1.8M views
Shared insights
on
CeleryCeleryRabbitMQRabbitMQ
at

As Sentry runs throughout the day, there are about 50 different offline tasks that we execute—anything from “process this event, pretty please” to “send all of these cool people some emails.” There are some that we execute once a day and some that execute thousands per second.

Managing this variety requires a reliably high-throughput message-passing technology. We use Celery's RabbitMQ implementation, and we stumbled upon a great feature called Federation that allows us to partition our task queue across any number of RabbitMQ servers and gives us the confidence that, if any single server gets backlogged, others will pitch in and distribute some of the backlogged tasks to their consumers.

#MessageQueue

See more

Around the time of their Series A, Pinterest’s stack included Python and Django, with Tornado and Node.js as web servers. Memcached / Membase and Redis handled caching, with RabbitMQ handling queueing. Nginx, HAproxy and Varnish managed static-delivery and load-balancing, with persistent data storage handled by MySQL.

See more
Amazon Kinesis logo

Amazon Kinesis

723
600
9
Store and process terabytes of data each hour from hundreds of thousands of sources
723
600
+ 1
9
PROS OF AMAZON KINESIS
  • 9
    Scalable
CONS OF AMAZON KINESIS
  • 3
    Cost

related Amazon Kinesis posts

Praveen Mooli
Engineering Manager at Taylor and Francis · | 19 upvotes · 4M views

We are in the process of building a modern content platform to deliver our content through various channels. We decided to go with Microservices architecture as we wanted scale. Microservice architecture style is an approach to developing an application as a suite of small independently deployable services built around specific business capabilities. You can gain modularity, extensive parallelism and cost-effective scaling by deploying services across many distributed servers. Microservices modularity facilitates independent updates/deployments, and helps to avoid single point of failure, which can help prevent large-scale outages. We also decided to use Event Driven Architecture pattern which is a popular distributed asynchronous architecture pattern used to produce highly scalable applications. The event-driven architecture is made up of highly decoupled, single-purpose event processing components that asynchronously receive and process events.

To build our #Backend capabilities we decided to use the following: 1. #Microservices - Java with Spring Boot , Node.js with ExpressJS and Python with Flask 2. #Eventsourcingframework - Amazon Kinesis , Amazon Kinesis Firehose , Amazon SNS , Amazon SQS, AWS Lambda 3. #Data - Amazon RDS , Amazon DynamoDB , Amazon S3 , MongoDB Atlas

To build #Webapps we decided to use Angular 2 with RxJS

#Devops - GitHub , Travis CI , Terraform , Docker , Serverless

See more
John Kodumal

As we've evolved or added additional infrastructure to our stack, we've biased towards managed services. Most new backing stores are Amazon RDS instances now. We do use self-managed PostgreSQL with TimescaleDB for time-series data—this is made HA with the use of Patroni and Consul.

We also use managed Amazon ElastiCache instances instead of spinning up Amazon EC2 instances to run Redis workloads, as well as shifting to Amazon Kinesis instead of Kafka.

See more
Apache Spark logo

Apache Spark

3K
3.5K
140
Fast and general engine for large-scale data processing
3K
3.5K
+ 1
140
PROS OF APACHE SPARK
  • 61
    Open-source
  • 48
    Fast and Flexible
  • 8
    One platform for every big data problem
  • 8
    Great for distributed SQL like applications
  • 6
    Easy to install and to use
  • 3
    Works well for most Datascience usecases
  • 2
    Interactive Query
  • 2
    Machine learning libratimery, Streaming in real
  • 2
    In memory Computation
CONS OF APACHE SPARK
  • 4
    Speed

related Apache Spark posts

Eric Colson
Chief Algorithms Officer at Stitch Fix · | 21 upvotes · 6.1M views

The algorithms and data infrastructure at Stitch Fix is housed in #AWS. Data acquisition is split between events flowing through Kafka, and periodic snapshots of PostgreSQL DBs. We store data in an Amazon S3 based data warehouse. Apache Spark on Yarn is our tool of choice for data movement and #ETL. Because our storage layer (s3) is decoupled from our processing layer, we are able to scale our compute environment very elastically. We have several semi-permanent, autoscaling Yarn clusters running to serve our data processing needs. While the bulk of our compute infrastructure is dedicated to algorithmic processing, we also implemented Presto for adhoc queries and dashboards.

Beyond data movement and ETL, most #ML centric jobs (e.g. model training and execution) run in a similarly elastic environment as containers running Python and R code on Amazon EC2 Container Service clusters. The execution of batch jobs on top of ECS is managed by Flotilla, a service we built in house and open sourced (see https://github.com/stitchfix/flotilla-os).

At Stitch Fix, algorithmic integrations are pervasive across the business. We have dozens of data products actively integrated systems. That requires serving layer that is robust, agile, flexible, and allows for self-service. Models produced on Flotilla are packaged for deployment in production using Khan, another framework we've developed internally. Khan provides our data scientists the ability to quickly productionize those models they've developed with open source frameworks in Python 3 (e.g. PyTorch, sklearn), by automatically packaging them as Docker containers and deploying to Amazon ECS. This provides our data scientist a one-click method of getting from their algorithms to production. We then integrate those deployments into a service mesh, which allows us to A/B test various implementations in our product.

For more info:

#DataScience #DataStack #Data

See more
Patrick Sun
Software Engineer at Stitch Fix · | 10 upvotes · 54.2K views

As a frontend engineer on the Algorithms & Analytics team at Stitch Fix, I work with data scientists to develop applications and visualizations to help our internal business partners make data-driven decisions. I envisioned a platform that would assist data scientists in the data exploration process, allowing them to visually explore and rapidly iterate through their assumptions, then share their insights with others. This would align with our team's philosophy of having engineers "deploy platforms, services, abstractions, and frameworks that allow the data scientists to conceive of, develop, and deploy their ideas with autonomy", and solve the pain of data exploration.

The final product, code-named Dora, is built with React, Redux.js and Victory, backed by Elasticsearch to enable fast and iterative data exploration, and uses Apache Spark to move data from our Amazon S3 data warehouse into the Elasticsearch cluster.

See more
Akka logo

Akka

1.1K
1K
88
Build powerful concurrent & distributed applications more easily
1.1K
1K
+ 1
88
PROS OF AKKA
  • 32
    Great concurrency model
  • 17
    Fast
  • 12
    Actor Library
  • 10
    Open source
  • 7
    Resilient
  • 5
    Message driven
  • 5
    Scalable
CONS OF AKKA
  • 3
    Mixing futures with Akka tell is difficult
  • 2
    Closing of futures
  • 2
    No type safety
  • 1
    Very difficult to refactor
  • 1
    Typed actors still not stable

related Akka posts

To solve the problem of scheduling and executing arbitrary tasks in its distributed infrastructure, PagerDuty created an open-source tool called Scheduler. Scheduler is written in Scala and uses Cassandra for task persistence. It also adds Apache Kafka to handle task queuing and partitioning, with Akka to structure the library’s concurrency.

The service’s logic schedules a task by passing it to the Scheduler’s Scala API, which serializes the task metadata and enqueues it into Kafka. Scheduler then consumes the tasks, and posts them to Cassandra to prevent data loss.

See more
Shared insights
on
AkkaAkkaKafkaKafka

I decided to use Akka instead of Kafka streams because I have personal relationships at @Lightbend.

See more
Apache Storm logo

Apache Storm

203
282
25
Distributed and fault-tolerant realtime computation
203
282
+ 1
25
PROS OF APACHE STORM
  • 10
    Flexible
  • 6
    Easy setup
  • 4
    Event Processing
  • 3
    Clojure
  • 2
    Real Time
CONS OF APACHE STORM
    Be the first to leave a con

    related Apache Storm posts

    Marc Bollinger
    Infra & Data Eng Manager at Thumbtack · | 5 upvotes · 1.9M views

    Lumosity is home to the world's largest cognitive training database, a responsibility we take seriously. For most of the company's history, our analysis of user behavior and training data has been powered by an event stream--first a simple Node.js pub/sub app, then a heavyweight Ruby app with stronger durability. Both supported decent throughput and latency, but they lacked some major features supported by existing open-source alternatives: replaying existing messages (also lacking in most message queue-based solutions), scaling out many different readers for the same stream, the ability to leverage existing solutions for reading and writing, and possibly most importantly: the ability to hire someone externally who already had expertise.

    We ultimately migrated to Kafka in early- to mid-2016, citing both industry trends in companies we'd talked to with similar durability and throughput needs, the extremely strong documentation and community. We pored over Kyle Kingsbury's Jepsen post (https://aphyr.com/posts/293-jepsen-Kafka), as well as Jay Kreps' follow-up (http://blog.empathybox.com/post/62279088548/a-few-notes-on-kafka-and-jepsen), talked at length with Confluent folks and community members, and still wound up running parallel systems for quite a long time, but ultimately, we've been very, very happy. Understanding the internals and proper levers takes some commitment, but it's taken very little maintenance once configured. Since then, the Confluent Platform community has grown and grown; we've gone from doing most development using custom Scala consumers and producers to being 60/40 Kafka Streams/Connects.

    We originally looked into Storm / Heron , and we'd moved on from Redis pub/sub. Heron looks great, but we already had a programming model across services that was more akin to consuming a message consumers than required a topology of bolts, etc. Heron also had just come out while we were starting to migrate things, and the community momentum and direction of Kafka felt more substantial than the older Storm. If we were to start the process over again today, we might check out Pulsar , although the ecosystem is much younger.

    To find out more, read our 2017 engineering blog post about the migration!

    See more
    Apache Flink logo

    Apache Flink

    525
    871
    38
    Fast and reliable large-scale data processing engine
    525
    871
    + 1
    38
    PROS OF APACHE FLINK
    • 16
      Unified batch and stream processing
    • 8
      Easy to use streaming apis
    • 8
      Out-of-the box connector to kinesis,s3,hdfs
    • 4
      Open Source
    • 2
      Low latency
    CONS OF APACHE FLINK
      Be the first to leave a con

      related Apache Flink posts

      Surabhi Bhawsar
      Technical Architect at Pepcus · | 7 upvotes · 727.1K views
      Shared insights
      on
      KafkaKafkaApache FlinkApache Flink

      I need to build the Alert & Notification framework with the use of a scheduled program. We will analyze the events from the database table and filter events that are falling under a day timespan and send these event messages over email. Currently, we are using Kafka Pub/Sub for messaging. The customer wants us to move on Apache Flink, I am trying to understand how Apache Flink could be fit better for us.

      See more

      I have to build a data processing application with an Apache Beam stack and Apache Flink runner on an Amazon EMR cluster. I saw some instability with the process and EMR clusters that keep going down. Here, the Apache Beam application gets inputs from Kafka and sends the accumulative data streams to another Kafka topic. Any advice on how to make the process more stable?

      See more
      Redis logo

      Redis

      59.3K
      45.6K
      3.9K
      Open source (BSD licensed), in-memory data structure store
      59.3K
      45.6K
      + 1
      3.9K
      PROS OF REDIS
      • 886
        Performance
      • 542
        Super fast
      • 513
        Ease of use
      • 444
        In-memory cache
      • 324
        Advanced key-value cache
      • 194
        Open source
      • 182
        Easy to deploy
      • 164
        Stable
      • 155
        Free
      • 121
        Fast
      • 42
        High-Performance
      • 40
        High Availability
      • 35
        Data Structures
      • 32
        Very Scalable
      • 24
        Replication
      • 22
        Great community
      • 22
        Pub/Sub
      • 19
        "NoSQL" key-value data store
      • 16
        Hashes
      • 13
        Sets
      • 11
        Sorted Sets
      • 10
        NoSQL
      • 10
        Lists
      • 9
        Async replication
      • 9
        BSD licensed
      • 8
        Bitmaps
      • 8
        Integrates super easy with Sidekiq for Rails background
      • 7
        Keys with a limited time-to-live
      • 7
        Open Source
      • 6
        Lua scripting
      • 6
        Strings
      • 5
        Awesomeness for Free
      • 5
        Hyperloglogs
      • 4
        Transactions
      • 4
        Outstanding performance
      • 4
        Runs server side LUA
      • 4
        LRU eviction of keys
      • 4
        Feature Rich
      • 4
        Written in ANSI C
      • 4
        Networked
      • 3
        Data structure server
      • 3
        Performance & ease of use
      • 2
        Dont save data if no subscribers are found
      • 2
        Automatic failover
      • 2
        Easy to use
      • 2
        Temporarily kept on disk
      • 2
        Scalable
      • 2
        Existing Laravel Integration
      • 2
        Channels concept
      • 2
        Object [key/value] size each 500 MB
      • 2
        Simple
      CONS OF REDIS
      • 15
        Cannot query objects directly
      • 3
        No secondary indexes for non-numeric data types
      • 1
        No WAL

      related Redis posts

      Russel Werner
      Lead Engineer at StackShare · | 32 upvotes · 2.8M views

      StackShare Feed is built entirely with React, Glamorous, and Apollo. One of our objectives with the public launch of the Feed was to enable a Server-side rendered (SSR) experience for our organic search traffic. When you visit the StackShare Feed, and you aren't logged in, you are delivered the Trending feed experience. We use an in-house Node.js rendering microservice to generate this HTML. This microservice needs to run and serve requests independent of our Rails web app. Up until recently, we had a mono-repo with our Rails and React code living happily together and all served from the same web process. In order to deploy our SSR app into a Heroku environment, we needed to split out our front-end application into a separate repo in GitHub. The driving factor in this decision was mostly due to limitations imposed by Heroku specifically with how processes can't communicate with each other. A new SSR app was created in Heroku and linked directly to the frontend repo so it stays in-sync with changes.

      Related to this, we need a way to "deploy" our frontend changes to various server environments without building & releasing the entire Ruby application. We built a hybrid Amazon S3 Amazon CloudFront solution to host our Webpack bundles. A new CircleCI script builds the bundles and uploads them to S3. The final step in our rollout is to update some keys in Redis so our Rails app knows which bundles to serve. The result of these efforts were significant. Our frontend team now moves independently of our backend team, our build & release process takes only a few minutes, we are now using an edge CDN to serve JS assets, and we have pre-rendered React pages!

      #StackDecisionsLaunch #SSR #Microservices #FrontEndRepoSplit

      See more
      Simon Reymann
      Senior Fullstack Developer at QUANTUSflow Software GmbH · | 30 upvotes · 11M views

      Our whole DevOps stack consists of the following tools:

      • GitHub (incl. GitHub Pages/Markdown for Documentation, GettingStarted and HowTo's) for collaborative review and code management tool
      • Respectively Git as revision control system
      • SourceTree as Git GUI
      • Visual Studio Code as IDE
      • CircleCI for continuous integration (automatize development process)
      • Prettier / TSLint / ESLint as code linter
      • SonarQube as quality gate
      • Docker as container management (incl. Docker Compose for multi-container application management)
      • VirtualBox for operating system simulation tests
      • Kubernetes as cluster management for docker containers
      • Heroku for deploying in test environments
      • nginx as web server (preferably used as facade server in production environment)
      • SSLMate (using OpenSSL) for certificate management
      • Amazon EC2 (incl. Amazon S3) for deploying in stage (production-like) and production environments
      • PostgreSQL as preferred database system
      • Redis as preferred in-memory database/store (great for caching)

      The main reason we have chosen Kubernetes over Docker Swarm is related to the following artifacts:

      • Key features: Easy and flexible installation, Clear dashboard, Great scaling operations, Monitoring is an integral part, Great load balancing concepts, Monitors the condition and ensures compensation in the event of failure.
      • Applications: An application can be deployed using a combination of pods, deployments, and services (or micro-services).
      • Functionality: Kubernetes as a complex installation and setup process, but it not as limited as Docker Swarm.
      • Monitoring: It supports multiple versions of logging and monitoring when the services are deployed within the cluster (Elasticsearch/Kibana (ELK), Heapster/Grafana, Sysdig cloud integration).
      • Scalability: All-in-one framework for distributed systems.
      • Other Benefits: Kubernetes is backed by the Cloud Native Computing Foundation (CNCF), huge community among container orchestration tools, it is an open source and modular tool that works with any OS.
      See more