Amazon SQS vs Kafka: What are the differences?
Developers describe Amazon SQS as "Fully managed message queuing service". Transmit any volume of data, at any level of throughput, without losing messages or requiring other services to be always available. With SQS, you can offload the administrative burden of operating and scaling a highly available messaging cluster, while paying a low price for only what you use. On the other hand, Kafka is detailed as "Distributed, fault tolerant, high throughput pub-sub messaging system". Kafka is a distributed, partitioned, replicated commit log service. It provides the functionality of a messaging system, but with a unique design.
Amazon SQS and Kafka can be categorized as "Message Queue" tools.
Some of the features offered by Amazon SQS are:
- A queue can be created in any region.
- The message payload can contain up to 256KB of text in any format. Each 64KB ‘chunk’ of payload is billed as 1 request. For example, a single API call with a 256KB payload will be billed as four requests.
- Messages can be sent, received or deleted in batches of up to 10 messages or 256KB. Batches cost the same amount as single messages, meaning SQS can be even more cost effective for customers that use batching.
On the other hand, Kafka provides the following key features:
- Written at LinkedIn in Scala
- Used by LinkedIn to offload processing of all page and other views
- Defaults to using persistence, uses OS disk cache for hot data (has higher throughput then any of the above having persistence enabled)
"Easy to use, reliable" is the top reason why over 45 developers like Amazon SQS, while over 95 developers mention "High-throughput" as the leading cause for choosing Kafka.
Kafka is an open source tool with 12.7K GitHub stars and 6.81K GitHub forks. Here's a link to Kafka's open source repository on GitHub.
Uber Technologies, Spotify, and Slack are some of the popular companies that use Kafka, whereas Amazon SQS is used by Medium, Lyft, and Coursera. Kafka has a broader approval, being mentioned in 509 company stacks & 470 developers stacks; compared to Amazon SQS, which is listed in 384 company stacks and 103 developer stacks.