Need advice about which tool to choose?Ask the StackShare community!

Amazon Kinesis

724
601
+ 1
9
Amazon SQS

2.2K
2K
+ 1
171
Add tool

Amazon Kinesis vs Amazon SQS: What are the differences?

Introduction

1. Data Volume and Throughput:

Amazon Kinesis is designed for handling massive streams of data with high throughput, allowing real-time processing of large amounts of data. On the other hand, Amazon SQS is best suited for decoupling application components and for smaller message volumes.

2. Data Retention:

In Amazon Kinesis, data retention is based on the time-to-live (TTL) setting, allowing data to be stored for a specified period. Conversely, Amazon SQS retains messages for a maximum of 14 days, after which they are automatically deleted.

3. Data Processing Model:

Amazon Kinesis offers real-time data streaming capabilities for processing continuously flowing data, whereas Amazon SQS follows a traditional message-queuing model, ensuring reliable message delivery between distributed components.

4. Scalability:

Amazon Kinesis provides dynamic scaling capabilities to handle fluctuating workloads efficiently, while Amazon SQS offers horizontal scaling by allowing multiple message queues and concurrent consumers to manage increased message processing.

5. Ordering of Messages:

Amazon Kinesis maintains the order of records within a shard, ensuring chronological processing of data within a partitioned stream. In contrast, Amazon SQS does not guarantee the ordering of messages across multiple queues, prioritizing fast and reliable message delivery.

6. Fan-Out Capability:

Amazon Kinesis supports fan-out architecture, allowing parallel processing of data streams by multiple consumers without impacting the performance. This feature is not available in Amazon SQS, where each message is processed by a single consumer exclusively.

In Summary, Amazon Kinesis and Amazon SQS differ in their handling of data volume, retention policies, data processing models, scalability options, message ordering, and fan-out capabilities.

Advice on Amazon Kinesis and Amazon SQS
MITHIRIDI PRASANTH
Software Engineer at LightMetrics · | 4 upvotes · 288.6K views
Needs advice
on
Amazon MQAmazon MQ
and
Amazon SQSAmazon SQS
in

I want to schedule a message. Amazon SQS provides a delay of 15 minutes, but I want it in some hours.

Example: Let's say a Message1 is consumed by a consumer A but somehow it failed inside the consumer. I would want to put it in a queue and retry after 4hrs. Can I do this in Amazon MQ? I have seen in some Amazon MQ videos saying scheduling messages can be done. But, I'm not sure how.

See more
Replies (1)
Andres Paredes
Lead Senior Software Engineer at InTouch Technology · | 1 upvotes · 219.8K views
Recommends
on
Amazon SQSAmazon SQS

Mithiridi, I believe you are talking about two different things. 1. If you need to process messages with delays of more 15m or at specific times, it's not a good idea to use queues, independently of tool SQM, Rabbit or Amazon MQ. you should considerer another approach using a scheduled job. 2. For dead queues and policy retries RabbitMQ, for example, doesn't support your use case. https://medium.com/@kiennguyen88/rabbitmq-delay-retry-schedule-with-dead-letter-exchange-31fb25a440fc I'm not sure if that is possible SNS/SQS support, they have a maximum delay for delivery (maxDelayTarget) in seconds but it's not clear the number. You can check this out: https://docs.aws.amazon.com/sns/latest/dg/sns-message-delivery-retries.html

See more
Manage your open source components, licenses, and vulnerabilities
Learn More
Pros of Amazon Kinesis
Pros of Amazon SQS
  • 9
    Scalable
  • 62
    Easy to use, reliable
  • 40
    Low cost
  • 28
    Simple
  • 14
    Doesn't need to maintain it
  • 8
    It is Serverless
  • 4
    Has a max message size (currently 256K)
  • 3
    Triggers Lambda
  • 3
    Easy to configure with Terraform
  • 3
    Delayed delivery upto 15 mins only
  • 3
    Delayed delivery upto 12 hours
  • 1
    JMS compliant
  • 1
    Support for retry and dead letter queue
  • 1
    D

Sign up to add or upvote prosMake informed product decisions

Cons of Amazon Kinesis
Cons of Amazon SQS
  • 3
    Cost
  • 2
    Has a max message size (currently 256K)
  • 2
    Proprietary
  • 2
    Difficult to configure
  • 1
    Has a maximum 15 minutes of delayed messages only

Sign up to add or upvote consMake informed product decisions

What is Amazon Kinesis?

Amazon Kinesis can collect and process hundreds of gigabytes of data per second from hundreds of thousands of sources, allowing you to easily write applications that process information in real-time, from sources such as web site click-streams, marketing and financial information, manufacturing instrumentation and social media, and operational logs and metering data.

What is Amazon SQS?

Transmit any volume of data, at any level of throughput, without losing messages or requiring other services to be always available. With SQS, you can offload the administrative burden of operating and scaling a highly available messaging cluster, while paying a low price for only what you use.

Need advice about which tool to choose?Ask the StackShare community!

What companies use Amazon Kinesis?
What companies use Amazon SQS?
Manage your open source components, licenses, and vulnerabilities
Learn More

Sign up to get full access to all the companiesMake informed product decisions

What tools integrate with Amazon Kinesis?
What tools integrate with Amazon SQS?

Sign up to get full access to all the tool integrationsMake informed product decisions

Blog Posts

Jul 2 2019 at 9:34PM

Segment

Google AnalyticsAmazon S3New Relic+25
10
6881
GitHubPythonNode.js+47
55
72784
GitGitHubSlack+30
27
18691
GitHubDockerAmazon EC2+23
12
6613
What are some alternatives to Amazon Kinesis and Amazon SQS?
Kafka
Kafka is a distributed, partitioned, replicated commit log service. It provides the functionality of a messaging system, but with a unique design.
Apache Spark
Spark is a fast and general processing engine compatible with Hadoop data. It can run in Hadoop clusters through YARN or Spark's standalone mode, and it can process data in HDFS, HBase, Cassandra, Hive, and any Hadoop InputFormat. It is designed to perform both batch processing (similar to MapReduce) and new workloads like streaming, interactive queries, and machine learning.
Amazon Kinesis Firehose
Amazon Kinesis Firehose is the easiest way to load streaming data into AWS. It can capture and automatically load streaming data into Amazon S3 and Amazon Redshift, enabling near real-time analytics with existing business intelligence tools and dashboards you’re already using today.
Firehose.io
Firehose is both a Rack application and JavaScript library that makes building real-time web applications possible.
Apache Storm
Apache Storm is a free and open source distributed realtime computation system. Storm makes it easy to reliably process unbounded streams of data, doing for realtime processing what Hadoop did for batch processing. Storm has many use cases: realtime analytics, online machine learning, continuous computation, distributed RPC, ETL, and more. Storm is fast: a benchmark clocked it at over a million tuples processed per second per node. It is scalable, fault-tolerant, guarantees your data will be processed, and is easy to set up and operate.
See all alternatives