Alternatives to Amazon Kinesis logo

Alternatives to Amazon Kinesis

Kafka, Apache Spark, Amazon SQS, Amazon Kinesis Firehose, and Firehose.io are the most popular alternatives and competitors to Amazon Kinesis.
723
600
+ 1
9

What is Amazon Kinesis and what are its top alternatives?

Amazon Kinesis can collect and process hundreds of gigabytes of data per second from hundreds of thousands of sources, allowing you to easily write applications that process information in real-time, from sources such as web site click-streams, marketing and financial information, manufacturing instrumentation and social media, and operational logs and metering data.
Amazon Kinesis is a tool in the Real-time Data Processing category of a tech stack.

Top Alternatives to Amazon Kinesis

  • Kafka
    Kafka

    Kafka is a distributed, partitioned, replicated commit log service. It provides the functionality of a messaging system, but with a unique design. ...

  • Apache Spark
    Apache Spark

    Spark is a fast and general processing engine compatible with Hadoop data. It can run in Hadoop clusters through YARN or Spark's standalone mode, and it can process data in HDFS, HBase, Cassandra, Hive, and any Hadoop InputFormat. It is designed to perform both batch processing (similar to MapReduce) and new workloads like streaming, interactive queries, and machine learning. ...

  • Amazon SQS
    Amazon SQS

    Transmit any volume of data, at any level of throughput, without losing messages or requiring other services to be always available. With SQS, you can offload the administrative burden of operating and scaling a highly available messaging cluster, while paying a low price for only what you use. ...

  • Amazon Kinesis Firehose
    Amazon Kinesis Firehose

    Amazon Kinesis Firehose is the easiest way to load streaming data into AWS. It can capture and automatically load streaming data into Amazon S3 and Amazon Redshift, enabling near real-time analytics with existing business intelligence tools and dashboards you’re already using today. ...

  • Firehose.io
    Firehose.io

    Firehose is both a Rack application and JavaScript library that makes building real-time web applications possible. ...

  • Apache Storm
    Apache Storm

    Apache Storm is a free and open source distributed realtime computation system. Storm makes it easy to reliably process unbounded streams of data, doing for realtime processing what Hadoop did for batch processing. Storm has many use cases: realtime analytics, online machine learning, continuous computation, distributed RPC, ETL, and more. Storm is fast: a benchmark clocked it at over a million tuples processed per second per node. It is scalable, fault-tolerant, guarantees your data will be processed, and is easy to set up and operate. ...

  • Confluent
    Confluent

    It is a data streaming platform based on Apache Kafka: a full-scale streaming platform, capable of not only publish-and-subscribe, but also the storage and processing of data within the stream ...

  • JavaScript
    JavaScript

    JavaScript is most known as the scripting language for Web pages, but used in many non-browser environments as well such as node.js or Apache CouchDB. It is a prototype-based, multi-paradigm scripting language that is dynamic,and supports object-oriented, imperative, and functional programming styles. ...

Amazon Kinesis alternatives & related posts

Kafka logo

Kafka

23.3K
21.8K
607
Distributed, fault tolerant, high throughput pub-sub messaging system
23.3K
21.8K
+ 1
607
PROS OF KAFKA
  • 126
    High-throughput
  • 119
    Distributed
  • 92
    Scalable
  • 86
    High-Performance
  • 66
    Durable
  • 38
    Publish-Subscribe
  • 19
    Simple-to-use
  • 18
    Open source
  • 12
    Written in Scala and java. Runs on JVM
  • 9
    Message broker + Streaming system
  • 4
    KSQL
  • 4
    Avro schema integration
  • 4
    Robust
  • 3
    Suport Multiple clients
  • 2
    Extremely good parallelism constructs
  • 2
    Partioned, replayable log
  • 1
    Simple publisher / multi-subscriber model
  • 1
    Fun
  • 1
    Flexible
CONS OF KAFKA
  • 32
    Non-Java clients are second-class citizens
  • 29
    Needs Zookeeper
  • 9
    Operational difficulties
  • 5
    Terrible Packaging

related Kafka posts

Nick Rockwell
SVP, Engineering at Fastly · | 46 upvotes · 3.6M views

When I joined NYT there was already broad dissatisfaction with the LAMP (Linux Apache HTTP Server MySQL PHP) Stack and the front end framework, in particular. So, I wasn't passing judgment on it. I mean, LAMP's fine, you can do good work in LAMP. It's a little dated at this point, but it's not ... I didn't want to rip it out for its own sake, but everyone else was like, "We don't like this, it's really inflexible." And I remember from being outside the company when that was called MIT FIVE when it had launched. And been observing it from the outside, and I was like, you guys took so long to do that and you did it so carefully, and yet you're not happy with your decisions. Why is that? That was more the impetus. If we're going to do this again, how are we going to do it in a way that we're gonna get a better result?

So we're moving quickly away from LAMP, I would say. So, right now, the new front end is React based and using Apollo. And we've been in a long, protracted, gradual rollout of the core experiences.

React is now talking to GraphQL as a primary API. There's a Node.js back end, to the front end, which is mainly for server-side rendering, as well.

Behind there, the main repository for the GraphQL server is a big table repository, that we call Bodega because it's a convenience store. And that reads off of a Kafka pipeline.

See more
Ashish Singh
Tech Lead, Big Data Platform at Pinterest · | 38 upvotes · 3M views

To provide employees with the critical need of interactive querying, we’ve worked with Presto, an open-source distributed SQL query engine, over the years. Operating Presto at Pinterest’s scale has involved resolving quite a few challenges like, supporting deeply nested and huge thrift schemas, slow/ bad worker detection and remediation, auto-scaling cluster, graceful cluster shutdown and impersonation support for ldap authenticator.

Our infrastructure is built on top of Amazon EC2 and we leverage Amazon S3 for storing our data. This separates compute and storage layers, and allows multiple compute clusters to share the S3 data.

We have hundreds of petabytes of data and tens of thousands of Apache Hive tables. Our Presto clusters are comprised of a fleet of 450 r4.8xl EC2 instances. Presto clusters together have over 100 TBs of memory and 14K vcpu cores. Within Pinterest, we have close to more than 1,000 monthly active users (out of total 1,600+ Pinterest employees) using Presto, who run about 400K queries on these clusters per month.

Each query submitted to Presto cluster is logged to a Kafka topic via Singer. Singer is a logging agent built at Pinterest and we talked about it in a previous post. Each query is logged when it is submitted and when it finishes. When a Presto cluster crashes, we will have query submitted events without corresponding query finished events. These events enable us to capture the effect of cluster crashes over time.

Each Presto cluster at Pinterest has workers on a mix of dedicated AWS EC2 instances and Kubernetes pods. Kubernetes platform provides us with the capability to add and remove workers from a Presto cluster very quickly. The best-case latency on bringing up a new worker on Kubernetes is less than a minute. However, when the Kubernetes cluster itself is out of resources and needs to scale up, it can take up to ten minutes. Some other advantages of deploying on Kubernetes platform is that our Presto deployment becomes agnostic of cloud vendor, instance types, OS, etc.

#BigData #AWS #DataScience #DataEngineering

See more
Apache Spark logo

Apache Spark

2.9K
3.5K
140
Fast and general engine for large-scale data processing
2.9K
3.5K
+ 1
140
PROS OF APACHE SPARK
  • 61
    Open-source
  • 48
    Fast and Flexible
  • 8
    One platform for every big data problem
  • 8
    Great for distributed SQL like applications
  • 6
    Easy to install and to use
  • 3
    Works well for most Datascience usecases
  • 2
    Interactive Query
  • 2
    Machine learning libratimery, Streaming in real
  • 2
    In memory Computation
CONS OF APACHE SPARK
  • 4
    Speed

related Apache Spark posts

Conor Myhrvold
Tech Brand Mgr, Office of CTO at Uber · | 44 upvotes · 11.2M views

How Uber developed the open source, end-to-end distributed tracing Jaeger , now a CNCF project:

Distributed tracing is quickly becoming a must-have component in the tools that organizations use to monitor their complex, microservice-based architectures. At Uber, our open source distributed tracing system Jaeger saw large-scale internal adoption throughout 2016, integrated into hundreds of microservices and now recording thousands of traces every second.

Here is the story of how we got here, from investigating off-the-shelf solutions like Zipkin, to why we switched from pull to push architecture, and how distributed tracing will continue to evolve:

https://eng.uber.com/distributed-tracing/

(GitHub Pages : https://www.jaegertracing.io/, GitHub: https://github.com/jaegertracing/jaeger)

Bindings/Operator: Python Java Node.js Go C++ Kubernetes JavaScript OpenShift C# Apache Spark

See more
Eric Colson
Chief Algorithms Officer at Stitch Fix · | 21 upvotes · 6.1M views

The algorithms and data infrastructure at Stitch Fix is housed in #AWS. Data acquisition is split between events flowing through Kafka, and periodic snapshots of PostgreSQL DBs. We store data in an Amazon S3 based data warehouse. Apache Spark on Yarn is our tool of choice for data movement and #ETL. Because our storage layer (s3) is decoupled from our processing layer, we are able to scale our compute environment very elastically. We have several semi-permanent, autoscaling Yarn clusters running to serve our data processing needs. While the bulk of our compute infrastructure is dedicated to algorithmic processing, we also implemented Presto for adhoc queries and dashboards.

Beyond data movement and ETL, most #ML centric jobs (e.g. model training and execution) run in a similarly elastic environment as containers running Python and R code on Amazon EC2 Container Service clusters. The execution of batch jobs on top of ECS is managed by Flotilla, a service we built in house and open sourced (see https://github.com/stitchfix/flotilla-os).

At Stitch Fix, algorithmic integrations are pervasive across the business. We have dozens of data products actively integrated systems. That requires serving layer that is robust, agile, flexible, and allows for self-service. Models produced on Flotilla are packaged for deployment in production using Khan, another framework we've developed internally. Khan provides our data scientists the ability to quickly productionize those models they've developed with open source frameworks in Python 3 (e.g. PyTorch, sklearn), by automatically packaging them as Docker containers and deploying to Amazon ECS. This provides our data scientist a one-click method of getting from their algorithms to production. We then integrate those deployments into a service mesh, which allows us to A/B test various implementations in our product.

For more info:

#DataScience #DataStack #Data

See more
Amazon SQS logo

Amazon SQS

2.2K
2K
171
Fully managed message queuing service
2.2K
2K
+ 1
171
PROS OF AMAZON SQS
  • 62
    Easy to use, reliable
  • 40
    Low cost
  • 28
    Simple
  • 14
    Doesn't need to maintain it
  • 8
    It is Serverless
  • 4
    Has a max message size (currently 256K)
  • 3
    Triggers Lambda
  • 3
    Easy to configure with Terraform
  • 3
    Delayed delivery upto 15 mins only
  • 3
    Delayed delivery upto 12 hours
  • 1
    JMS compliant
  • 1
    Support for retry and dead letter queue
  • 1
    D
CONS OF AMAZON SQS
  • 2
    Has a max message size (currently 256K)
  • 2
    Proprietary
  • 2
    Difficult to configure
  • 1
    Has a maximum 15 minutes of delayed messages only

related Amazon SQS posts

Jeyabalaji Subramanian

Recently we were looking at a few robust and cost-effective ways of replicating the data that resides in our production MongoDB to a PostgreSQL database for data warehousing and business intelligence.

We set ourselves the following criteria for the optimal tool that would do this job: - The data replication must be near real-time, yet it should NOT impact the production database - The data replication must be horizontally scalable (based on the load), asynchronous & crash-resilient

Based on the above criteria, we selected the following tools to perform the end to end data replication:

We chose MongoDB Stitch for picking up the changes in the source database. It is the serverless platform from MongoDB. One of the services offered by MongoDB Stitch is Stitch Triggers. Using stitch triggers, you can execute a serverless function (in Node.js) in real time in response to changes in the database. When there are a lot of database changes, Stitch automatically "feeds forward" these changes through an asynchronous queue.

We chose Amazon SQS as the pipe / message backbone for communicating the changes from MongoDB to our own replication service. Interestingly enough, MongoDB stitch offers integration with AWS services.

In the Node.js function, we wrote minimal functionality to communicate the database changes (insert / update / delete / replace) to Amazon SQS.

Next we wrote a minimal micro-service in Python to listen to the message events on SQS, pickup the data payload & mirror the DB changes on to the target Data warehouse. We implemented source data to target data translation by modelling target table structures through SQLAlchemy . We deployed this micro-service as AWS Lambda with Zappa. With Zappa, deploying your services as event-driven & horizontally scalable Lambda service is dumb-easy.

In the end, we got to implement a highly scalable near realtime Change Data Replication service that "works" and deployed to production in a matter of few days!

See more
Praveen Mooli
Engineering Manager at Taylor and Francis · | 19 upvotes · 3.9M views

We are in the process of building a modern content platform to deliver our content through various channels. We decided to go with Microservices architecture as we wanted scale. Microservice architecture style is an approach to developing an application as a suite of small independently deployable services built around specific business capabilities. You can gain modularity, extensive parallelism and cost-effective scaling by deploying services across many distributed servers. Microservices modularity facilitates independent updates/deployments, and helps to avoid single point of failure, which can help prevent large-scale outages. We also decided to use Event Driven Architecture pattern which is a popular distributed asynchronous architecture pattern used to produce highly scalable applications. The event-driven architecture is made up of highly decoupled, single-purpose event processing components that asynchronously receive and process events.

To build our #Backend capabilities we decided to use the following: 1. #Microservices - Java with Spring Boot , Node.js with ExpressJS and Python with Flask 2. #Eventsourcingframework - Amazon Kinesis , Amazon Kinesis Firehose , Amazon SNS , Amazon SQS, AWS Lambda 3. #Data - Amazon RDS , Amazon DynamoDB , Amazon S3 , MongoDB Atlas

To build #Webapps we decided to use Angular 2 with RxJS

#Devops - GitHub , Travis CI , Terraform , Docker , Serverless

See more
Amazon Kinesis Firehose logo

Amazon Kinesis Firehose

234
185
0
Simple and Scalable Data Ingestion
234
185
+ 1
0
PROS OF AMAZON KINESIS FIREHOSE
    Be the first to leave a pro
    CONS OF AMAZON KINESIS FIREHOSE
      Be the first to leave a con

      related Amazon Kinesis Firehose posts

      Praveen Mooli
      Engineering Manager at Taylor and Francis · | 19 upvotes · 3.9M views

      We are in the process of building a modern content platform to deliver our content through various channels. We decided to go with Microservices architecture as we wanted scale. Microservice architecture style is an approach to developing an application as a suite of small independently deployable services built around specific business capabilities. You can gain modularity, extensive parallelism and cost-effective scaling by deploying services across many distributed servers. Microservices modularity facilitates independent updates/deployments, and helps to avoid single point of failure, which can help prevent large-scale outages. We also decided to use Event Driven Architecture pattern which is a popular distributed asynchronous architecture pattern used to produce highly scalable applications. The event-driven architecture is made up of highly decoupled, single-purpose event processing components that asynchronously receive and process events.

      To build our #Backend capabilities we decided to use the following: 1. #Microservices - Java with Spring Boot , Node.js with ExpressJS and Python with Flask 2. #Eventsourcingframework - Amazon Kinesis , Amazon Kinesis Firehose , Amazon SNS , Amazon SQS, AWS Lambda 3. #Data - Amazon RDS , Amazon DynamoDB , Amazon S3 , MongoDB Atlas

      To build #Webapps we decided to use Angular 2 with RxJS

      #Devops - GitHub , Travis CI , Terraform , Docker , Serverless

      See more
      Glenn Gillen

      I'm currently building out a Twitter analysis tool that's using AWS Lambda to stream data into Amazon Kinesis Firehose, which in turns saves the result to Amazon S3. The plan is to have Amazon S3 operate as both a data store and quasi-messaging bus with any post-processing work (e.g., notifications of new tweets going into Slack) fanning out from there. I went with this approach as I can get things up and running quickly and only pay for things on a pay-per-use basis rather than having lots of worker nodes sitting around waiting for work. Amazon Kinesis Firehose also makes it easy to add a different or additional data store in the future.

      See more
      Firehose.io logo

      Firehose.io

      2
      16
      7
      Build realtime Ruby web applications
      2
      16
      + 1
      7
      PROS OF FIREHOSE.IO
      • 2
        RESTful
      • 2
        Rails gem
      • 2
        Works with ActiveRecord
      • 1
        Clean way to build real-time web applications
      CONS OF FIREHOSE.IO
        Be the first to leave a con

        related Firehose.io posts

        Apache Storm logo

        Apache Storm

        202
        282
        25
        Distributed and fault-tolerant realtime computation
        202
        282
        + 1
        25
        PROS OF APACHE STORM
        • 10
          Flexible
        • 6
          Easy setup
        • 4
          Event Processing
        • 3
          Clojure
        • 2
          Real Time
        CONS OF APACHE STORM
          Be the first to leave a con

          related Apache Storm posts

          Marc Bollinger
          Infra & Data Eng Manager at Thumbtack · | 5 upvotes · 1.9M views

          Lumosity is home to the world's largest cognitive training database, a responsibility we take seriously. For most of the company's history, our analysis of user behavior and training data has been powered by an event stream--first a simple Node.js pub/sub app, then a heavyweight Ruby app with stronger durability. Both supported decent throughput and latency, but they lacked some major features supported by existing open-source alternatives: replaying existing messages (also lacking in most message queue-based solutions), scaling out many different readers for the same stream, the ability to leverage existing solutions for reading and writing, and possibly most importantly: the ability to hire someone externally who already had expertise.

          We ultimately migrated to Kafka in early- to mid-2016, citing both industry trends in companies we'd talked to with similar durability and throughput needs, the extremely strong documentation and community. We pored over Kyle Kingsbury's Jepsen post (https://aphyr.com/posts/293-jepsen-Kafka), as well as Jay Kreps' follow-up (http://blog.empathybox.com/post/62279088548/a-few-notes-on-kafka-and-jepsen), talked at length with Confluent folks and community members, and still wound up running parallel systems for quite a long time, but ultimately, we've been very, very happy. Understanding the internals and proper levers takes some commitment, but it's taken very little maintenance once configured. Since then, the Confluent Platform community has grown and grown; we've gone from doing most development using custom Scala consumers and producers to being 60/40 Kafka Streams/Connects.

          We originally looked into Storm / Heron , and we'd moved on from Redis pub/sub. Heron looks great, but we already had a programming model across services that was more akin to consuming a message consumers than required a topology of bolts, etc. Heron also had just come out while we were starting to migrate things, and the community momentum and direction of Kafka felt more substantial than the older Storm. If we were to start the process over again today, we might check out Pulsar , although the ecosystem is much younger.

          To find out more, read our 2017 engineering blog post about the migration!

          See more
          Confluent logo

          Confluent

          240
          235
          14
          A stream data platform to help companies harness their high volume real-time data streams
          240
          235
          + 1
          14
          PROS OF CONFLUENT
          • 4
            Free for casual use
          • 3
            No hypercloud lock-in
          • 3
            Dashboard for kafka insight
          • 2
            Easily scalable
          • 2
            Zero devops
          CONS OF CONFLUENT
          • 1
            Proprietary

          related Confluent posts

          I have recently started using Confluent/Kafka cloud. We want to do some stream processing. As I was going through Kafka I came across Kafka Streams and KSQL. Both seem to be A good fit for stream processing. But I could not understand which one should be used and one has any advantage over another. We will be using Confluent/Kafka Managed Cloud Instance. In near future, our Producers and Consumers are running on premise and we will be interacting with Confluent Cloud.

          Also, Confluent Cloud Kafka has a primitive interface; is there any better UI interface to manage Kafka Cloud Cluster?

          See more
          JavaScript logo

          JavaScript

          354.3K
          269.4K
          8.1K
          Lightweight, interpreted, object-oriented language with first-class functions
          354.3K
          269.4K
          + 1
          8.1K
          PROS OF JAVASCRIPT
          • 1.7K
            Can be used on frontend/backend
          • 1.5K
            It's everywhere
          • 1.2K
            Lots of great frameworks
          • 897
            Fast
          • 745
            Light weight
          • 425
            Flexible
          • 392
            You can't get a device today that doesn't run js
          • 286
            Non-blocking i/o
          • 237
            Ubiquitousness
          • 191
            Expressive
          • 55
            Extended functionality to web pages
          • 49
            Relatively easy language
          • 46
            Executed on the client side
          • 30
            Relatively fast to the end user
          • 25
            Pure Javascript
          • 21
            Functional programming
          • 15
            Async
          • 13
            Full-stack
          • 12
            Setup is easy
          • 12
            Future Language of The Web
          • 12
            Its everywhere
          • 11
            Because I love functions
          • 11
            JavaScript is the New PHP
          • 10
            Like it or not, JS is part of the web standard
          • 9
            Expansive community
          • 9
            Everyone use it
          • 9
            Can be used in backend, frontend and DB
          • 9
            Easy
          • 8
            Most Popular Language in the World
          • 8
            Powerful
          • 8
            Can be used both as frontend and backend as well
          • 8
            For the good parts
          • 8
            No need to use PHP
          • 8
            Easy to hire developers
          • 7
            Agile, packages simple to use
          • 7
            Love-hate relationship
          • 7
            Photoshop has 3 JS runtimes built in
          • 7
            Evolution of C
          • 7
            It's fun
          • 7
            Hard not to use
          • 7
            Versitile
          • 7
            Its fun and fast
          • 7
            Nice
          • 7
            Popularized Class-Less Architecture & Lambdas
          • 7
            Supports lambdas and closures
          • 6
            It let's me use Babel & Typescript
          • 6
            Can be used on frontend/backend/Mobile/create PRO Ui
          • 6
            1.6K Can be used on frontend/backend
          • 6
            Client side JS uses the visitors CPU to save Server Res
          • 6
            Easy to make something
          • 5
            Clojurescript
          • 5
            Promise relationship
          • 5
            Stockholm Syndrome
          • 5
            Function expressions are useful for callbacks
          • 5
            Scope manipulation
          • 5
            Everywhere
          • 5
            Client processing
          • 5
            What to add
          • 4
            Because it is so simple and lightweight
          • 4
            Only Programming language on browser
          • 1
            Test
          • 1
            Hard to learn
          • 1
            Test2
          • 1
            Not the best
          • 1
            Easy to understand
          • 1
            Subskill #4
          • 1
            Easy to learn
          • 0
            Hard 彤
          CONS OF JAVASCRIPT
          • 22
            A constant moving target, too much churn
          • 20
            Horribly inconsistent
          • 15
            Javascript is the New PHP
          • 9
            No ability to monitor memory utilitization
          • 8
            Shows Zero output in case of ANY error
          • 7
            Thinks strange results are better than errors
          • 6
            Can be ugly
          • 3
            No GitHub
          • 2
            Slow
          • 0
            HORRIBLE DOCUMENTS, faulty code, repo has bugs

          related JavaScript posts

          Zach Holman

          Oof. I have truly hated JavaScript for a long time. Like, for over twenty years now. Like, since the Clinton administration. It's always been a nightmare to deal with all of the aspects of that silly language.

          But wowza, things have changed. Tooling is just way, way better. I'm primarily web-oriented, and using React and Apollo together the past few years really opened my eyes to building rich apps. And I deeply apologize for using the phrase rich apps; I don't think I've ever said such Enterprisey words before.

          But yeah, things are different now. I still love Rails, and still use it for a lot of apps I build. But it's that silly rich apps phrase that's the problem. Users have way more comprehensive expectations than they did even five years ago, and the JS community does a good job at building tools and tech that tackle the problems of making heavy, complicated UI and frontend work.

          Obviously there's a lot of things happening here, so just saying "JavaScript isn't terrible" might encompass a huge amount of libraries and frameworks. But if you're like me, yeah, give things another shot- I'm somehow not hating on JavaScript anymore and... gulp... I kinda love it.

          See more
          Conor Myhrvold
          Tech Brand Mgr, Office of CTO at Uber · | 44 upvotes · 11.2M views

          How Uber developed the open source, end-to-end distributed tracing Jaeger , now a CNCF project:

          Distributed tracing is quickly becoming a must-have component in the tools that organizations use to monitor their complex, microservice-based architectures. At Uber, our open source distributed tracing system Jaeger saw large-scale internal adoption throughout 2016, integrated into hundreds of microservices and now recording thousands of traces every second.

          Here is the story of how we got here, from investigating off-the-shelf solutions like Zipkin, to why we switched from pull to push architecture, and how distributed tracing will continue to evolve:

          https://eng.uber.com/distributed-tracing/

          (GitHub Pages : https://www.jaegertracing.io/, GitHub: https://github.com/jaegertracing/jaeger)

          Bindings/Operator: Python Java Node.js Go C++ Kubernetes JavaScript OpenShift C# Apache Spark

          See more