Alternatives to Amazon Redshift logo

Alternatives to Amazon Redshift

Google BigQuery, Amazon Athena, Amazon DynamoDB, Amazon Redshift Spectrum, and Hadoop are the most popular alternatives and competitors to Amazon Redshift.
1.5K
1.4K
+ 1
108

What is Amazon Redshift and what are its top alternatives?

It is optimized for data sets ranging from a few hundred gigabytes to a petabyte or more and costs less than $1,000 per terabyte per year, a tenth the cost of most traditional data warehousing solutions.
Amazon Redshift is a tool in the Big Data as a Service category of a tech stack.

Top Alternatives to Amazon Redshift

  • Google BigQuery
    Google BigQuery

    Run super-fast, SQL-like queries against terabytes of data in seconds, using the processing power of Google's infrastructure. Load data with ease. Bulk load your data using Google Cloud Storage or stream it in. Easy access. Access BigQuery by using a browser tool, a command-line tool, or by making calls to the BigQuery REST API with client libraries such as Java, PHP or Python. ...

  • Amazon Athena
    Amazon Athena

    Amazon Athena is an interactive query service that makes it easy to analyze data in Amazon S3 using standard SQL. Athena is serverless, so there is no infrastructure to manage, and you pay only for the queries that you run. ...

  • Amazon DynamoDB
    Amazon DynamoDB

    With it , you can offload the administrative burden of operating and scaling a highly available distributed database cluster, while paying a low price for only what you use. ...

  • Amazon Redshift Spectrum
    Amazon Redshift Spectrum

    With Redshift Spectrum, you can extend the analytic power of Amazon Redshift beyond data stored on local disks in your data warehouse to query vast amounts of unstructured data in your Amazon S3 “data lake” -- without having to load or transform any data. ...

  • Hadoop
    Hadoop

    The Apache Hadoop software library is a framework that allows for the distributed processing of large data sets across clusters of computers using simple programming models. It is designed to scale up from single servers to thousands of machines, each offering local computation and storage. ...

  • Microsoft Azure
    Microsoft Azure

    Azure is an open and flexible cloud platform that enables you to quickly build, deploy and manage applications across a global network of Microsoft-managed datacenters. You can build applications using any language, tool or framework. And you can integrate your public cloud applications with your existing IT environment. ...

  • Snowflake
    Snowflake

    Snowflake eliminates the administration and management demands of traditional data warehouses and big data platforms. Snowflake is a true data warehouse as a service running on Amazon Web Services (AWS)—no infrastructure to manage and no knobs to turn. ...

  • Apache Aurora
    Apache Aurora

    Apache Aurora is a service scheduler that runs on top of Mesos, enabling you to run long-running services that take advantage of Mesos' scalability, fault-tolerance, and resource isolation. ...

Amazon Redshift alternatives & related posts

Google BigQuery logo

Google BigQuery

1.6K
1.5K
152
Analyze terabytes of data in seconds
1.6K
1.5K
+ 1
152
PROS OF GOOGLE BIGQUERY
  • 28
    High Performance
  • 25
    Easy to use
  • 22
    Fully managed service
  • 19
    Cheap Pricing
  • 16
    Process hundreds of GB in seconds
  • 12
    Big Data
  • 11
    Full table scans in seconds, no indexes needed
  • 8
    Always on, no per-hour costs
  • 6
    Good combination with fluentd
  • 4
    Machine learning
  • 1
    Easy to manage
  • 0
    Easy to learn
CONS OF GOOGLE BIGQUERY
  • 1
    You can't unit test changes in BQ data

related Google BigQuery posts

Context: I wanted to create an end to end IoT data pipeline simulation in Google Cloud IoT Core and other GCP services. I never touched Terraform meaningfully until working on this project, and it's one of the best explorations in my development career. The documentation and syntax is incredibly human-readable and friendly. I'm used to building infrastructure through the google apis via Python , but I'm so glad past Sung did not make that decision. I was tempted to use Google Cloud Deployment Manager, but the templates were a bit convoluted by first impression. I'm glad past Sung did not make this decision either.

Solution: Leveraging Google Cloud Build Google Cloud Run Google Cloud Bigtable Google BigQuery Google Cloud Storage Google Compute Engine along with some other fun tools, I can deploy over 40 GCP resources using Terraform!

Check Out My Architecture: CLICK ME

Check out the GitHub repo attached

See more
Tim Specht
‎Co-Founder and CTO at Dubsmash · | 14 upvotes · 948.5K views

In order to accurately measure & track user behaviour on our platform we moved over quickly from the initial solution using Google Analytics to a custom-built one due to resource & pricing concerns we had.

While this does sound complicated, it’s as easy as clients sending JSON blobs of events to Amazon Kinesis from where we use AWS Lambda & Amazon SQS to batch and process incoming events and then ingest them into Google BigQuery. Once events are stored in BigQuery (which usually only takes a second from the time the client sends the data until it’s available), we can use almost-standard-SQL to simply query for data while Google makes sure that, even with terabytes of data being scanned, query times stay in the range of seconds rather than hours. Before ingesting their data into the pipeline, our mobile clients are aggregating events internally and, once a certain threshold is reached or the app is going to the background, sending the events as a JSON blob into the stream.

In the past we had workers running that continuously read from the stream and would validate and post-process the data and then enqueue them for other workers to write them to BigQuery. We went ahead and implemented the Lambda-based approach in such a way that Lambda functions would automatically be triggered for incoming records, pre-aggregate events, and write them back to SQS, from which we then read them, and persist the events to BigQuery. While this approach had a couple of bumps on the road, like re-triggering functions asynchronously to keep up with the stream and proper batch sizes, we finally managed to get it running in a reliable way and are very happy with this solution today.

#ServerlessTaskProcessing #GeneralAnalytics #RealTimeDataProcessing #BigDataAsAService

See more
Amazon Athena logo

Amazon Athena

487
831
49
Query S3 Using SQL
487
831
+ 1
49
PROS OF AMAZON ATHENA
  • 16
    Use SQL to analyze CSV files
  • 8
    Glue crawlers gives easy Data catalogue
  • 7
    Cheap
  • 6
    Query all my data without running servers 24x7
  • 4
    No data base servers yay
  • 3
    Easy integration with QuickSight
  • 2
    Query and analyse CSV,parquet,json files in sql
  • 2
    Also glue and athena use same data catalog
  • 1
    No configuration required
  • 0
    Ad hoc checks on data made easy
CONS OF AMAZON ATHENA
    Be the first to leave a con

    related Amazon Athena posts

    So, I have data in Amazon S3 as parquet files and I have it available in the Glue data catalog too. I want to build an AppSync API on top of this data. Now the two options that I am considering are:

    1. Bring the data to Amazon DynamoDB and then build my API on top of this Database.

    2. Add a Lambda function that resolves Amazon Athena queries made by AppSync.

    Which of the two approaches will be cost effective?

    I would really appreciate some back of the envelope estimates too.

    Note: I only expect to make read queries. Thanks.

    See more

    I use Amazon Athena because similar to Google BigQuery , you can store and query data easily. Especially since you can define data schema in the Glue data catalog, there's a central way to define data models.

    However, I would not recommend for batch jobs. I typically use this to check intermediary datasets in data engineering workloads. It's good for getting a look and feel of the data along its ETL journey.

    See more
    Amazon DynamoDB logo

    Amazon DynamoDB

    3.7K
    3.2K
    195
    Fully managed NoSQL database service
    3.7K
    3.2K
    + 1
    195
    PROS OF AMAZON DYNAMODB
    • 62
      Predictable performance and cost
    • 56
      Scalable
    • 35
      Native JSON Support
    • 21
      AWS Free Tier
    • 7
      Fast
    • 3
      No sql
    • 3
      To store data
    • 2
      Serverless
    • 2
      No Stored procedures is GOOD
    • 1
      ORM with DynamoDBMapper
    • 1
      Elastic Scalability using on-demand mode
    • 1
      Elastic Scalability using autoscaling
    • 1
      DynamoDB Stream
    CONS OF AMAZON DYNAMODB
    • 4
      Only sequential access for paginate data
    • 1
      Scaling
    • 1
      Document Limit Size

    related Amazon DynamoDB posts

    Praveen Mooli
    Engineering Manager at Taylor and Francis · | 18 upvotes · 3.8M views

    We are in the process of building a modern content platform to deliver our content through various channels. We decided to go with Microservices architecture as we wanted scale. Microservice architecture style is an approach to developing an application as a suite of small independently deployable services built around specific business capabilities. You can gain modularity, extensive parallelism and cost-effective scaling by deploying services across many distributed servers. Microservices modularity facilitates independent updates/deployments, and helps to avoid single point of failure, which can help prevent large-scale outages. We also decided to use Event Driven Architecture pattern which is a popular distributed asynchronous architecture pattern used to produce highly scalable applications. The event-driven architecture is made up of highly decoupled, single-purpose event processing components that asynchronously receive and process events.

    To build our #Backend capabilities we decided to use the following: 1. #Microservices - Java with Spring Boot , Node.js with ExpressJS and Python with Flask 2. #Eventsourcingframework - Amazon Kinesis , Amazon Kinesis Firehose , Amazon SNS , Amazon SQS, AWS Lambda 3. #Data - Amazon RDS , Amazon DynamoDB , Amazon S3 , MongoDB Atlas

    To build #Webapps we decided to use Angular 2 with RxJS

    #Devops - GitHub , Travis CI , Terraform , Docker , Serverless

    See more
    Julien DeFrance
    Principal Software Engineer at Tophatter · | 16 upvotes · 3.1M views

    Back in 2014, I was given an opportunity to re-architect SmartZip Analytics platform, and flagship product: SmartTargeting. This is a SaaS software helping real estate professionals keeping up with their prospects and leads in a given neighborhood/territory, finding out (thanks to predictive analytics) who's the most likely to list/sell their home, and running cross-channel marketing automation against them: direct mail, online ads, email... The company also does provide Data APIs to Enterprise customers.

    I had inherited years and years of technical debt and I knew things had to change radically. The first enabler to this was to make use of the cloud and go with AWS, so we would stop re-inventing the wheel, and build around managed/scalable services.

    For the SaaS product, we kept on working with Rails as this was what my team had the most knowledge in. We've however broken up the monolith and decoupled the front-end application from the backend thanks to the use of Rails API so we'd get independently scalable micro-services from now on.

    Our various applications could now be deployed using AWS Elastic Beanstalk so we wouldn't waste any more efforts writing time-consuming Capistrano deployment scripts for instance. Combined with Docker so our application would run within its own container, independently from the underlying host configuration.

    Storage-wise, we went with Amazon S3 and ditched any pre-existing local or network storage people used to deal with in our legacy systems. On the database side: Amazon RDS / MySQL initially. Ultimately migrated to Amazon RDS for Aurora / MySQL when it got released. Once again, here you need a managed service your cloud provider handles for you.

    Future improvements / technology decisions included:

    Caching: Amazon ElastiCache / Memcached CDN: Amazon CloudFront Systems Integration: Segment / Zapier Data-warehousing: Amazon Redshift BI: Amazon Quicksight / Superset Search: Elasticsearch / Amazon Elasticsearch Service / Algolia Monitoring: New Relic

    As our usage grows, patterns changed, and/or our business needs evolved, my role as Engineering Manager then Director of Engineering was also to ensure my team kept on learning and innovating, while delivering on business value.

    One of these innovations was to get ourselves into Serverless : Adopting AWS Lambda was a big step forward. At the time, only available for Node.js (Not Ruby ) but a great way to handle cost efficiency, unpredictable traffic, sudden bursts of traffic... Ultimately you want the whole chain of services involved in a call to be serverless, and that's when we've started leveraging Amazon DynamoDB on these projects so they'd be fully scalable.

    See more
    Amazon Redshift Spectrum logo

    Amazon Redshift Spectrum

    101
    146
    3
    Exabyte-Scale In-Place Queries of S3 Data
    101
    146
    + 1
    3
    PROS OF AMAZON REDSHIFT SPECTRUM
    • 1
      Good Performance
    • 1
      Great Documentation
    • 1
      Economical
    CONS OF AMAZON REDSHIFT SPECTRUM
      Be the first to leave a con

      related Amazon Redshift Spectrum posts

      Hadoop logo

      Hadoop

      2.5K
      2.3K
      56
      Open-source software for reliable, scalable, distributed computing
      2.5K
      2.3K
      + 1
      56
      PROS OF HADOOP
      • 39
        Great ecosystem
      • 11
        One stack to rule them all
      • 4
        Great load balancer
      • 1
        Amazon aws
      • 1
        Java syntax
      CONS OF HADOOP
        Be the first to leave a con

        related Hadoop posts

        Shared insights
        on
        KafkaKafkaHadoopHadoop
        at

        The early data ingestion pipeline at Pinterest used Kafka as the central message transporter, with the app servers writing messages directly to Kafka, which then uploaded log files to S3.

        For databases, a custom Hadoop streamer pulled database data and wrote it to S3.

        Challenges cited for this infrastructure included high operational overhead, as well as potential data loss occurring when Kafka broker outages led to an overflow of in-memory message buffering.

        See more
        Conor Myhrvold
        Tech Brand Mgr, Office of CTO at Uber · | 7 upvotes · 3M views

        Why we built Marmaray, an open source generic data ingestion and dispersal framework and library for Apache Hadoop :

        Built and designed by our Hadoop Platform team, Marmaray is a plug-in-based framework built on top of the Hadoop ecosystem. Users can add support to ingest data from any source and disperse to any sink leveraging the use of Apache Spark . The name, Marmaray, comes from a tunnel in Turkey connecting Europe and Asia. Similarly, we envisioned Marmaray within Uber as a pipeline connecting data from any source to any sink depending on customer preference:

        https://eng.uber.com/marmaray-hadoop-ingestion-open-source/

        (Direct GitHub repo: https://github.com/uber/marmaray Kafka Kafka Manager )

        See more
        Microsoft Azure logo

        Microsoft Azure

        24.3K
        17K
        768
        Integrated cloud services and infrastructure to support computing, database, analytics, mobile, and web scenarios.
        24.3K
        17K
        + 1
        768
        PROS OF MICROSOFT AZURE
        • 114
          Scales well and quite easy
        • 96
          Can use .Net or open source tools
        • 81
          Startup friendly
        • 73
          Startup plans via BizSpark
        • 62
          High performance
        • 38
          Wide choice of services
        • 32
          Low cost
        • 32
          Lots of integrations
        • 31
          Reliability
        • 19
          Twillio & Github are directly accessible
        • 13
          RESTful API
        • 10
          PaaS
        • 10
          Enterprise Grade
        • 10
          Startup support
        • 8
          DocumentDB
        • 7
          In person support
        • 6
          Free for students
        • 6
          Service Bus
        • 6
          Virtual Machines
        • 5
          Redis Cache
        • 5
          It rocks
        • 4
          Storage, Backup, and Recovery
        • 4
          Infrastructure Services
        • 4
          SQL Databases
        • 4
          CDN
        • 3
          Integration
        • 3
          Scheduler
        • 3
          Preview Portal
        • 3
          HDInsight
        • 3
          Built on Node.js
        • 3
          Big Data
        • 3
          BizSpark 60k Azure Benefit
        • 3
          IaaS
        • 2
          Backup
        • 2
          Open cloud
        • 2
          Web
        • 2
          SaaS
        • 2
          Big Compute
        • 2
          Mobile
        • 2
          Media
        • 2
          Dev-Test
        • 2
          Storage
        • 2
          StorSimple
        • 2
          Machine Learning
        • 2
          Stream Analytics
        • 2
          Data Factory
        • 2
          Event Hubs
        • 2
          Virtual Network
        • 2
          ExpressRoute
        • 2
          Traffic Manager
        • 2
          Media Services
        • 2
          BizTalk Services
        • 2
          Site Recovery
        • 2
          Active Directory
        • 2
          Multi-Factor Authentication
        • 2
          Visual Studio Online
        • 2
          Application Insights
        • 2
          Automation
        • 2
          Operational Insights
        • 2
          Key Vault
        • 2
          Infrastructure near your customers
        • 2
          Easy Deployment
        • 1
          Enterprise customer preferences
        • 1
          Documentation
        • 1
          Security
        • 1
          Best cloud platfrom
        • 1
          Easy and fast to start with
        • 1
          Remote Debugging
        CONS OF MICROSOFT AZURE
        • 7
          Confusing UI
        • 2
          Expensive plesk on Azure

        related Microsoft Azure posts

        Dale Ross
        Independent Contractor at Self Employed · | 22 upvotes · 1.5M views

        I've heard that I have the ability to write well, at times. When it flows, it flows. I decided to start blogging in 2013 on Blogger. I started a company and joined BizPark with the Microsoft Azure allotment. I created a WordPress blog and did a migration at some point. A lot happened in the time after that migration but I stopped coding and changed cities during tumultuous times that taught me many lessons concerning mental health and productivity. I eventually graduated from BizSpark and outgrew the credit allotment. That killed the WordPress blog.

        I blogged about writing again on the existing Blogger blog but it didn't feel right. I looked at a few options where I wouldn't have to worry about hosting cost indefinitely and Jekyll stood out with GitHub Pages. The Importer was fairly straightforward for the existing blog posts.

        Todo * Set up redirects for all posts on blogger. The URI format is different so a complete redirect wouldn't work. Although, there may be something in Jekyll that could manage the redirects. I did notice the old URLs were stored in the front matter. I'm working on a command-line Ruby gem for the current plan. * I did find some of the lost WordPress posts on archive.org that I downloaded with the waybackmachinedownloader. I think I might write an importer for that. * I still have a few Disqus comment threads to map

        See more
        Ganesa Vijayakumar
        Full Stack Coder | Technical Lead · | 19 upvotes · 4.7M views

        I'm planning to create a web application and also a mobile application to provide a very good shopping experience to the end customers. Shortly, my application will be aggregate the product details from difference sources and giving a clear picture to the user that when and where to buy that product with best in Quality and cost.

        I have planned to develop this in many milestones for adding N number of features and I have picked my first part to complete the core part (aggregate the product details from different sources).

        As per my work experience and knowledge, I have chosen the followings stacks to this mission.

        UI: I would like to develop this application using React, React Router and React Native since I'm a little bit familiar on this and also most importantly these will help on developing both web and mobile apps. In addition, I'm gonna use the stacks JavaScript, jQuery, jQuery UI, jQuery Mobile, Bootstrap wherever required.

        Service: I have planned to use Java as the main business layer language as I have 7+ years of experience on this I believe I can do better work using Java than other languages. In addition, I'm thinking to use the stacks Node.js.

        Database and ORM: I'm gonna pick MySQL as DB and Hibernate as ORM since I have a piece of good knowledge and also work experience on this combination.

        Search Engine: I need to deal with a large amount of product data and it's in-detailed info to provide enough details to end user at the same time I need to focus on the performance area too. so I have decided to use Solr as a search engine for product search and suggestions. In addition, I'm thinking to replace Solr by Elasticsearch once explored/reviewed enough about Elasticsearch.

        Host: As of now, my plan to complete the application with decent features first and deploy it in a free hosting environment like Docker and Heroku and then once it is stable then I have planned to use the AWS products Amazon S3, EC2, Amazon RDS and Amazon Route 53. I'm not sure about Microsoft Azure that what is the specialty in it than Heroku and Amazon EC2 Container Service. Anyhow, I will do explore these once again and pick the best suite one for my requirement once I reached this level.

        Build and Repositories: I have decided to choose Apache Maven and Git as these are my favorites and also so popular on respectively build and repositories.

        Additional Utilities :) - I would like to choose Codacy for code review as their Startup plan will be very helpful to this application. I'm already experienced with Google CheckStyle and SonarQube even I'm looking something on Codacy.

        Happy Coding! Suggestions are welcome! :)

        Thanks, Ganesa

        See more
        Snowflake logo

        Snowflake

        1.1K
        1.2K
        27
        The data warehouse built for the cloud
        1.1K
        1.2K
        + 1
        27
        PROS OF SNOWFLAKE
        • 7
          Public and Private Data Sharing
        • 4
          Multicloud
        • 4
          Good Performance
        • 4
          User Friendly
        • 3
          Great Documentation
        • 2
          Serverless
        • 1
          Economical
        • 1
          Usage based billing
        • 1
          Innovative
        CONS OF SNOWFLAKE
          Be the first to leave a con

          related Snowflake posts

          I'm wondering if any Cloud Firestore users might be open to sharing some input and challenges encountered when trying to create a low-cost, low-latency data pipeline to their Analytics warehouse (e.g. Google BigQuery, Snowflake, etc...)

          I'm working with a platform by the name of Estuary.dev, an ETL/ELT and we are conducting some research on the pain points here to see if there are drawbacks of the Firestore->BQ extension and/or if users are seeking easy ways for getting nosql->fine-grained tabular data

          Please feel free to drop some knowledge/wish list stuff on me for a better pipeline here!

          See more
          Shared insights
          on
          Google BigQueryGoogle BigQuerySnowflakeSnowflake

          I use Google BigQuery because it makes is super easy to query and store data for analytics workloads. If you're using GCP, you're likely using BigQuery. However, running data viz tools directly connected to BigQuery will run pretty slow. They recently announced BI Engine which will hopefully compete well against big players like Snowflake when it comes to concurrency.

          What's nice too is that it has SQL-based ML tools, and it has great GIS support!

          See more
          Apache Aurora logo

          Apache Aurora

          69
          96
          0
          An Apcahe Mesos framework for scheduling jobs, originally developed by Twitter
          69
          96
          + 1
          0
          PROS OF APACHE AURORA
            Be the first to leave a pro
            CONS OF APACHE AURORA
              Be the first to leave a con

              related Apache Aurora posts

              Docker containers on Mesos run their microservices with consistent configurations at scale, along with Aurora for long-running services and cron jobs.

              See more