AWS Glue vs Apache Spark: What are the differences?
Developers describe AWS Glue as "Fully managed extract, transform, and load (ETL) service". A fully managed extract, transform, and load (ETL) service that makes it easy for customers to prepare and load their data for analytics. On the other hand, Apache Spark is detailed as "Fast and general engine for large-scale data processing". Spark is a fast and general processing engine compatible with Hadoop data. It can run in Hadoop clusters through YARN or Spark's standalone mode, and it can process data in HDFS, HBase, Cassandra, Hive, and any Hadoop InputFormat. It is designed to perform both batch processing (similar to MapReduce) and new workloads like streaming, interactive queries, and machine learning.
AWS Glue and Apache Spark belong to "Big Data Tools" category of the tech stack.
Some of the features offered by AWS Glue are:
- Easy - AWS Glue automates much of the effort in building, maintaining, and running ETL jobs. AWS Glue crawls your data sources, identifies data formats, and suggests schemas and transformations. AWS Glue automatically generates the code to execute your data transformations and loading processes.
- Integrated - AWS Glue is integrated across a wide range of AWS services.
- Serverless - AWS Glue is serverless. There is no infrastructure to provision or manage. AWS Glue handles provisioning, configuration, and scaling of the resources required to run your ETL jobs on a fully managed, scale-out Apache Spark environment. You pay only for the resources used while your jobs are running.
On the other hand, Apache Spark provides the following key features:
- Run programs up to 100x faster than Hadoop MapReduce in memory, or 10x faster on disk
- Write applications quickly in Java, Scala or Python
- Combine SQL, streaming, and complex analytics
Apache Spark is an open source tool with 22.5K GitHub stars and 19.4K GitHub forks. Here's a link to Apache Spark's open source repository on GitHub.
According to the StackShare community, Apache Spark has a broader approval, being mentioned in 266 company stacks & 112 developers stacks; compared to AWS Glue, which is listed in 13 company stacks and 7 developer stacks.
Sign up to add or upvote prosMake informed product decisions
Sign up to add or upvote consMake informed product decisions
What is AWS Glue?
What is Apache Spark?
Need advice about which tool to choose?Ask the StackShare community!
Sign up to get full access to all the companiesMake informed product decisions
Sign up to get full access to all the tool integrationsMake informed product decisions