Google BigQuery vs Impala

Overview

Google BigQuery

Stacks1.8K

Followers1.5K

Votes152

Apache Impala

Stacks145

Followers301

Votes18

GitHub Stars34

Forks33

Google BigQuery vs Impala: What are the differences?

Introduction

Google BigQuery and Impala are both popular data processing platforms used for querying and analyzing large datasets. While they share similarities, there are some key differences between the two. The following paragraphs highlight the main distinctions between Google BigQuery and Impala.

Distributed Processing Model: Google BigQuery uses a fully managed serverless architecture, where the entire data processing is handled by Google infrastructure. It automatically scales resources based on the query complexity and data size, making it effortless for users to focus on analysis rather than system management. Impala, on the other hand, adopts a distributed processing model where queries are executed on a cluster of machines. Users are responsible for managing the cluster and allocating appropriate resources to ensure efficient query performance.
Data Storage: Google BigQuery utilizes columnar storage for data storage, known as Capacitor. It compresses and encodes data effectively, leading to significant storage savings. Impala, however, uses the Hadoop Distributed File System (HDFS) for data storage, which provides fault-tolerance and scalability. HDFS stores data in a distributed manner across multiple nodes in a cluster.
Query Language: Google BigQuery employs an SQL-like query language named BigQuery SQL. It supports a wide range of SQL functions and allows users to write complex queries for data analysis. Impala, on the other hand, uses a subset of SQL known as Impala SQL. Although Impala SQL is compatible with most SQL implementations, it lacks support for some advanced features and functions present in BigQuery SQL.
Performance Optimization: Google BigQuery automatically optimizes query execution by utilizing dynamic query optimization techniques. It intelligently chooses the best execution plans based on data statistics and parallelizes processing for large datasets. In contrast, Impala relies on metadata information provided by users to optimize query execution. Users need to analyze and modify table statistics manually to enhance performance.
Integration with Ecosystem: Google BigQuery seamlessly integrates with other Google Cloud Platform services, enabling easy integration with data sources, data pipelines, and machine learning services. Impala, being a part of the Apache Hadoop ecosystem, integrates well with other Hadoop components like Apache Hive, Apache HBase, and Apache Spark. This integration provides a wide range of tools and libraries for various data processing tasks.
Cost Model: Google BigQuery follows a pay-as-you-go pricing model. Users are billed based on the amount of data processed and storage consumed. It offers various pricing tiers to cater to different usage scenarios. Impala, being an open-source project, is free to use. However, users need to consider the cost of managing and maintaining the cluster infrastructure when opting for Impala.

In summary, Google BigQuery utilizes a serverless architecture, offers efficient data storage, automated query optimization, and extensive ecosystem integration. Impala, on the other hand, provides a distributed processing model, compatibility with Apache Hadoop components, and a cost advantage in terms of software licensing. The choice between the two depends on specific requirements, budget constraints, and the level of control desired by users.

Share your Stack

Help developers discover the tools you use. Get visibility for your team's tech choices and contribute to the community's knowledge.

View Docs

CLI (Node.js)

Manual

Advice on Google BigQuery, Apache Impala

Julien

CTO at Hawk

Sep 19, 2020

Decided

Cloud Data-warehouse is the centerpiece of modern Data platform. The choice of the most suitable solution is therefore fundamental.

Our benchmark was conducted over BigQuery and Snowflake. These solutions seem to match our goals but they have very different approaches.

BigQuery is notably the only 100% serverless cloud data-warehouse, which requires absolutely NO maintenance: no re-clustering, no compression, no index optimization, no storage management, no performance management. Snowflake requires to set up (paid) reclustering processes, to manage the performance allocated to each profile, etc. We can also mention Redshift, which we have eliminated because this technology requires even more ops operation.

BigQuery can therefore be set up with almost zero cost of human resources. Its on-demand pricing is particularly adapted to small workloads. 0 cost when the solution is not used, only pay for the query you're running. But quickly the use of slots (with monthly or per-minute commitment) will drastically reduce the cost of use. We've reduced by 10 the cost of our nightly batches by using flex slots.

Finally, a major advantage of BigQuery is its almost perfect integration with Google Cloud Platform services: Cloud functions, Dataflow, Data Studio, etc.

BigQuery is still evolving very quickly. The next milestone, BigQuery Omni, will allow to run queries over data stored in an external Cloud platform (Amazon S3 for example). It will be a major breakthrough in the history of cloud data-warehouses. Omni will compensate a weakness of BigQuery: transferring data in near real time from S3 to BQ is not easy today. It was even simpler to implement via Snowflake's Snowpipe solution.

We also plan to use the Machine Learning features built into BigQuery to accelerate our deployment of Data-Science-based projects. An opportunity only offered by the BigQuery solution

193k views193k

Comments

Detailed Comparison

Google BigQuery	Apache Impala
Run super-fast, SQL-like queries against terabytes of data in seconds, using the processing power of Google's infrastructure. Load data with ease. Bulk load your data using Google Cloud Storage or stream it in. Easy access. Access BigQuery by using a browser tool, a command-line tool, or by making calls to the BigQuery REST API with client libraries such as Java, PHP or Python.	Impala is a modern, open source, MPP SQL query engine for Apache Hadoop. Impala is shipped by Cloudera, MapR, and Amazon. With Impala, you can query data, whether stored in HDFS or Apache HBase – including SELECT, JOIN, and aggregate functions – in real time.
All behind the scenes- Your queries can execute asynchronously in the background, and can be polled for status.;Import data with ease- Bulk load your data using Google Cloud Storage or stream it in bursts of up to 1,000 rows per second.;Affordable big data- The first Terabyte of data processed each month is free.;The right interface- Separate interfaces for administration and developers will make sure that you have access to the tools you need.	Do BI-style Queries on Hadoop;Unify Your Infrastructure;Implement Quickly;Count on Enterprise-class Security;Retain Freedom from Lock-in;Expand the Hadoop User-verse
Statistics
GitHub Stars -	GitHub Stars 34
GitHub Forks -	GitHub Forks 33
Stacks 1.8K	Stacks 145
Followers 1.5K	Followers 301
Votes 152	Votes 18
Pros & Cons
Pros 28 High Performance 25 Easy to use 22 Fully managed service 19 Cheap Pricing 16 Process hundreds of GB in seconds Cons 1 You can't unit test changes in BQ data 0 Sdas	Pros 11 Super fast 1 Massively Parallel Processing 1 Replication 1 Scalability 1 Distributed
Integrations
Xplenty Fluentd Looker Chartio Treasure Data	Hadoop Mode Redash Apache Kudu

What are some alternatives to Google BigQuery, Apache Impala?

Apache Spark

Spark is a fast and general processing engine compatible with Hadoop data. It can run in Hadoop clusters through YARN or Spark's standalone mode, and it can process data in HDFS, HBase, Cassandra, Hive, and any Hadoop InputFormat. It is designed to perform both batch processing (similar to MapReduce) and new workloads like streaming, interactive queries, and machine learning.

Amazon Redshift

It is optimized for data sets ranging from a few hundred gigabytes to a petabyte or more and costs less than $1,000 per terabyte per year, a tenth the cost of most traditional data warehousing solutions.

Qubole

Qubole is a cloud based service that makes big data easy for analysts and data engineers.

Presto

Distributed SQL Query Engine for Big Data

Amazon EMR

It is used in a variety of applications, including log analysis, data warehousing, machine learning, financial analysis, scientific simulation, and bioinformatics.

Amazon Athena

Amazon Athena is an interactive query service that makes it easy to analyze data in Amazon S3 using standard SQL. Athena is serverless, so there is no infrastructure to manage, and you pay only for the queries that you run.

Apache Flink

Apache Flink is an open source system for fast and versatile data analytics in clusters. Flink supports batch and streaming analytics, in one system. Analytical programs can be written in concise and elegant APIs in Java and Scala.

lakeFS

It is an open-source data version control system for data lakes. It provides a “Git for data” platform enabling you to implement best practices from software engineering on your data lake, including branching and merging, CI/CD, and production-like dev/test environments.

Druid

Druid is a distributed, column-oriented, real-time analytics data store that is commonly used to power exploratory dashboards in multi-tenant environments. Druid excels as a data warehousing solution for fast aggregate queries on petabyte sized data sets. Druid supports a variety of flexible filters, exact calculations, approximate algorithms, and other useful calculations.

Altiscale

we run Apache Hadoop for you. We not only deploy Hadoop, we monitor, manage, fix, and update it for you. Then we take it a step further: We monitor your jobs, notify you when something’s wrong with them, and can help with tuning.

Related Comparisons

Google BigQuery vs Impala: What are the differences?

Introduction

Distributed Processing Model: Google BigQuery uses a fully managed serverless architecture, where the entire data processing is handled by Google infrastructure. It automatically scales resources based on the query complexity and data size, making it effortless for users to focus on analysis rather than system management. Impala, on the other hand, adopts a distributed processing model where queries are executed on a cluster of machines. Users are responsible for managing the cluster and allocating appropriate resources to ensure efficient query performance.
Data Storage: Google BigQuery utilizes columnar storage for data storage, known as Capacitor. It compresses and encodes data effectively, leading to significant storage savings. Impala, however, uses the Hadoop Distributed File System (HDFS) for data storage, which provides fault-tolerance and scalability. HDFS stores data in a distributed manner across multiple nodes in a cluster.
Query Language: Google BigQuery employs an SQL-like query language named BigQuery SQL. It supports a wide range of SQL functions and allows users to write complex queries for data analysis. Impala, on the other hand, uses a subset of SQL known as Impala SQL. Although Impala SQL is compatible with most SQL implementations, it lacks support for some advanced features and functions present in BigQuery SQL.
Performance Optimization: Google BigQuery automatically optimizes query execution by utilizing dynamic query optimization techniques. It intelligently chooses the best execution plans based on data statistics and parallelizes processing for large datasets. In contrast, Impala relies on metadata information provided by users to optimize query execution. Users need to analyze and modify table statistics manually to enhance performance.
Integration with Ecosystem: Google BigQuery seamlessly integrates with other Google Cloud Platform services, enabling easy integration with data sources, data pipelines, and machine learning services. Impala, being a part of the Apache Hadoop ecosystem, integrates well with other Hadoop components like Apache Hive, Apache HBase, and Apache Spark. This integration provides a wide range of tools and libraries for various data processing tasks.
Cost Model: Google BigQuery follows a pay-as-you-go pricing model. Users are billed based on the amount of data processed and storage consumed. It offers various pricing tiers to cater to different usage scenarios. Impala, being an open-source project, is free to use. However, users need to consider the cost of managing and maintaining the cluster infrastructure when opting for Impala.

Google BigQuery vs Impala

Overview

Google BigQuery vs Impala: What are the differences?