Need advice about which tool to choose?Ask the StackShare community!
Denodo vs Presto: What are the differences?
Introduction
This article compares the key differences between Denodo and Presto.
Data Virtualization vs. Distributed SQL Engine: Denodo is a data virtualization platform that provides a single virtual layer to access and consume data from disparate sources in real-time, enabling a logical view of data without physically moving or replicating it. On the other hand, Presto is a distributed SQL query engine that enables fast and interactive analytics on multiple data sources, including Hadoop, Cassandra, and relational databases. It focuses on executing queries in a distributed manner, leveraging parallel processing to provide high performance.
Data Integration Capabilities: Denodo offers comprehensive data integration capabilities, allowing users to combine, cleanse, transform, and integrate data from multiple sources. It supports various data integration techniques like data virtualization, data replication, data services, and ETL (Extract, Transform, Load). In contrast, Presto primarily focuses on querying and analyzing data rather than data integration. While Presto supports some data connectors, it does not provide the same level of data integration functionality as Denodo.
Ecosystem Support: Denodo has a broader ecosystem support, with connectors and support for a wide range of data sources and technologies, including relational databases, NoSQL databases, cloud storage, ERP systems, CRM systems, and more. It also supports data governance and security features. On the other hand, Presto has a smaller ecosystem compared to Denodo, but it is better suited for big data analytics use cases and integrates well with Hadoop ecosystem components like HDFS, Hive, and HBase.
Performance and Scalability: Denodo focuses on providing high-performance data virtualization and caching mechanisms to optimize query performance. It leverages caching, query optimization, and other optimization techniques to minimize data movement and deliver fast query responses. Presto, being a distributed SQL engine, excels in scalability and can process large amounts of data by harnessing the power of a distributed cluster. It can execute queries in parallel across multiple nodes, providing fast query performance for complex analytical queries.
SQL Dialect and Query Optimizations: Denodo supports multiple SQL dialects, making it easier to work with different databases and technologies. It provides query optimization capabilities through Denodo's optimizer, which can rewrite and optimize queries for better performance. Presto, being a SQL-based query engine, supports standard SQL queries and provides advanced query optimization techniques to optimize query execution plans, including predicate pushdown, join reordering, and vectorization.
Customization and Extensibility: Denodo offers extensive customization and extensibility options, allowing users to create custom functions, data sources, connectors, and even develop custom query optimization strategies. This flexibility enables users to tailor Denodo to their specific requirements. Presto, on the other hand, is more focused on providing a stable distributed SQL engine and does not offer the same level of customization and extensibility as Denodo.
In Summary, Denodo is a data virtualization platform with comprehensive data integration capabilities, broader ecosystem support, and extensive customization options. On the other hand, Presto is a distributed SQL engine optimized for performance, scalability, and analytics on big data sources, with better integration capabilities with Hadoop ecosystem components.
To provide employees with the critical need of interactive querying, we’ve worked with Presto, an open-source distributed SQL query engine, over the years. Operating Presto at Pinterest’s scale has involved resolving quite a few challenges like, supporting deeply nested and huge thrift schemas, slow/ bad worker detection and remediation, auto-scaling cluster, graceful cluster shutdown and impersonation support for ldap authenticator.
Our infrastructure is built on top of Amazon EC2 and we leverage Amazon S3 for storing our data. This separates compute and storage layers, and allows multiple compute clusters to share the S3 data.
We have hundreds of petabytes of data and tens of thousands of Apache Hive tables. Our Presto clusters are comprised of a fleet of 450 r4.8xl EC2 instances. Presto clusters together have over 100 TBs of memory and 14K vcpu cores. Within Pinterest, we have close to more than 1,000 monthly active users (out of total 1,600+ Pinterest employees) using Presto, who run about 400K queries on these clusters per month.
Each query submitted to Presto cluster is logged to a Kafka topic via Singer. Singer is a logging agent built at Pinterest and we talked about it in a previous post. Each query is logged when it is submitted and when it finishes. When a Presto cluster crashes, we will have query submitted events without corresponding query finished events. These events enable us to capture the effect of cluster crashes over time.
Each Presto cluster at Pinterest has workers on a mix of dedicated AWS EC2 instances and Kubernetes pods. Kubernetes platform provides us with the capability to add and remove workers from a Presto cluster very quickly. The best-case latency on bringing up a new worker on Kubernetes is less than a minute. However, when the Kubernetes cluster itself is out of resources and needs to scale up, it can take up to ten minutes. Some other advantages of deploying on Kubernetes platform is that our Presto deployment becomes agnostic of cloud vendor, instance types, OS, etc.
#BigData #AWS #DataScience #DataEngineering
The platform deals with time series data from sensors aggregated against things( event data that originates at periodic intervals). We use Cassandra as our distributed database to store time series data. Aggregated data insights from Cassandra is delivered as web API for consumption from other applications. Presto as a distributed sql querying engine, can provide a faster execution time provided the queries are tuned for proper distribution across the cluster. Another objective that we had was to combine Cassandra table data with other business data from RDBMS or other big data systems where presto through its connector architecture would have opened up a whole lot of options for us.
Pros of Denodo
Pros of Presto
- Works directly on files in s3 (no ETL)18
- Open-source13
- Join multiple databases12
- Scalable10
- Gets ready in minutes7
- MPP6
