What is dbt and what are its top alternatives?
Dbt (data build tool) is an open-source tool that enables data analysts and engineers to transform data in their warehouses by writing SQL queries. Key features include data transformation workflows, version control, testing, documentation, and a powerful CLI. However, some limitations of dbt include limited support for non-SQL transformations, lack of real-time processing capabilities, and a steeper learning curve for beginners.
- Airflow: Apache Airflow is a platform to programmatically author, schedule, and monitor workflows. Key features include workflow automation, extensibility, and rich ecosystem of integrations. Pros include advanced scheduling capabilities and a wide range of integrations, while cons include complex setup and configuration.
- Prefect: Prefect is a workflow management system designed for modern infrastructure. Key features include dynamic workflows, parameterization, and advanced monitoring. Pros include simplicity and flexibility, while cons include a smaller community compared to other tools.
- Dagster: Dagster is a data orchestration system for machine learning, analytics, and ETL. Key features include data pipelines, declarative configuration, and testing. Pros include a focus on data quality and seamless integration with ML frameworks, while cons include a newer tool with less extensive documentation.
- Singer: Singer is an open-source framework for ETL that allows you to easily move data between systems. Key features include modular taps and targets, easy extensibility, and simple configuration. Pros include a lightweight and flexible approach to data integration, while cons include less support for complex data transformation logic.
- Luigi: Luigi is a Python-based workflow management system that helps automate complex pipelines. Key features include task dependency management, workflow visualization, and task retry mechanisms. Pros include a simple and intuitive interface, while cons include a lack of built-in data transformation capabilities.
- Kedro: Kedro is a development workflow tool that helps you build data pipelines. Key features include data modeling, pipeline visualization, and project template. Pros include a focus on reproducibility and modularity, while cons include a more developer-centric tool compared to dbt.
- Dataform: Dataform is a SQL-based tool for data transformation and orchestration. Key features include data testing, scheduling, and collaborative workflows. Pros include seamless SQL integration and version control, while cons include a more limited range of data sources compared to dbt.
- dagster-millenium-falcon: This library provides a set of tools to create Airflow-native pipelines with the structure and confidence of dagster. Key features include integration with Airflow, dependency graph visualization, and reusable components. Pros include leveraging the strengths of both dagster and Airflow, while cons include potential compatibility issues as a newer tool.
- Cube.js: Cube.js is an open-source analytics layer that helps you create analytics APIs fast. Key features include pre-aggregated data, real-time analytics, and charting libraries integration. Pros include real-time data processing capabilities, while cons include a focus on analytics rather than ETL transformations.
- Pachyderm: Pachyderm is a data versioning tool that allows you to keep track of changes to your data pipeline. Key features include versioned data processing, containerized data pipelines, and data lineage tracking. Pros include a strong focus on data versioning and reproducibility, while cons include a more complex setup compared to dbt.
Top Alternatives to dbt
- act
Rather than having to commit/push every time you want test out the changes you are making to your .github/workflows/ files (or for any changes to embedded GitHub actions), you can use this tool to run the actions locally. The environment variables and filesystem are all configured to match what GitHub provides. ...
- Airflow
Use Airflow to author workflows as directed acyclic graphs (DAGs) of tasks. The Airflow scheduler executes your tasks on an array of workers while following the specified dependencies. Rich command lines utilities makes performing complex surgeries on DAGs a snap. The rich user interface makes it easy to visualize pipelines running in production, monitor progress and troubleshoot issues when needed. ...
- Looker
We've built a unique data modeling language, connections to today's fastest analytical databases, and a service that you can deploy on any infrastructure, and explore on any device. Plus, we'll help you every step of the way. ...
- Apache Spark
Spark is a fast and general processing engine compatible with Hadoop data. It can run in Hadoop clusters through YARN or Spark's standalone mode, and it can process data in HDFS, HBase, Cassandra, Hive, and any Hadoop InputFormat. It is designed to perform both batch processing (similar to MapReduce) and new workloads like streaming, interactive queries, and machine learning. ...
- Slick
It is a modern database query and access library for Scala. It allows you to work with stored data almost as if you were using Scala collections while at the same time giving you full control over when a database access happens and which data is transferred. ...
- Spring Data
It makes it easy to use data access technologies, relational and non-relational databases, map-reduce frameworks, and cloud-based data services. This is an umbrella project which contains many subprojects that are specific to a given database. ...
- DataGrip
A cross-platform IDE that is aimed at DBAs and developers working with SQL databases. ...
- DBeaver
It is a free multi-platform database tool for developers, SQL programmers, database administrators and analysts. Supports all popular databases: MySQL, PostgreSQL, SQLite, Oracle, DB2, SQL Server, Sybase, Teradata, MongoDB, Cassandra, Redis, etc. ...
dbt alternatives & related posts
related act posts
Airflow
- Features51
- Task Dependency Management14
- Beautiful UI12
- Cluster of workers12
- Extensibility10
- Open source6
- Complex workflows5
- Python5
- Good api3
- Apache project3
- Custom operators3
- Dashboard2
- Observability is not great when the DAGs exceed 2502
- Running it on kubernetes cluster relatively complex2
- Open source - provides minimum or no support2
- Logical separation of DAGs is not straight forward1
related Airflow posts
I am working on a project that grabs a set of input data from AWS S3, pre-processes and divvies it up, spins up 10K batch containers to process the divvied data in parallel on AWS Batch, post-aggregates the data, and pushes it to S3.
I already have software patterns from other projects for Airflow + Batch but have not dealt with the scaling factors of 10k parallel tasks. Airflow is nice since I can look at which tasks failed and retry a task after debugging. But dealing with that many tasks on one Airflow EC2 instance seems like a barrier. Another option would be to have one task that kicks off the 10k containers and monitors it from there.
I have no experience with AWS Step Functions but have heard it's AWS's Airflow. There looks to be plenty of patterns online for Step Functions + Batch. Do Step Functions seem like a good path to check out for my use case? Do you get the same insights on failing jobs / ability to retry tasks as you do with Airflow?
I am looking for an open-source scheduler tool with cross-functional application dependencies. Some of the tasks I am looking to schedule are as follows:
- Trigger Matillion ETL loads
- Trigger Attunity Replication tasks that have downstream ETL loads
- Trigger Golden gate Replication Tasks
- Shell scripts, wrappers, file watchers
- Event-driven schedules
I have used Airflow in the past, and I know we need to create DAGs for each pipeline. I am not familiar with Jenkins, but I know it works with configuration without much underlying code. I want to evaluate both and appreciate any advise
- Real time in app customer chat support4
- GitHub integration4
- Reduces the barrier of entry to utilizing data1
- Price3
related Looker posts
We are a consumer mobile app IOS/Android startup. The app is instrumented with branch and Firebase. We use Google BigQuery. We are looking at tools that can support engagement and cohort analysis at an early stage price which we can grow with. Data Studio is the default but it would seem Looker provides more power. We don't have much insight into Amplitude other than the fact it is a popular PM tool. Please provide some insight.
- Open-source61
- Fast and Flexible48
- One platform for every big data problem8
- Great for distributed SQL like applications8
- Easy to install and to use6
- Works well for most Datascience usecases3
- Interactive Query2
- Machine learning libratimery, Streaming in real2
- In memory Computation2
- Speed4
related Apache Spark posts
The algorithms and data infrastructure at Stitch Fix is housed in #AWS. Data acquisition is split between events flowing through Kafka, and periodic snapshots of PostgreSQL DBs. We store data in an Amazon S3 based data warehouse. Apache Spark on Yarn is our tool of choice for data movement and #ETL. Because our storage layer (s3) is decoupled from our processing layer, we are able to scale our compute environment very elastically. We have several semi-permanent, autoscaling Yarn clusters running to serve our data processing needs. While the bulk of our compute infrastructure is dedicated to algorithmic processing, we also implemented Presto for adhoc queries and dashboards.
Beyond data movement and ETL, most #ML centric jobs (e.g. model training and execution) run in a similarly elastic environment as containers running Python and R code on Amazon EC2 Container Service clusters. The execution of batch jobs on top of ECS is managed by Flotilla, a service we built in house and open sourced (see https://github.com/stitchfix/flotilla-os).
At Stitch Fix, algorithmic integrations are pervasive across the business. We have dozens of data products actively integrated systems. That requires serving layer that is robust, agile, flexible, and allows for self-service. Models produced on Flotilla are packaged for deployment in production using Khan, another framework we've developed internally. Khan provides our data scientists the ability to quickly productionize those models they've developed with open source frameworks in Python 3 (e.g. PyTorch, sklearn), by automatically packaging them as Docker containers and deploying to Amazon ECS. This provides our data scientist a one-click method of getting from their algorithms to production. We then integrate those deployments into a service mesh, which allows us to A/B test various implementations in our product.
For more info:
- Our Algorithms Tour: https://algorithms-tour.stitchfix.com/
- Our blog: https://multithreaded.stitchfix.com/blog/
- Careers: https://multithreaded.stitchfix.com/careers/
#DataScience #DataStack #Data
Why we built Marmaray, an open source generic data ingestion and dispersal framework and library for Apache Hadoop :
Built and designed by our Hadoop Platform team, Marmaray is a plug-in-based framework built on top of the Hadoop ecosystem. Users can add support to ingest data from any source and disperse to any sink leveraging the use of Apache Spark . The name, Marmaray, comes from a tunnel in Turkey connecting Europe and Asia. Similarly, we envisioned Marmaray within Uber as a pipeline connecting data from any source to any sink depending on customer preference:
https://eng.uber.com/marmaray-hadoop-ingestion-open-source/
(Direct GitHub repo: https://github.com/uber/marmaray Kafka Kafka Manager )
related Slick posts
Spring Data
related Spring Data posts
I need some advice to choose an engine for generation web pages from the Spring Boot app. Which technology is the best solution today? 1) JSP + JSTL 2) Apache FreeMarker 3) Thymeleaf Or you can suggest even other perspective tools. I am using Spring Boot, Spring Web, Spring Data, Spring Security, PostgreSQL, Apache Tomcat in my project. I have already tried to generate pages using jsp, jstl, and it went well. However, I had huge problems via carrying already created static pages, to jsp format, because of syntax. Thanks.
- Works on Linux, Windows and MacOS4
- Code analysis3
- Diff viewer2
- Wide range of DBMS support2
- Generate ERD1
- Quick-fixes using keyboard shortcuts1
- Database introspection on 21 different dbms1
- Export data using a variety of formats using open api1
- Import data1
- Code completion1
related DataGrip posts
- Free21
- Platform independent13
- Automatic driver download9
- Import-Export Data7
- Simple to use6
- Move data between databases4
- Wide range of DBMS support4
- SAP Hana DB support1
- Themes1
related DBeaver posts
Which tools are preferred if I choose to work on more data side? Which one is good if I decide to work on web development? I'm using DBeaver and am now considering a move to AzureDataStudio to break the monotony while working. I would like to hear your opinion. Which one are you using, and what are the things you are missing in dbeaver or data studio.