Orchestration Tool: dbt
Learn about dbt, a SQL-first data transformation tool.
Both Airflow and Dagster are great data orchestration tools, but they require significant engineering efforts. It can become a bottleneck for data analysts and analytics engineers to build data transformation pipelines using SQL as the primary language. In this chapter, it would be a shame not to mention dbt, whose revolution has transformed how data is managed, transformed, and analyzed, leading to high data quality and greater efficiency and collaboration within data teams. To some extent, dbt is a transformation workflow, rather than a pure orchestration tool.
History
Dbt was created by Tristan Handy in 2016 while he was working as a data analyst at RJMetric, a SaaS-based data analytics company. He built dbt, which was an internal tool back then to address complex data transformation pipelines. In 2018, Handy founded Fishtown Analytics, a data engineering and analytics consulting company, with the goal of helping other organizations improve data analytics workflows.
Since then, dbt has become a widely adopted open-source tool, with a growing community of users and contributors. It has been used by companies of all sizes to manage their data transformation pipelines and improve their data analytics workflows.
Dbt products
At the time of writing this course, dbt offers two types of products: dbt Core and dbt Cloud. The first product, dbt Core, is an open-source tool enabling data teams to transform data using best engineering practices. We can install dbt Core with its ...