Go Even Further with dbt
Discover how to enhance your dbt projects using advanced features like state-based selection for efficient model runs, environment variables for state management, and model versioning to handle breaking changes safely. Learn to define, reference, and deprecate multiple model versions while maintaining stable pipelines.
We'll cover the following...
Let’s take a look at a few features that can take our dbt game to the next level!
State comparison
dbt provides a powerful feature called state-based selection. This feature allows us to run only the models that have changed compared to a previous version.
manifest.json
To identify changes, dbt compares our current manifest.json file to a previous version. manifest.json is a single file, stored in the target directory, that contains information about our entire dbt project. This file is used both for state comparison and generating documentation.
Every time a user runs dbt run, dbt ls, or dbt compile, the manifest.json file is updated.
The DBT_STATE environment
dbt must compare our current project to a previous version. To do so, we need to give dbt a previous version of the manifest.json file. A common way to do so is to copy that file when a pull request is made to the main branch. This way, users can compare their changes to the main branch.
Once we’ve stored a copy of the previous manifest.json file, we need to tell dbt where that file is located. We can do so by setting the DBT_STATE environment variable and pass it the folder that contains the previous manifest.json file:
Using state in the CLI
Once we’ve set up our comparison manifest, we can use the state selection method. To select all resources that were ...