Change tracking in branch deployments
This guide is applicable to Dagster+.
Branch Deployments Change Tracking makes it easier for you and your team to identify how changes in a pull request will impact data assets. By the end of this guide, you'll understand how Change Tracking works and what types of asset changes can be detected.
How it works
Branch Deployments compare asset definitions in the branch deployment against the asset definitions in the main deployment. The UI will then mark changed assets accordingly. For example, if the pull request associated with the branch deployment adds a new asset, the UI will display a label indicating the addition.
You can also apply filters to show only new and changed assets in the UI. This makes it easy to understand which assets will be impacted by the changes in the pull request associated with the branch deployment.
Note: The default main deployment is prod. To configure a different deployment as the main deployment, create a branch deployment using the dagster-cloud CLI and specify it using the optional --base-deployment-name parameter.
Supported change types
Change Tracking can detect the following changes to assets:
New assets
If an asset is new in the branch deployment, the asset will have a New in branch label in the asset graph:
Code versions
If using the code_version argument on the asset decorator, Change Tracking can detect when this value changes.
Some Dagster integrations, like dagster-dbt, automatically compute code versions for you. For more information on code versions, refer to the Code versioning guide.
- Asset in the Dagster UI
- Asset definition
In this example, the customers asset has a Changed in branch label indicating its code_version has been changed.
Click the Asset definition tab to see the code change that created this label.
In the main branch, we have a customers asset with a code version of v1:
@asset(code_version="v1")
def customers(): ...
In the pull request, customers is modified to change the code version to v2:
@asset(code_version="v2")
def customers(): ...
Upstream dependencies
Change Tracking can detect when an asset's upstream dependencies have changed, whether they've been added or removed.
Note: If an asset is marked as having changed dependencies, it means that the AssetKeys defining its upstream dependencies have changed. It doesn't mean that an upstream dependency has new data.
- Asset in the Dagster UI
In this example, the returns asset has a Changed in branch label indicating it has changed dependencies.
Click the Asset definition tab to see the code change that created this label.
@asset(deps=[orders, customers])
def returns(): ...
Partitions definitions
Change Tracking can detect if an asset's PartitionsDefinition has been changed, whether it's been added, removed, or updated.
- Asset in the Dagster UI
- Asset definition
In this example, the weekly_orders asset has a Changed in branch label indicating a changed partitions definition.
Click the Asset definition tab to see the code change that created this label.
In the main branch, we have a weekly_orders asset:
@asset(partitions_def=WeeklyPartitionsDefinition(start_date="2024-01-01"))
def weekly_orders(): ...
In the pull request, we updated the WeeklyPartitionsDefinition to start one year earlier:
@asset(partitions_def=WeeklyPartitionsDefinition(start_date="2023-01-01"))
def weekly_orders(): ...
Tags
Change Tracking can detect when an asset's tags have changed, whether they've been added, modified, or removed.
- Asset in the Dagster UI
- Asset definition
In this example, the fruits_in_stock asset has a Changed in branch label indicating it has changed tags.
Click the Asset definition tab to see the code change that created this label.
In the main branch, we have a fruits_in_stock asset:
@asset(tags={"section": "produce"})
def fruits_in_stock(): ...
In the pull request, we added the type: perishable tag to fruits_in_stock:
@asset(tags={"section": "produce", "type": "perishable"})
def fruits_in_stock(): ...
Metadata
Change Tracking can detect when an asset's definition metadata has changed, whether it's been added, modified, or removed.
- Asset in the Dagster UI
- Asset definition
In this example, the produtcs asset has a Changed in branch label indicating it has changed metadata.
Click the Asset definition tab to see the code change that created this label.
In the main branch, we have a products asset:
@asset(metadata={"expected_columns": ["sku", "price", "supplier"]})
def products(): ...
In the pull request, we update the value of the expected_columns metadata on products:
@asset(metadata={"expected_columns": ["sku", "price", "supplier", "backstock"]})
def products(): ...