Skip to main content

Adding tags and metadata to assets

Assets feature prominently in the Dagster UI. Attaching information to assets allows you to understand where they're stored, what they contain, and how they should be organized.

Using metadata in Dagster, you can:

  • Attach ownership information
  • Organize assets with tags
  • Attach rich, complex information such as a Markdown description, a table schema, or a time series
  • Link assets with their source code
Prerequisites

To follow the steps in this guide, you'll need:

Adding owners to assets​

In a large organization, it's important to know which individuals and teams are responsible for a given data asset:

Loading...

owners must either be an email address or a team name prefixed by team:.

tip

With Dagster+ Pro, you can create asset-based alerts that automatically notify an asset's owners when triggered. Refer to the Dagster+ alert documentation for more information.

Organizing assets with tags​

Tags are the primary way to organize assets in Dagster. You can attach several tags to an asset when it's defined, and they will appear in the UI. You can also use tags to search and filter for assets in the Asset catalog. They're structured as key-value pairs of strings.

Here's an example of some tags you might apply to an asset:

{"domain": "marketing", "pii": "true"}

Like owners, just pass a dictionary of tags to the tags argument when defining an asset:

Loading...

Keep in mind that tags must contain only strings as keys and values. Additionally, the Dagster UI will render tags with the empty string as a "label" rather than a key-value pair.

Attaching metadata to assets​

Metadata allows you to attach rich information to the asset, like a Markdown description, a table schema, or a time series. Metadata is more flexible than tags, as it can store more complex information.

Metadata can be attached to an asset at definition time, when the code is first imported, or at runtime when an asset is materialized.

At definition time​

Using definition metadata to describe assets can make it easy to provide context for you and your team. This metadata could be descriptions of the assets, the types of assets, or links to relevant documentation.

Loading...

To learn more about the different types of metadata you can attach, see the MetadataValue API docs.

Some metadata keys will be given special treatment in the Dagster UI. See the Standard metadata types section for more information.

At runtime​

With runtime metadata, you can surface information about an asset's materialization, such as how many records were processed or when the materialization occurred. This allows you to update an asset's information when it changes and track historical metadata as a time series.

Loading...

Numerical metadata is treated as a time series in the Dagster UI.

Standard metadata types​

The following metadata keys are given special treatment in the Dagster UI.

KeyDescription
dagster/uriType: str

The URI for the asset, for example: "s3://my_bucket/my_object"
dagster/column_schemaType: TableSchema

For an asset that's a table, the schema of the columns in the table. Refer to the Table and column metadata section for details.
dagster/column_lineageType: TableColumnLineage

For an asset that's a table, the lineage of column inputs to column outputs for the table. Refer to the Table and column metadata section for details.
dagster/row_countType: int

For an asset that's a table, the number of rows in the table. Refer to the Table metadata documentation for details.
dagster/partition_row_countType: int

For a partition of an asset that's a table, the number of rows in the partition.
dagster/table_nameType: str

A unique identifier for the table/view, typically fully qualified. For example, my_database.my_schema.my_table
dagster/code_referencesType: CodeReferencesMetadataValue

A list of code references for the asset, such as file locations or references to GitHub URLs. Refer to the Linking assets with their source code section for details. Should only be provided in definition-level metadata, not materialization metadata.

Table and column metadata​

Two of the most powerful metadata types are TableSchema and TableColumnLineage. These metadata types allow stakeholders to view the schema of a table right within Dagster, and, in Dagster+, navigate the Asset catalog with the column lineage.

Table schema metadata​

The following example attaches table and column schema metadata at both definition time and runtime:

Loading...

There are several data types and constraints available on TableColumn objects. Refer to the API documentation for more information.

Column lineage metadata​

tip

Many integrations such as dbt automatically attach column lineage metadata out-of-the-box.

Column lineage metadata is a powerful way to track how columns in a table are derived from other columns:

Table column lineage metadata
Loading...
tip

Dagster+ provides rich visualization and navigation of column lineage in the Asset catalog. Refer to the Dagster+ documentation for more information.

Linking assets with source code​

warning

This feature is considered experimental and is under active development.

To link assets with their source code, you can attach a code reference. Code references are a type of metadata that allow you to easily view those assets' source code from the Dagster UI, both in local development and in production.

tip

Many integrations such as dbt support this capability out of the box.

Attaching Python code references for local development​

Dagster can automatically attach code references to assets during local development with one line of code:

Loading...

Customizing code references​

If you want to customize how code references are attached - such as when you are building domain-specific languages with asset factories - you can manually add the dagster/code_references metadata to asset definitions:

Loading...

Attaching code references in production​

Dagster+ can automatically annotate assets with code references to source control, such as GitHub or GitLab.

Loading...