Metadata

Dagster uses metadata to communicate arbitrary user-specified metadata about structured events.

Refer to the Metadata documentation for more information.

Utility class to wrap metadata values passed into Dagster events so that they can be displayed in the Dagster UI and other tooling.

@op
def emit_metadata(context, df):
    yield AssetMaterialization(
        asset_key="my_dataset",
        metadata=\{
            "my_text_label": "hello",
            "dashboard_url": MetadataValue.url("http://mycoolsite.com/my_dashboard"),
            "num_rows": 0,
        },
    )

static asset

Static constructor for a metadata value referencing a Dagster asset, by key.

For example:

@op
def validate_table(context, df):
    yield AssetMaterialization(
        asset_key=AssetKey("my_table"),
        metadata=\{
            "Related asset": MetadataValue.asset(AssetKey('my_other_table')),
        },
    )

Parameters: asset_key (AssetKeyAssetKey) – The asset key referencing the asset.

static bool

Static constructor for a metadata value wrapping a bool as BoolMetadataValuye. Can be used as the value type for the metadata parameter for supported events.

Example:

@op
def emit_metadata(context, df):
    yield AssetMaterialization(
        asset_key="my_dataset",
        metadata=\{
            "num rows > 1000": MetadataValue.bool(len(df) > 1000),
        },
    )

Parameters: value (bool) – The bool value for a metadata entry.

static column_lineage

Static constructor for a metadata value wrapping a column lineage as TableColumnLineageMetadataValueTableColumnLineageMetadataValue. Can be used as the value type for the metadata parameter for supported events.

Parameters: lineage (TableColumnLineageTableColumnLineage) – The column lineage for a metadata entry.

static dagster_run

Static constructor for a metadata value wrapping a reference to a Dagster run.

Parameters: run_id (str) – The ID of the run.

static float

Static constructor for a metadata value wrapping a float as FloatMetadataValueFloatMetadataValue. Can be used as the value type for the metadata parameter for supported events.

Example:

@op
def emit_metadata(context, df):
    yield AssetMaterialization(
        asset_key="my_dataset",
        metadata=\{
            "size (bytes)": MetadataValue.float(calculate_bytes(df)),
        }
    )

Parameters: value (float) – The float value for a metadata entry.

static int

Static constructor for a metadata value wrapping an int as IntMetadataValueIntMetadataValue. Can be used as the value type for the metadata parameter for supported events.

Example:

@op
def emit_metadata(context, df):
    yield AssetMaterialization(
        asset_key="my_dataset",
        metadata=\{
            "number of rows": MetadataValue.int(len(df)),
        },
    )

Parameters: value (int) – The int value for a metadata entry.

static job

Static constructor for a metadata value referencing a Dagster job, by name.

For example:

@op
def emit_metadata(context, df):
    yield AssetMaterialization(
        asset_key="my_dataset"
        metadata=\{
            "Producing job": MetadataValue.job('my_other_job'),
        },
    )

Parameters:

job_name (str) – The name of the job.
location_name (Optional[str]) – The code location name for the job.
repository_name (Optional[str]) – The repository name of the job, if different from the

static json

Static constructor for a metadata value wrapping a json-serializable list or dict as JsonMetadataValueJsonMetadataValue. Can be used as the value type for the metadata parameter for supported events.

Example:

@op
def emit_metadata(context):
    yield ExpectationResult(
        success=not missing_things,
        label="is_present",
        metadata=\{
            "about my dataset": MetadataValue.json(\{"missing_columns": missing_things})
        },
    )

Parameters: data (Union[Sequence[Any], Mapping[str, Any]]) – The JSON data for a metadata entry.

static md

Static constructor for a metadata value wrapping markdown data as MarkdownMetadataValueMarkdownMetadataValue. Can be used as the value type for the metadata parameter for supported events.

Example:

@op
def emit_metadata(context, md_str):
    yield AssetMaterialization(
        asset_key="info",
        metadata=\{
            'Details': MetadataValue.md(md_str)
        },
    )

Parameters: md_str (str) – The markdown for a metadata entry.

static notebook

Static constructor for a metadata value wrapping a notebook path as NotebookMetadataValueNotebookMetadataValue.

Example:

@op
def emit_metadata(context):
    yield AssetMaterialization(
        asset_key="my_dataset",
        metadata=\{
            "notebook_path": MetadataValue.notebook("path/to/notebook.ipynb"),
        }
    )

Parameters: path (str) – The path to a notebook for a metadata entry.

static null: Static constructor for a metadata value representing null. Can be used as the value type for the metadata parameter for supported events.

static path

Static constructor for a metadata value wrapping a path as PathMetadataValuePathMetadataValue.

Example:

@op
def emit_metadata(context):
    yield AssetMaterialization(
        asset_key="my_dataset",
        metadata=\{
            "filepath": MetadataValue.path("path/to/file"),
        }
    )

Parameters: path (str) – The path for a metadata entry.

static python_artifact

Static constructor for a metadata value wrapping a python artifact as PythonArtifactMetadataValuePythonArtifactMetadataValue. Can be used as the value type for the metadata parameter for supported events.

Example:

@op
def emit_metadata(context, df):
    yield AssetMaterialization(
        asset_key="my_dataset",
        metadata=\{
            "class": MetadataValue.python_artifact(MyClass),
            "function": MetadataValue.python_artifact(my_function),
        }
    )

Parameters: value (Callable) – The python class or function for a metadata entry.

static table

experimental

This API may break in future versions, even between dot releases.

Static constructor for a metadata value wrapping arbitrary tabular data as TableMetadataValueTableMetadataValue. Can be used as the value type for the metadata parameter for supported events.

Example:

@op
def emit_metadata(context):
    yield ExpectationResult(
        success=not has_errors,
        label="is_valid",
        metadata=\{
            "errors": MetadataValue.table(
                records=[
                    TableRecord(code="invalid-data-type", row=2, col="name"),
                ],
                schema=TableSchema(
                    columns=[
                        TableColumn(name="code", type="string"),
                        TableColumn(name="row", type="int"),
                        TableColumn(name="col", type="string"),
                    ]
                )
            ),
        },
    )

static table_schema

Static constructor for a metadata value wrapping a table schema as TableSchemaMetadataValueTableSchemaMetadataValue. Can be used as the value type for the metadata parameter for supported events.

Example:

schema = TableSchema(
    columns = [
        TableColumn(name="id", type="int"),
        TableColumn(name="status", type="bool"),
    ]
)

DagsterType(
    type_check_fn=some_validation_fn,
    name='MyTable',
    metadata=\{
        'my_table_schema': MetadataValue.table_schema(schema),
    }
)

Parameters: schema (TableSchemaTableSchema) – The table schema for a metadata entry.

static text

Static constructor for a metadata value wrapping text as TextMetadataValueTextMetadataValue. Can be used as the value type for the metadata parameter for supported events.

Example:

@op
def emit_metadata(context, df):
    yield AssetMaterialization(
        asset_key="my_dataset",
        metadata=\{
            "my_text_label": MetadataValue.text("hello")
        },
    )

Parameters: text (str) – The text string for a metadata entry.

static timestamp

Static constructor for a metadata value wrapping a UNIX timestamp as a TimestampMetadataValueTimestampMetadataValue. Can be used as the value type for the metadata parameter for supported events.

Parameters: value (Union[float, datetime]) – The unix timestamp value for a metadata entry. If a datetime is provided, the timestamp will be extracted. datetimes without timezones are not accepted, because their timestamps can be ambiguous.

static url

Static constructor for a metadata value wrapping a URL as UrlMetadataValueUrlMetadataValue. Can be used as the value type for the metadata parameter for supported events.

Example:

@op
def emit_metadata(context):
    yield AssetMaterialization(
        asset_key="my_dashboard",
        metadata=\{
            "dashboard_url": MetadataValue.url("http://mycoolsite.com/my_dashboard"),
        }
    )

Parameters: url (str) – The URL for a metadata entry.

abstract property value: The wrapped value.

class dagster.MetadataEntry

deprecated

This API will be removed in version 2.0. Please use a dict with MetadataValue values instead..

A structure for describing metadata for Dagster events.

Note: This class is no longer usable in any Dagster API, and will be completely removed in 2.0.

Lists of objects of this type can be passed as arguments to Dagster events and will be displayed in the Dagster UI and other tooling.

Should be yielded from within an IO manager to append metadata for a given input/output event. For other event types, passing a dict with MetadataValue values to the metadata argument is preferred.

Parameters:

label (str) – Short display label for this metadata entry.
description (Optional[str]) – A human-readable description of this metadata entry.
value (MetadataValueMetadataValue) – Typed metadata entry data. The different types allow

Metadata types

All metadata types inherit from MetadataValue. The following types are defined:

class dagster.DagsterAssetMetadataValue

Representation of a dagster asset.

Parameters: asset_key (AssetKeyAssetKey) – The dagster asset key

property value

The wrapped AssetKeyAssetKey.

Type: AssetKey

class dagster.DagsterRunMetadataValue

Representation of a dagster run.

Parameters: run_id (str) – The run id

property value

The wrapped run id.

Type: str

class dagster.FloatMetadataValue

Container class for float metadata entry data.

Parameters: value (Optional[float]) – The float value.

class dagster.IntMetadataValue

Container class for int metadata entry data.

Parameters: value (Optional[int]) – The int value.

class dagster.JsonMetadataValue

Container class for JSON metadata entry data.

Parameters: data (Union[Sequence[Any], Dict[str, Any]]) – The JSON data.

property value

The wrapped JSON data.

Type: Optional[Union[Sequence[Any], Dict[str, Any]]]

class dagster.MarkdownMetadataValue

Container class for markdown metadata entry data.

Parameters: md_str (Optional[str]) – The markdown as a string.

property value

The wrapped markdown as a string.

Type: Optional[str]

class dagster.PathMetadataValue

Container class for path metadata entry data.

Parameters: path (Optional[str]) – The path as a string or conforming to os.PathLike.

property value

The wrapped path.

Type: Optional[str]

class dagster.NotebookMetadataValue

Container class for notebook metadata entry data.

Parameters: path (Optional[str]) – The path to the notebook as a string or conforming to os.PathLike.

property value

The wrapped path to the notebook as a string.

Type: Optional[str]

class dagster.PythonArtifactMetadataValue

Container class for python artifact metadata entry data.

Parameters:

module (str) – The module where the python artifact can be found
name (str) – The name of the python artifact

property value

Identity function.

Type: PythonArtifactMetadataValue

class dagster.TableColumnLineageMetadataValue

Representation of the lineage of column inputs to column outputs of arbitrary tabular data.

Parameters: column_lineage (TableColumnLineageTableColumnLineage) – The lineage of column inputs to column outputs for the table.

property value

The wrapped TableSpec.

Type: TableSpec

class dagster.TableMetadataValue

experimental

This API may break in future versions, even between dot releases.

Container class for table metadata entry data.

Parameters:

records (TableRecordTableRecord) – The data as a list of records (i.e. rows).
schema (Optional[TableSchemaTableSchema]) – A schema for the table.

Example:

from dagster import TableMetadataValue, TableRecord

TableMetadataValue(
    schema=None,
    records=[
        TableRecord(\{"column1": 5, "column2": "x"}),
        TableRecord(\{"column1": 7, "column2": "y"}),
    ]
)

static infer_column_type: str: Infer the TableSchemaTableSchema column type that will be used for a value.

property value

Identity function.

Type: TableMetadataValue

class dagster.TableSchemaMetadataValue

Representation of a schema for arbitrary tabular data.

Parameters: schema (TableSchemaTableSchema) – The dictionary containing the schema representation.

property value

The wrapped TableSchemaTableSchema.

Type: TableSchema

class dagster.TextMetadataValue

Container class for text metadata entry data.

Parameters: text (Optional[str]) – The text data.

property value

The wrapped text data.

Type: Optional[str]

class dagster.TimestampMetadataValue

Container class for metadata value that’s a unix timestamp.

Parameters: value (float) – Seconds since the unix epoch.

class dagster.UrlMetadataValue

Container class for URL metadata entry data.

Parameters: url (Optional[str]) – The URL as a string.

property value

The wrapped URL.

Type: Optional[str]

class dagster.CodeReferencesMetadataValue

experimental

This API may break in future versions, even between dot releases.

Metadata value type which represents source locations (locally or otherwise) of the asset in question. For example, the file path and line number where the asset is defined.

sources

A list of code references for the asset, such as file locations or references to source control.

Type: List[Union[LocalFileCodeReference, SourceControlCodeReference]]

Tables

These APIs provide the ability to express column schemas (TableSchema), rows/records (TableRecord), and column lineage (TableColumnLineage) in Dagster as metadata.

class dagster.TableRecord: experimental
This API may break in future versions, even between dot releases.
Represents one record in a table. Field keys are arbitrary strings– field values must be strings, integers, floats, or bools.

class dagster.TableSchema

Representation of a schema for tabular data.

Schema is composed of two parts:

A required list of columns (TableColumn). Each column specifies a

An optional list of table-level constraints (TableConstraints). A

# example schema
TableSchema(
    constraints = TableConstraints(
        other = [
            "foo > bar",
        ],
    ),
    columns = [
        TableColumn(
            name = "foo",
            type = "string",
            description = "Foo description",
            constraints = TableColumnConstraints(
                nullable = False,
                other = [
                    "starts with the letter 'a'",
                ],
            ),
        ),
        TableColumn(
            name = "bar",
            type = "string",
        ),
        TableColumn(
            name = "baz",
            type = "custom_type",
            constraints = TableColumnConstraints(
                unique = True,
            )
        ),
    ],
)

Parameters:

columns (List[TableColumnTableColumn]) – The columns of the table.
constraints (Optional[TableConstraintsTableConstraints]) – The constraints of the table.

static from_name_type_dict: Constructs a TableSchema from a dictionary whose keys are column names and values are the names of data types of those columns.

class dagster.TableConstraints

Descriptor for “table-level” constraints. Presently only one property, other is supported. This contains strings describing arbitrary table-level constraints. A table-level constraint is a constraint defined in terms of multiple columns (e.g. col_A > col_B) or in terms of rows.

Parameters: other (List[str]) – Descriptions of arbitrary table-level constraints.

class dagster.TableColumn

Descriptor for a table column. The only property that must be specified by the user is name. If no type is specified, string is assumed. If no constraints are specified, the column is assumed to be nullable (i.e. required = False) and have no other constraints beyond the data type.

Parameters:

name (List[str]) – Descriptions of arbitrary table-level constraints.
type (Optional[str]) – The type of the column. Can be an arbitrary
description (Optional[str]) – Description of this column. Defaults to None.
constraints (Optional[TableColumnConstraintsTableColumnConstraints]) – Column-level constraints.
tags (Optional[Mapping[str, str]]) – Tags for filtering or organizing columns.

class dagster.TableColumnConstraints

Descriptor for a table column’s constraints. Nullability and uniqueness are specified with boolean properties. All other constraints are described using arbitrary strings under the other property.

Parameters:

nullable (Optional[bool]) – If true, this column can hold null values.
unique (Optional[bool]) – If true, all values in this column must be unique.
other (List[str]) – Descriptions of arbitrary column-level constraints

class dagster.TableColumnLineage

experimental

This API may break in future versions, even between dot releases.

Represents the lineage of column outputs to column inputs for a tabular asset.

Parameters: deps_by_column (Mapping[str, Sequence[TableColumnDepTableColumnDep]]) – A mapping from column names to the columns that the column depends on. Examples:

Defining column lineage at materialization time, where the resulting asset has two columns, new_column_foo and new_column_qux. The first column, new_column_foo, depends on column_bar in source_bar and column_baz in source_baz. The second column, new_column_qux, depends on column_quuz in source_bar.

from dagster import (
    AssetKey,
    MaterializeResult,
    TableColumnDep,
    TableColumnLineage,
    asset,
)


@asset(deps=[AssetKey("source_bar"), AssetKey("source_baz")])
def my_asset():
    yield MaterializeResult(
        metadata=\{
            "dagster/column_lineage": TableColumnLineage(
                deps_by_column=\{
                    "new_column_foo": [
                        TableColumnDep(
                            asset_key=AssetKey("source_bar"),
                            column_name="column_bar",
                        ),
                        TableColumnDep(
                            asset_key=AssetKey("source_baz"),
                            column_name="column_baz",
                        ),
                    ],
                    "new_column_qux": [
                        TableColumnDep(
                            asset_key=AssetKey("source_bar"),
                            column_name="column_quuz",
                        ),
                    ],
                }
            )
        }
    )

class dagster.TableColumnDep: experimental
This API may break in future versions, even between dot releases.
Object representing an identifier for a column in an asset.

Code references

The following functions are used to attach source code references to your assets. For more information, refer to the Linking to asset definition code with code references guide.

dagster.with_source_code_references

experimental

This API may break in future versions, even between dot releases.

Wrapper function which attaches local code reference metadata to the provided asset definitions. This points to the filepath and line number where the asset body is defined.

Parameters: assets_defs (Sequence[Union[AssetsDefinitionAssetsDefinition, SourceAssetSourceAsset, CacheableAssetsDefinition]]) – The asset definitions to which source code metadata should be attached.Returns: The asset definitions with source code metadata attached.Return type: Sequence[AssetsDefinition]

dagster.link_code_references_to_git

experimental

This API may break in future versions, even between dot releases.

Wrapper function which converts local file path code references to source control URLs based on the provided source control URL and branch.

Parameters:

assets_defs (Sequence[Union[AssetsDefinitionAssetsDefinition, SourceAssetSourceAsset, CacheableAssetsDefinition]]) – The asset definitions to which source control metadata should be attached.
git_url (str) – The base URL for the source control system. For example,
git_branch (str) – The branch in the source control system, such as “master”.
file_path_mapping (FilePathMappingFilePathMapping) – Specifies the mapping between local file paths and their corresponding paths in a source control repository.

Example:

defs = Definitions(
    assets=link_code_references_to_git(
        with_source_code_references([my_dbt_assets]),
        git_url="https://github.com/dagster-io/dagster",
        git_branch="master",
        file_path_mapping=AnchorBasedFilePathMapping(
            local_file_anchor=Path(__file__),
            file_anchor_path_in_repository="python_modules/my_module/my-module/__init__.py",
        ),
    )
)

class dagster.FilePathMapping

experimental

This API may break in future versions, even between dot releases.

Base class which defines a file path mapping function. These functions are used to map local file paths to their corresponding paths in a source control repository.

In many cases where a source control repository is reproduced exactly on a local machine, the included AnchorBasedFilePathMapping class can be used to specify a direct mapping between the local file paths and the repository paths. However, in cases where the repository structure differs from the local structure, a custom mapping function can be provided to handle these cases.

abstract convert_to_source_control_path

Maps a local file path to the corresponding path in a source control repository.

Parameters: local_path (Path) – The local file path to map.Returns: The corresponding path in the hosted source control repository, relative to the repository root.Return type: str

class dagster.AnchorBasedFilePathMapping

experimental

This API may break in future versions, even between dot releases.

Specifies the mapping between local file paths and their corresponding paths in a source control repository, using a specific file “anchor” as a reference point. All other paths are calculated relative to this anchor file.

For example, if the chosen anchor file is /Users/dagster/Documents/python_modules/my_module/my-module/init.py locally, and python_modules/my_module/my-module/init.py in a source control repository, in order to map a different file /Users/dagster/Documents/python_modules/my_module/my-module/my_asset.py to the repository path, the mapping function will position the file in the repository relative to the anchor file’s position in the repository, resulting in python_modules/my_module/my-module/my_asset.py.

Parameters:

local_file_anchor (Path) – The path to a local file that is present in the repository.
file_anchor_path_in_repository (str) – The path to the anchor file in the repository.

Example:

mapping_fn = AnchorBasedFilePathMapping(
    local_file_anchor=Path(__file__),
    file_anchor_path_in_repository="python_modules/my_module/my-module/__init__.py",
)

convert_to_source_control_path

Maps a local file path to the corresponding path in a source control repository based on the anchor file and its corresponding path in the repository.

Parameters: local_path (Path) – The local file path to map.Returns: The corresponding path in the hosted source control repository, relative to the repository root.Return type: str

Metadata types​

Tables​

Code references​

Metadata types

Tables

Code references