Skip to main content

Dynamic Mapping & Collect

These APIs provide the means for a simple kind of dynamic orchestration — where the work to be orchestrated is determined not at job definition time but at runtime, dependent on data that’s observed as part of job execution.

class dagster.DynamicOut

Variant of OutOut for an output that will dynamically alter the graph at runtime.

When using in a composition function such as @graph@graph, dynamic outputs must be used with either

  • map - clone downstream ops for each separate DynamicOutDynamicOut
  • collect - gather across all DynamicOutDynamicOut in to a list

Uses the same constructor as OutOut

@op(
    config_schema=\{
        "path": Field(str, default_value=file_relative_path(__file__, "sample"))
    },
    out=DynamicOut(str),
)
def files_in_directory(context):
    path = context.op_config["path"]
    dirname, _, filenames = next(os.walk(path))
    for file in filenames:
        yield DynamicOutput(os.path.join(dirname, file), mapping_key=_clean(file))

@job
def process_directory():
    files = files_in_directory()

    # use map to invoke an op on each dynamic output
    file_results = files.map(process_file)

    # use collect to gather the results in to a list
    summarize_directory(file_results.collect())
class dagster.DynamicOutput

Variant of OutputOutput used to support dynamic mapping & collect. Each DynamicOutput produced by an op represents one item in a set that can be processed individually with map or gathered with collect.

Each DynamicOutput must have a unique mapping_key to distinguish it with it’s set.

Parameters:

  • value (Any) – The value returned by the compute function.
  • mapping_key (str) – The key that uniquely identifies this dynamic value relative to its peers.
  • output_name (Optional[str]) – Name of the corresponding DynamicOutDynamicOut defined on the op.
  • metadata (Optional[Dict[str, Union[str, float, int, MetadataValueMetadataValue]]]) – Arbitrary metadata about the failure. Keys are displayed string labels, and values are
property mapping_key

The mapping_key that was set for this DynamicOutput at instantiation.

property output_name

Name of the DynamicOutDynamicOut defined on the op that this DynamicOut is associated with.

property value

The value that is returned by the compute function for this DynamicOut.