Types

Dagster includes facilities for typing the input and output values of ops (“runtime” types).

Built-in types

Use this type only for inputs and outputs, in order to establish an execution dependency without communicating a value. Inputs of this type will not be passed to the op compute function, so it is necessary to use the explicit InIn API to define them rather than the Python 3 type hint syntax.

All values are considered to be instances of Nothing.

Examples:

@op
def wait(_) -> Nothing:
    time.sleep(1)
    return

@op(
    ins=\{"ready": In(dagster_type=Nothing)},
)
def done(_) -> str:
    return 'done'

@job
def nothing_job():
    done(wait())

# Any value will pass the type check for Nothing
@op
def wait_int(_) -> Int:
    time.sleep(1)
    return 1

@job
def nothing_int_job():
    done(wait_int())

Making New Types

class dagster.DagsterType

Define a type in dagster. These can be used in the inputs and outputs of ops.

Parameters:

type_check_fn (Callable[[TypeCheckContextTypeCheckContext, Any], [Union[bool, TypeCheckTypeCheck]]]) – The function that defines the type check. It takes the value flowing
key (Optional[str]) –

The unique key to identify types programmatically. The key property always has a value. If you omit key to the argument to the init function, it instead receives the value of name. If neither key nor name is provided, a CheckError is thrown.

In the case of a generic type such as List or Optional, this is generated programmatically based on the type parameters.
name (Optional[str]) – A unique name given by a user. If key is None, key
description (Optional[str]) – A markdown-formatted string, displayed in tooling.
loader (Optional[DagsterTypeLoaderDagsterTypeLoader]) – An instance of a class that
required_resource_keys (Optional[Set[str]]) – Resource keys required by the type_check_fn.
is_builtin (bool) – Defaults to False. This is used by tools to display or
kind (DagsterTypeKind) – Defaults to None. This is used to determine the kind of runtime type
typing_type – Defaults to None. A valid python typing type (e.g. Optional[List[int]]) for the

type_check

Type check the value against the type.

Parameters:

context (TypeCheckContextTypeCheckContext) – The context of the type check.
value (Any) – The value to check.

Returns: The result of the type check.Return type: TypeCheck

property description

Description of the type, or None if not provided.

Type: Optional[str]

property display_name: Either the name or key (if name is None) of the type, overridden in many subclasses.

property has_unique_name

Whether the type has a unique name.

Type: bool

property loader

Loader for this type, if any.

Type: Optional[DagsterTypeLoader]

property required_resource_keys

Set of resource keys required by the type check function.

Type: AbstractSet[str]

property typing_type

The python typing type for this type.

Type: Any

property unique_name: The unique name of this type. Can be None if the type is not unique, such as container types.

dagster.PythonObjectDagsterType

Define a type in dagster whose typecheck is an isinstance check.

Specifically, the type can either be a single python type (e.g. int), or a tuple of types (e.g. (int, float)) which is treated as a union.

Examples:

ntype = PythonObjectDagsterType(python_type=int)
assert ntype.name == 'int'
assert_success(ntype, 1)
assert_failure(ntype, 'a')

ntype = PythonObjectDagsterType(python_type=(int, float))
assert ntype.name == 'Union[int, float]'
assert_success(ntype, 1)
assert_success(ntype, 1.5)
assert_failure(ntype, 'a')

Parameters:

python_type (Union[Type, Tuple[Type, ...]) – The dagster typecheck function calls instanceof on
name (Optional[str]) – Name the type. Defaults to the name of python_type.
key (Optional[str]) – Key of the type. Defaults to name.
description (Optional[str]) – A markdown-formatted string, displayed in tooling.
loader (Optional[DagsterTypeLoaderDagsterTypeLoader]) – An instance of a class that

dagster.dagster_type_loader

Create an dagster type loader that maps config data to a runtime value.

The decorated function should take the execution context and parsed config value and return the appropriate runtime value.

Parameters: config_schema (ConfigSchemaConfigSchema) – The schema for the config that’s passed to the decorated function. Examples:

@dagster_type_loader(Permissive())
def load_dict(_context, value):
    return value

class dagster.DagsterTypeLoader

Dagster type loaders are used to load unconnected inputs of the dagster type they are attached to.

The recommended way to define a type loader is with the @dagster_type_loader@dagster_type_loader decorator.

class dagster.DagsterTypeLoaderContext

The context object provided to a @dagster_type_loader@dagster_type_loader-decorated function during execution.

Users should not construct this object directly.

property job_def: The underlying job definition being executed.

property op_def: The op for which type loading is occurring.

property resources: The resources available to the type loader, specified by the required_resource_keys argument of the decorator.

dagster.usable_as_dagster_type

Decorate a Python class to make it usable as a Dagster Type.

This is intended to make it straightforward to annotate existing business logic classes to make them dagster types whose typecheck is an isinstance check against that python class.

Parameters:

python_type (cls) – The python type to make usable as python type.
name (Optional[str]) – Name of the new Dagster type. If None, the name (__name__) of
description (Optional[str]) – A user-readable description of the type.
loader (Optional[DagsterTypeLoaderDagsterTypeLoader]) – An instance of a class that

Examples:

# dagster_aws.s3.file_manager.S3FileHandle
@usable_as_dagster_type
class S3FileHandle(FileHandle):
    def __init__(self, s3_bucket, s3_key):
        self._s3_bucket = check.str_param(s3_bucket, 's3_bucket')
        self._s3_key = check.str_param(s3_key, 's3_key')

    @property
    def s3_bucket(self):
        return self._s3_bucket

    @property
    def s3_key(self):
        return self._s3_key

    @property
    def path_desc(self):
        return self.s3_path

    @property
    def s3_path(self):
        return 's3://\{bucket}/\{key}'.format(bucket=self.s3_bucket, key=self.s3_key)

dagster.make_python_type_usable_as_dagster_type: Take any existing python type and map it to a dagster type (generally created with DagsterTypeDagsterType) This can only be called once on a given python type.

Testing Types

dagster.check_dagster_type

Test a custom Dagster type.

Parameters:

dagster_type (Any) – The Dagster type to test. Should be one of the
value (Any) – The runtime value to test.

Returns: The result of the type check.Return type: TypeCheck Examples:

assert check_dagster_type(Dict[Any, Any], \{'foo': 'bar'}).success

Built-in types​

Making New Types​

Testing Types​

Built-in types

Making New Types

Testing Types