Resources

Pythonic resource system

The following classes are used as part of the new Pythonic resources system.

class dagster.ConfigurableResource

Base class for Dagster resources that utilize structured config.

This class is a subclass of both ResourceDefinitionResourceDefinition and ConfigConfig.

Example definition:

class WriterResource(ConfigurableResource):
    prefix: str

    def output(self, text: str) -> None:
        print(f"\{self.prefix}\{text}")

Example usage:

@asset
def asset_that_uses_writer(writer: WriterResource):
    writer.output("text")

defs = Definitions(
    assets=[asset_that_uses_writer],
    resources=\{"writer": WriterResource(prefix="a_prefix")},
)

You can optionally use this class to model configuration only and vend an object of a different type for use at runtime. This is useful for those who wish to have a separate object that manages configuration and a separate object at runtime. Or where you want to directly use a third-party class that you do not control.

To do this you override the create_resource methods to return a different object.

class WriterResource(ConfigurableResource):
    str: prefix

    def create_resource(self, context: InitResourceContext) -> Writer:
        # Writer is pre-existing class defined else
        return Writer(self.prefix)

Example usage:

@asset
def use_preexisting_writer_as_resource(writer: ResourceParam[Writer]):
    writer.output("text")

defs = Definitions(
    assets=[use_preexisting_writer_as_resource],
    resources=\{"writer": WriterResource(prefix="a_prefix")},
)

class dagster.ResourceDefinition

Core class for defining resources.

Resources are scoped ways to make external resources (like database connections) available to ops and assets during job execution and to clean up after execution resolves.

If resource_fn yields once rather than returning (in the manner of functions decorable with @contextlib.contextmanager) then the body of the function after the yield will be run after execution resolves, allowing users to write their own teardown/cleanup logic.

Depending on your executor, resources may be instantiated and cleaned up more than once in a job execution.

Parameters:

• resource_fn (Callable[[InitResourceContextInitResourceContext], Any]) – User-provided function to instantiate
• config_schema (Optional[ConfigSchemaConfigSchema) – The schema for the config. If set, Dagster will check
• description (Optional[str]) – A human-readable description of the resource.
• required_resource_keys – (Optional[Set[str]]) Keys for the resources required by this
• version (Optional[str]) – experimental

static hardcoded_resource

A helper function that creates a ResourceDefinition with a hardcoded object.

Parameters:

• value (Any) – The value that will be accessible via context.resources.resource_name.
• description ([Optional[str]]) – The description of the resource. Defaults to None.

Returns: A hardcoded resource.Return type: [ResourceDefinition]

static mock_resource

A helper function that creates a ResourceDefinition which wraps a mock.MagicMock.

Parameters: description ([Optional[str]]) – The description of the resource. Defaults to None.Returns: A resource that creates the magic methods automatically and helps you mock existing resources.

Return type: [ResourceDefinition]

static none_resource

A helper function that returns a none resource.

Parameters: description ([Optional[str]]) – The description of the resource. Defaults to None.Returns: A resource that does nothing.Return type: [ResourceDefinition]

static string_resource

Creates a ResourceDefinition which takes in a single string as configuration and returns this configured string to any ops or assets which depend on it.

Parameters: description ([Optional[str]]) – The description of the string resource. Defaults to None.Returns: A resource that takes in a single string as configuration and returns that string.

Return type: [ResourceDefinition]

property description

A human-readable description of the resource.

property required_resource_keys

A set of the resource keys that this resource depends on. These keys will be made available to the resource’s init context during execution, and the resource will not be instantiated until all required resources are available.

property version

A string which can be used to identify a particular code version of a resource definition.

class dagster.InitResourceContext

The context object available as the argument to the initialization function of a dagster.ResourceDefinitiondagster.ResourceDefinition.

Users should not instantiate this object directly. To construct an InitResourceContext for testing purposes, use dagster.build_init_resource_context()dagster.build_init_resource_context().

Example:

from dagster import resource, InitResourceContext

@resource
def the_resource(init_context: InitResourceContext):
    init_context.log.info("Hello, world!")

property instance

The Dagster instance configured for the current execution context.

property log

The Dagster log manager configured for the current execution context.

property log_manager

The log manager for this run of the job.

property resource_config

The configuration data provided by the run config. The schema for this data is defined by the config_field argument to ResourceDefinitionResourceDefinition.

property resource_def

The definition of the resource currently being constructed.

property resources

The resources that are available to the resource that we are initializing.

property run_id

The id for this run of the job or pipeline. When initializing resources outside of execution context, this will be None.

dagster.make_values_resource

A helper function that creates a ResourceDefinition to take in user-defined values.

This is useful for sharing values between ops.

Parameters: **kwargs – Arbitrary keyword arguments that will be passed to the config schema of the returned resource definition. If not set, Dagster will accept any config provided for the resource. For example:

@op(required_resource_keys=\{"globals"})
def my_op(context):
    print(context.resources.globals["my_str_var"])

@job(resource_defs=\{"globals": make_values_resource(my_str_var=str, my_int_var=int)})
def my_job():
    my_op()

Returns: A resource that passes in user-defined values.Return type: ResourceDefinition

dagster.build_init_resource_context

Builds resource initialization context from provided parameters.

build_init_resource_context can be used as either a function or context manager. If there is a provided resource to build_init_resource_context that is a context manager, then it must be used as a context manager. This function can be used to provide the context argument to the invocation of a resource.

Parameters:

• resources (Optional[Dict[str, Any]]) – The resources to provide to the context. These can be
• config (Optional[Any]) – The resource config to provide to the context.
• instance (Optional[DagsterInstanceDagsterInstance]) – The dagster instance configured for the context.

Examples:

context = build_init_resource_context()
resource_to_init(context)

with build_init_resource_context(
    resources=\{"foo": context_manager_resource}
) as context:
    resource_to_init(context)

dagster.build_resources

Context manager that yields resources using provided resource definitions and run config.

This API allows for using resources in an independent context. Resources will be initialized with the provided run config, and optionally, dagster_run. The resulting resources will be yielded on a dictionary keyed identically to that provided for resource_defs. Upon exiting the context, resources will also be torn down safely.

Parameters:

• resources (Mapping[str, Any]) – Resource instances or definitions to build. All
• instance (Optional[DagsterInstanceDagsterInstance]) – The dagster instance configured to instantiate
• resource_config (Optional[Mapping[str, Any]]) – A dict representing the config to be
• dagster_run (Optional[PipelineRun]) – The pipeline run to provide during resource
• log_manager (Optional[DagsterLogManagerDagsterLogManager]) – Log Manager to use during resource

Examples:

from dagster import resource, build_resources

@resource
def the_resource():
    return "foo"

with build_resources(resources=\{"from_def": the_resource, "from_val": "bar"}) as resources:
    assert resources.from_def == "foo"
    assert resources.from_val == "bar"

dagster.with_resources

Adds dagster resources to copies of resource-requiring dagster definitions.

An error will be thrown if any provided definitions have a conflicting resource definition provided for a key provided to resource_defs. Resource config can be provided, with keys in the config dictionary corresponding to the keys for each resource definition. If any definition has unsatisfied resource keys after applying with_resources, an error will be thrown.

Parameters:

• definitions (Iterable[ResourceAddable]) – Dagster definitions to provide resources to.
• resource_defs (Mapping[str, object]) – Mapping of resource keys to objects to satisfy
• resource_config_by_key (Optional[Mapping[str, Any]]) – Specifies config for provided resources. The key in this dictionary

Examples:

from dagster import asset, resource, with_resources

@resource(config_schema=\{"bar": str})
def foo_resource():
    ...

@asset(required_resource_keys=\{"foo"})
def asset1(context):
    foo = context.resources.foo
    ...

@asset(required_resource_keys=\{"foo"})
def asset2(context):
    foo = context.resources.foo
    ...

asset1_with_foo, asset2_with_foo = with_resources(
    [asset1, asset2],
    resource_defs=\{
        "foo": foo_resource
    },
    resource_config_by_key=\{
        "foo": \{
            "config": \{"bar": ...}
        }
    }
)

Utilities

class dagster.EnvVar

Class used to represent an environment variable in the Dagster config system.

This class is intended to be used to populate config fields or resources. The environment variable will be resolved to a string value when the config is loaded.

To access the value of the environment variable, use the get_value method.

Legacy resource system

The following classes are used as part of the legacy resource system.

@dagster.resource

Define a resource.

The decorated function should accept an InitResourceContextInitResourceContext and return an instance of the resource. This function will become the resource_fn of an underlying ResourceDefinitionResourceDefinition.

If the decorated function yields once rather than returning (in the manner of functions decorable with @contextlib.contextmanager) then the body of the function after the yield will be run after execution resolves, allowing users to write their own teardown/cleanup logic.

Parameters:

• config_schema (Optional[ConfigSchemaConfigSchema]) – The schema for the config. Configuration data available in
• description (Optional[str]) – A human-readable description of the resource.
• version (Optional[str]) – (Experimental) The version of a resource function. Two wrapped
• required_resource_keys (Optional[Set[str]]) – Keys for the resources required by this resource.