Orchestration on Docker

APIs

dagster_docker.DockerRunLauncher RunLauncher: Launches runs in a Docker container.

dagster_docker.docker_executor ExecutorDefinition

experimental

This API may break in future versions, even between dot releases.

Executor which launches steps as Docker containers.

To use the docker_executor, set it as the executor_def when defining a job:

from dagster_docker import docker_executor

from dagster import job

@job(executor_def=docker_executor)
def docker_job():
    pass

Then you can configure the executor with run config as follows:

execution:
  config:
    registry: ...
    network: ...
    networks: ...
    container_kwargs: ...

If you’re using the DockerRunLauncher, configuration set on the containers created by the run launcher will also be set on the containers that are created for each step.

Ops

dagster_docker.docker_container_op = <dagster._core.definitions.op_definition.OpDefinition object>

experimental

This API may break in future versions, even between dot releases.

An op that runs a Docker container using the docker Python API.

Contrast with the docker_executor, which runs each Dagster op in a Dagster job in its own Docker container.

This op may be useful when:

You need to orchestrate a command that isn’t a Dagster op (or isn’t written in Python)
You want to run the rest of a Dagster job using a specific executor, and only a single

For example:

from dagster_docker import docker_container_op

from dagster import job

first_op = docker_container_op.configured(
    \{
        "image": "busybox",
        "command": ["echo HELLO"],
    },
    name="first_op",
)
second_op = docker_container_op.configured(
    \{
        "image": "busybox",
        "command": ["echo GOODBYE"],
    },
    name="second_op",
)

@job
def full_job():
    second_op(first_op())

You can create your own op with the same implementation by calling the execute_docker_container function inside your own op.

dagster_docker.execute_docker_container

experimental

This API may break in future versions, even between dot releases.

This function is a utility for executing a Docker container from within a Dagster op.

Parameters:

image (str) – The image to use for the launched Docker container.
entrypoint (Optional[Sequence[str]]) – The ENTRYPOINT to run in the launched Docker
command (Optional[Sequence[str]]) – The CMD to run in the launched Docker container.
networks (Optional[Sequence[str]]) – Names of the Docker networks to which to connect the
registry – (Optional[Mapping[str, str]]): Information for using a non local/public Docker
env_vars (Optional[Sequence[str]]) – List of environemnt variables to include in the launched
container_kwargs (Optional[Dict[str[Any]]]) – key-value pairs that can be passed into

Pipes

class dagster_docker.PipesDockerClient

experimental

This API may break in future versions, even between dot releases.

A pipes client that runs external processes in docker containers.

By default context is injected via environment variables and messages are parsed out of the log stream, with other logs forwarded to stdout of the orchestration process.

Parameters:

env (Optional[Mapping[str, str]]) – An optional dict of environment variables to pass to the
register (Optional[Mapping[str, str]]) – An optional dict of registry credentials to login to
context_injector (Optional[PipesContextInjectorPipesContextInjector]) – A context injector to use to inject
message_reader (Optional[PipesMessageReaderPipesMessageReader]) – A message reader to use to read messages

class dagster_docker.PipesDockerLogsMessageReader: experimental
This API may break in future versions, even between dot releases.

APIs

Ops​

Pipes​

Ops

Pipes