Skip to main content

Dagster & SDF

SDF can integrate seamlessly with your existing Dagster projects, providing the best-in-class transformation layer while enabling you to schedule, orchestrate, and monitor your dags in Dagster.

When it comes time to materialize your Dagster assets, you can be confident that SDF has successfully compiled your workspace, making it safe to execute locally or against your cloud data warehouse.

Installation​

pip install dagster-sdf

Example​

from pathlib import Path

from dagster_sdf import SdfCliResource, SdfWorkspace, sdf_assets

import dagster as dg

workspace_dir = Path(__file__).joinpath("..", "./my_sdf_workspace").resolve()
target_dir = workspace_dir.joinpath(
    "sdf_dagster_out"
)  # The destination for outputs generated by SDF during execution
environment = "dbg"  # Replace with your environment, e.g. "prod"

workspace = SdfWorkspace(
    workspace_dir=workspace_dir,
    target_dir=target_dir,
    environment=environment,
)


@sdf_assets(workspace=workspace)
def my_sdf_assets(context: dg.AssetExecutionContext, sdf: SdfCliResource):
    yield from sdf.cli(
        ["run", "--save", "info-schema"],
        target_dir=target_dir,
        environment=environment,
        context=context,
    ).stream()


defs = dg.Definitions(
    assets=[my_sdf_assets],
    resources={
        "sdf": SdfCliResource(workspace_dir=workspace_dir),
    },
)

About SDF​

SDF is a multi-dialect SQL compiler, transformation framework, and analytical database engine. It natively compiles SQL dialects, like Snowflake, and connects to their corresponding data warehouses to materialize models.