Skip to main content

Dagster & DuckDB

This library provides an integration with the DuckDB database, and allows for an out-of-the-box I/O Manager so that you can make DuckDB your storage of choice.

Installation​

pip install dagster-duckdb

Example​

import pandas as pd
from dagster_duckdb_pandas import DuckDBPandasIOManager

import dagster as dg


@dg.asset(
    key_prefix=["my_schema"]  # will be used as the schema in duckdb
)
def my_table() -> pd.DataFrame:  # the name of the asset will be the table name
    return pd.DataFrame()


defs = dg.Definitions(
    assets=[my_table],
    resources={"io_manager": DuckDBPandasIOManager(database="my_db.duckdb")},
)

About DuckDB​

DuckDB is a column-oriented in-process OLAP database. A typical OLTP relational database like SQLite is row-oriented. In row-oriented database, data is organised physically as consecutive tuples.