Skip to main content

Dagster & OpenAI

The dagster-openai library allows you to easily interact with the OpenAI REST API using the OpenAI Python API to build AI steps into your Dagster pipelines. You can also log OpenAI API usage metadata in Dagster Insights, giving you detailed observability on API call credit consumption.

When paired with Dagster assets, the resource automatically logs OpenAI usage metadata in asset metadata.

Installation​

pip install dagster dagster-openai

Example​

from dagster_openai import OpenAIResource

import dagster as dg


@dg.asset(compute_kind="OpenAI")
def openai_asset(context: dg.AssetExecutionContext, openai: OpenAIResource):
    with openai.get_client(context) as client:
        client.chat.completions.create(
            model="gpt-3.5-turbo",
            messages=[{"role": "user", "content": "Say this is a test."}],
        )


openai_asset_job = dg.define_asset_job(
    name="openai_asset_job", selection="openai_asset"
)

defs = dg.Definitions(
    assets=[openai_asset],
    jobs=[openai_asset_job],
    resources={
        "openai": OpenAIResource(api_key=dg.EnvVar("OPENAI_API_KEY")),
    },
)

About OpenAI​

OpenAI is a U.S. based artificial intelligence (AI) research organization with the goal of developing "safe and beneficial" artificial general intelligence, which it defines as "highly autonomous systems that outperform humans at most economically valuable work".