Skip to main content

Airlift (dagster-airlift)

Core (dagster_airlift.core)​

AirflowInstance​

class dagster_airlift.core.AirflowInstance

A class that represents a running Airflow Instance and provides methods for interacting with its REST API.

Parameters:

  • auth_backend (AirflowAuthBackendAirflowAuthBackend) – The authentication backend to use when making requests to the Airflow instance.
  • name (str) – The name of the Airflow instance. This will be prefixed to any assets automatically created using this instance.
  • batch_task_instance_limit (int) – The number of task instances to query at a time when fetching task instances. Defaults to 100.
  • batch_dag_runs_limit (int) – The number of dag runs to query at a time when fetching dag runs. Defaults to 100.
class dagster_airlift.core.AirflowAuthBackend

An abstract class that represents an authentication backend for an Airflow instance.

Requires two methods to be implemented by subclasses:

  • get_session: Returns a requests.Session object that can be used to make requests to the Airflow instance, and handles authentication.
  • get_webserver_url: Returns the base URL of the Airflow webserver.

The dagster-airlift package provides the following default implementations:

  • dagster-airlift.core.AirflowBasicAuthBackend: An authentication backend that uses Airflow’s basic auth to authenticate with the Airflow instance.
  • dagster-airlift.mwaa.MwaaSessionAuthBackend: An authentication backend that uses AWS MWAA’s web login token to authenticate with the Airflow instance (requires dagster-airlift[mwaa]).
class dagster_airlift.core.AirflowBasicAuthBackend

A dagster_airlift.core.AirflowAuthBackenddagster_airlift.core.AirflowAuthBackend that authenticates using basic auth.

Parameters:

  • webserver_url (str) – The URL of the webserver.
  • username (str) – The username to authenticate with.
  • password (str) – The password to authenticate with.

Examples:

Creating a AirflowInstanceAirflowInstance using this backend.

from dagster_airlift.core import AirflowInstance, AirflowBasicAuthBackend

af_instance = AirflowInstance(
    name="my-instance",
    auth_backend=AirflowBasicAuthBackend(
        webserver_url="https://my-webserver-hostname",
        username="my-username",
        password="my-password"
    )
)

MWAA (dagster_airlift.mwaa)​

class dagster_airlift.mwaa.MwaaSessionAuthBackend

A dagster_airlift.core.AirflowAuthBackenddagster_airlift.core.AirflowAuthBackend that authenticates to AWS MWAA.

Under the hood, this class uses the MWAA boto3 session to request a web login token and then uses the token to authenticate to the MWAA web server.

Parameters:

  • mwaa_session (boto3.Session) – The boto3 MWAA session
  • env_name (str) – The name of the MWAA environment

Examples:

Creating an AirflowInstance pointed at a MWAA environment.

import boto3
from dagster_airlift.mwaa import MwaaSessionAuthBackend
from dagster_airlift.core import AirflowInstance

boto_session = boto3.Session(profile_name="my_profile", region_name="us-west-2")
af_instance = AirflowInstance(
    name="my-mwaa-instance",
    auth_backend=MwaaSessionAuthBackend(
        mwaa_session=boto_session,
        env_name="my-mwaa-env"
    )
)