Skip to main content

Environment variables

Environment variables, which are key-value pairs configured outside your source code, allow you to dynamically modify application behavior depending on environment.

Using environment variables, you can define various configuration options for your Dagster application and securely set up secrets. For example, instead of hard-coding database credentials - which is bad practice and cumbersome for development - you can use environment variables to supply user details. This allows you to parameterize your pipeline without modifying code or insecurely storing sensitive data.

There are two ways to declare and manage variables in Dagster+:

Dagster+ UIAgent configuration
Deployment type supportServerless, HybridHybrid
How it worksEnvironment variables are managed in the Dagster+ UI. Values are pulled from storage and decrypted when your code is executed.Environment variables are defined in the agent's configuration. Variables set at the code location level will pass through Dagster+, while those set at the deployment level bypass Dagster+ entirely. For more information, see "Setting environment variables using agent config".
Requirements
  • Dagster code must use version 1.0.17 or later
  • If using Hybrid, the agent must use Dagster version 1.0.17 or later
  • Editor, Admin, or Organization Admin permissions in Dagster+. Note: Editors and Admins can only set environment variables for deployments where they're an Editor or Admin.
Ability to modify your dagster.yaml and dagster_cloud.yaml files
Limitations
  • Maximum of 1,000 variables per full deployment
  • Variables must be less than or equal to 4KB in size
  • Variable names:
    • Must be 512 characters or less in length
    • Must start with a letter or underscore
    • Must contain only letters, numbers, and underscores
    • May not be the same as built-in (system) variables
Variable names:
  • Must start with a letter or underscore
  • Must contain only letters, numbers, and underscores
Storage and encryptionUses Amazon Key Management Services (KMS) and envelope encryption. For more information, see "Setting environment variables in the Dagster+ UI".Dependent on agent type.
ScopeScoped by deployment (full and branch) and optionally, code location.Scoped by code location. Variables can be set for a full deployment (all code locations) or on a per-code location basis.