Setting up alerts on Dagster+
Dagster+ allows you to configure alerts to automatically fire in response to a range of events. These alerts can be sent to a variety of different services, depending on your organization's needs.
These alerts can be configured in the Dagster+ UI, or using the dagster-cloud
CLI tool.
Prerequisites
- Organization, Admin, or Editor permissions on Dagster+
Configuring a notification service
To start, you'll need to configure a service to send alerts. Dagster+ current supports sending alerts through email, Microsoft Teams, PagerDuty, and Slack.
- Microsoft Teams
- PagerDuty
- Slack
No additional configuration is required to send emails from Dagster+.
All alert emails will be sent by "no-reply@dagster.cloud"
or "no-reply@<subdomain>.dagster.cloud"
. Alerts can be configured to be sent to any number of emails.
Create an incoming webhook by following the Microsoft Teams documentation.
This will provide you with a webhook URL which will be required when configuring alerts in the UI (after selecting "Microsoft Teams" as your Notification Service) or using the CLI (in the notification_service
configuration).
You will need sufficient permissions in PagerDuty to add or edit services.
In PagerDuty, you can either:
- Create a new service, and add Dagster+ as an integration, or
- Edit an existing service to include Dagster+ as an integration
When configuring the integration, choose Dagster+ as the integration type, and choose an integration name in the format dagster-plus-{your_service_name}
.
After adding your new integration, you will be taken to a screen containing an Integration Key. This value will be required when configuring alerts in the UI (after selecting "PagerDuty" as your Notification Service) or using the CLI (in the notification_service
configuration).
You will need sufficient permissions in Slack to add apps to your workspace.
Navigate to Deployment > Alerts in the Dagster+ UI and click Connect to Slack. From there, you can complete the installation process.
When setting up an alert, you can choose a Slack channel to send those alerts to. Make sure to invite the @Dagster+
bot to any channel that you'd like to receive an alert in.
Alerting when a run fails
You can set up alerts to notify you when a run fails.
By default, these alerts will target all runs in the deployment, but they can be scoped to runs with a specific tag.
- In the UI
- In code
-
In the Dagster UI, click Deployment.
-
Click the Alerts tab.
-
Click Add alert policy.
-
Select Run alert from the dropdown.
-
Select Job failure.
If desired, add tags in the format {key}:{value}
to filter the runs that will be considered.
Execute the following command to sync the configured alert policy to your Dagster+ deployment.
dagster-cloud deployment alert-policies sync -a /path/to/alert_policies.yaml
- Microsoft Teams
- PagerDuty
- Slack
Loading...
Loading...
Loading...
Loading...
Alerting when a run is taking too long to complete
You can set up alerts to notify you whenever a run takes more than some threshold amount of time.
By default, these alerts will target all runs in the deployment, but they can be scoped to runs with a specific tag.
- In the UI
- In code
-
In the Dagster UI, click Deployment.
-
Click the Alerts tab.
-
Click Add alert policy.
-
Select Run alert from the dropdown.
-
Select Job running over and how many hours to alert after.
If desired, add tags in the format {key}:{value}
to filter the runs that will be considered.
Execute the following command to sync the configured alert policy to your Dagster+ deployment.
dagster-cloud deployment alert-policies sync -a /path/to/alert_policies.yaml
- Microsoft Teams
- PagerDuty
- Slack
Loading...
Loading...
Loading...
Loading...
Alerting when an asset fails to materialize
You can set up alerts to notify you when an asset materialization attempt fails.
By default, these alerts will target all assets in the deployment, but they can be scoped to a specific asset or group of assets.
- In the UI
- In code
-
In the Dagster UI, click Deployment.
-
Click the Alerts tab.
-
Click Add alert policy.
-
Select Asset alert from the dropdown.
-
Select Failure under the Materializations heading.
If desired, select a target from the dropdown menu to scope this alert to a specific asset or group.
Execute the following command to sync the configured alert policy to your Dagster+ deployment.
dagster-cloud deployment alert-policies sync -a /path/to/alert_policies.yaml
- Microsoft Teams
- PagerDuty
- Slack
Loading...
Loading...
Loading...
Loading...
Alerting when an asset check fails
You can set up alerts to notify you when an asset check on an asset fails.
By default, these alerts will target all assets in the deployment, but they can be scoped to checks on a specific asset or group of assets.
- In the UI
- In code
-
In the Dagster UI, click Deployment.
-
Click the Alerts tab.
-
Click Add alert policy.
-
Select Asset alert from the dropdown.
-
Select Failed (ERROR) under the Asset Checks heading.
If desired, select a target from the dropdown menu to scope this alert to a specific asset or group.
Execute the following command to sync the configured alert policy to your Dagster+ deployment.
dagster-cloud deployment alert-policies sync -a /path/to/alert_policies.yaml
- Microsoft Teams
- PagerDuty
- Slack
Loading...
Loading...
Loading...
Loading...
Alerting when a schedule or sensor tick fails
You can set up alerts to fire when any schedule or sensor tick across your entire deployment fails.
Alerts are sent only when a schedule/sensor transitions from success to failure, so only the initial failure will trigger the alert.
- In the UI
- In code
- In the Dagster UI, click Deployment.
- Click the Alerts tab.
- Click Add alert policy.
- Select Schedule/Sensor alert from the dropdown.
Execute the following command to sync the configured alert policy to your Dagster+ deployment.
dagster-cloud deployment alert-policies sync -a /path/to/alert_policies.yaml
- Microsoft Teams
- PagerDuty
- Slack
Loading...
Loading...
Loading...
Loading...
Alerting when a code location fails to load
You can set up alerts to fire when any code location fails to load due to an error.
- In the UI
- In code
- In the Dagster UI, click Deployment.
- Click the Alerts tab.
- Click Add alert policy.
- Select Code location error alert from the dropdown.
Execute the following command to sync the configured alert policy to your Dagster+ deployment.
dagster-cloud deployment alert-policies sync -a /path/to/alert_policies.yaml
- Microsoft Teams
- PagerDuty
- Slack
Loading...
Loading...
Loading...
Loading...
Alerting when a Hybrid agent becomes unavailable
This is only available for Hybrid deployments.
You can set up alerts to fire if your Hybrid agent hasn't sent a heartbeat in the last 5 minutes.
- In the UI
- In code
- In the Dagster UI, click Deployment.
- Click the Alerts tab.
- Click Add alert policy.
- Select Code location error alert from the dropdown.
Execute the following command to sync the configured alert policy to your Dagster+ deployment.
dagster-cloud deployment alert-policies sync -a /path/to/alert_policies.yaml
- Microsoft Teams
- PagerDuty
- Slack
Loading...
Loading...
Loading...
Loading...