On-call integrations

Datadog

Page on Datadog monitors with a webhook, and resolve automatically on recovery.

Start with a test incident API key. Test incidents never call an email, SMS, voice or chat provider.

Create the webhook

  1. Create an incident API key with incidents:ingest and incidents:write.
  2. In Datadog, open Integrations → Webhooks and add a webhook named, for example, transmit, with this URL. Copy your own from the on-call configuration page.
  3. Use this payload exactly, and these custom headers.
  4. Mention @webhook-transmit in each monitor's notification message, including its recovery notification.
https://api.transmit.dev/v1/integrations/datadog/00000000-0000-4000-8000-000000000001/00000000-0000-4000-8000-000000000002
{
  "alert_id": "$ALERT_ID",
  "aggregate": "$AGGREG_KEY",
  "transition": "$ALERT_TRANSITION",
  "title": "$EVENT_TITLE",
  "body": "$EVENT_MSG",
  "priority": "$ALERT_PRIORITY",
  "link": "$LINK"
}
{"Authorization": "Bearer YOUR_TEST_INCIDENT_KEY", "Content-Type": "application/json"}

How alerts map

The aggregate key (or the monitor id when it is empty) identifies the alert, so each monitor group pages once while it is active. A Recovered transition resolves the incident and cancels pending escalation. Every other transition, including warnings, no data and re-notifications, opens or joins one. Monitor priority P1 is critical, P2 high, P3 medium, P4 and P5 low; without a priority, warnings are medium and alerts high. A monitor link is appended to the description.