On-call integrations

Prometheus Alertmanager

Route standard Alertmanager webhooks to a Transmit on-call configuration and escalation policy.

Start with a test incident API key. Test incidents never call an email, SMS or chat provider. Production paging requires a separate current configuration activation and eligible notification contacts.

Configure the receiver

  1. In Transmit, create an incident API key with both incidents:ingest and incidents:write. Keep the key in your monitoring secret store, outside the webhook URL.
  2. Copy the native configuration and escalation-policy IDs from the on-call configuration. Generate a UUID for this Alertmanager source and retain it. Independent sources need different source UUIDs; high-availability replicas of the same source share one.
  3. Use the configuration below, replacing its three example UUIDs. Configure grouping so notifications contain no more than twenty alerts. The example disables grouping and leaves truncation off. Validate it with your Alertmanager tooling before reloading.
route:
  receiver: transmit
  group_by: ['...']
receivers:
  - name: transmit
    webhook_configs:
      - url: https://api.transmit.dev/v1/integrations/alertmanager/00000000-0000-4000-8000-000000000001/00000000-0000-4000-8000-000000000002/00000000-0000-4000-8000-000000000003
        send_resolved: true
        max_alerts: 0
        http_config:
          authorization:
            type: Bearer
            credentials_file: /run/secrets/transmit_incident_key

The integration accepts the standard version-4 webhook. It rejects a truncated batch or more than twenty alerts, and limits the body to 256 KiB. Keep send_resolved enabled. Prometheus documents webhook receivers and HTTP authorization and alert fingerprints and timestamps.

Understand deduplication and recovery

Transmit identifies an alert by its source UUID and fingerprint within its configuration, policy and test/live namespace. A repeated firing notification for the same startsAt does not create another occurrence or notification intent. A later episode may join the still-active incident, retaining its original title and severity.

Episode ordering preserves nanosecond timestamps. Older firing and recovery events cannot replace a newer episode. Recovery received before firing records a terminal episode without creating an incident. A later firing retry for that episode is ignored. Manually resolving an incident also leaves repeated firing ignored; a newer source episode can open another incident.

Individual alert status drives recovery, rather than the batch status. The batch commits source ordering, incident updates, cancellation and notification intent together. No Idempotency-Key header is needed. Source startsAt must be known; resolved events require an endsAt after their start. Timestamps more than five minutes in the future are rejected. Do not reset source UUIDs to handle a delayed event.

Test and cut over

  1. Send a firing event using the test key, repeat it, then send recovery. Confirm one incident and canceled pending escalation after recovery.
  2. Verify recovery-before-firing and an old recovery after a newer firing episode. Confirm the newer failure remains open.
  3. Review coverage, verified contacts and provider checks, then explicitly activate ongoing paging. Replace the test credential with a dedicated live key after a controlled delivery drill.

The API response confirms committed ingestion, not notification delivery. Keep monitoring retries enabled for temporary failures. Fix invalid or truncated payloads before retrying. Revoking the dedicated key disconnects the source; revocation and scope changes are rechecked within each write transaction.