Alert sources

How alert sources work

An alert source is a tool that opens incidents by sending Opsphie a webhook: Grafana, Datadog, Sentry, or anything that can make an HTTP request. Each source has its own URL with a secret in it, and can also sign what it sends.

• Setting one up. Choose the tool, and Opsphie fills in where each field is in its alerts. You see the URL, and a signing secret if Opsphie makes one, only once: copy them into the tool. Rotate URL gives you new ones and stops the old ones at once; Delete stops the source.
• One incident per alert. The first alert opens an incident, with its channel, as if someone had declared it. Each alert has a dedupe key (Grafana's group, Datadog's alert cycle, Sentry's metric alert): while its incident is active, the same key adds to that incident instead of opening another. Once the incident is resolved or canceled, the next alert with that key opens a new one.
• Alerts close together share an incident. A new alert from a source that sent another in the last 5 minutes joins that alert's incident, if it's still active, and is posted in its channel. Each alert restarts the 5 minutes, so a tool alerting every few minutes keeps adding to one incident. Alerts from different sources never share one.
• Repeats and resolutions. An alert firing again is posted in the incident channel, at most every 15 minutes. When the tool says the alert resolved, that's posted too, and the incident stays open: someone still resolves it. Both go on the timeline, where admins can hide them under Alerts.
• Severity. Each source maps its tool's severities to P0–P3, and has a default for alerts without one. The incident also gets a link back to the alert, and “monitoring” as how it was detected.
• Limits. A source can send 120 requests a minute. Past that, Opsphie answers 429 and drops the alert, and the org's admins get a Slack message (at most once a day per source).

Who commands an alert's incident

An alert's incident always has a commander, who's added to its channel and gets a Slack message saying so. Each source can read an owner from the alert, and Opsphie picks, in order:

1. A member the owner names, by email or Slack user ID.
2. A team the owner names, by its handle or its Slack user group's handle: the team's incident lead, set on the team's page.
3. Otherwise, the source's default commander.

The commander can hand the incident on as usual, with /inc roles commander @someone in its channel. An alert that joins an incident leaves its commander as it is.

A team the owner names is also the incident's team involved, even without an incident lead. Otherwise the commander's primary team is.

Grafana

1. In Opsphie, add an alert source and choose Grafana. Copy the URL and signing secret.
2. In Grafana, go to Alerting → Contact points, add a contact point, and choose the Webhook integration. Paste the URL.
3. Under the optional settings, set the HMAC signature secret to the signing secret, and leave its header as X-Grafana-Alerting-Signature. Leave the timestamp header empty.
4. Send alerts to it with a notification policy. To route them to a team, give alert rules a team label with the team's handle; a severity label of critical, warning or info sets P1, P2 or P3.

Opsphie reads the title from the alertname label, and dedupes by Grafana's group: alerts Grafana groups into one notification open one incident.

Following Grafana's webhook contact point docs.

Datadog

1. In Opsphie, add an alert source and choose Datadog. Copy the URL.
2. In Datadog, open the Webhooks integration and add a webhook. Name it (say, opsphie), paste the URL, and use this payload, leaving Encode as form off:

{
  "title": "$ALERT_TITLE",
  "alert_id": "$ALERT_ID",
  "cycle_key": "$ALERT_CYCLE_KEY",
  "transition": "$ALERT_TRANSITION",
  "priority": "$ALERT_PRIORITY",
  "link": "$LINK",
  "team": "$TAGS[team]"
}

3. Add @webhook-opsphie to the message of each monitor that should open incidents. Tag a monitor team:<handle> to route it to a team; its priority (P1–P5) sets the severity, with P4 and P5 as P3.

Datadog can't sign webhooks, so the secret in the URL is what protects it: keep it out of monitor messages and dashboards. A recovery posts in the incident's channel.

Following Datadog's Webhooks integration docs.

Sentry

For Sentry's metric alerts.

1. In Opsphie, add an alert source and choose Sentry. Copy the URL.
2. In Sentry, go to Settings → Developer Settings and create an internal integration. Paste the URL as its webhook URL, and turn on Alert Rule Action.
3. Copy the integration's Client Secret, edit the source in Opsphie, and paste it as the signing secret. Until you do, Sentry's alerts are checked against the URL alone.
4. In each metric alert, add the action Send a notification via your integration, for its critical and warning triggers. Critical opens a P1, warning a P2.

Following Sentry's metric alert webhook docs.

Your own tools and scripts

Choose Your own tool or script, and POST a JSON object to the URL. Only title is needed; dedupe_key (else the title is the key), severity (P0–P3), owner, url and status (resolved to resolve) are optional. The source's fields can be changed to match JSON you already send.

Sign the body with the source's secret: the hex HMAC-SHA256 of the exact bytes sent, in X-Opsphie-Signature: sha256=<signature>.

A script can keep the token out of the URL, where logs and proxies may record it: POST to the URL without its last part (…/webhooks/alerts) and send the last part in Authorization: Bearer <token>.

body='{"title": "Checkout error rate above 5%", "dedupe_key": "checkout-errors", "severity": "P1", "owner": "payments", "url": "https://dashboards.example.com/checkout"}'
signature=$(printf '%s' "$body" | openssl dgst -sha256 -hmac "$OPSPHIE_SECRET" | sed 's/^.* //')
curl -X POST "$OPSPHIE_ALERT_URL" \
  -H "Content-Type: application/json" \
  -H "X-Opsphie-Signature: sha256=$signature" \
  -d "$body"

import hashlib, hmac, json, os, urllib.request

body = json.dumps({"title": "Disk 95% full on db-1", "dedupe_key": "db-1-disk"}).encode()
signature = hmac.new(os.environ["OPSPHIE_SECRET"].encode(), body, hashlib.sha256).hexdigest()
request = urllib.request.Request(
    os.environ["OPSPHIE_ALERT_URL"],
    data=body,
    headers={"Content-Type": "application/json", "X-Opsphie-Signature": f"sha256={signature}"},
)
print(urllib.request.urlopen(request).read())  # {"incident": 42}