Deliver escalations to a command¶

An escalation is the supervisor telling a person that it cannot proceed on its own: a job failed and its rules ran out, a submit attempt stayed unknown, or a foreign object turned up under the job's name. dunscaith always prints each one in the terminal of the supervisor. It can also hand each one to a command of yours, which is how you send it to email, a chat channel or a pager.

Configure the command¶

Set escalation.command to a list: the program, then its arguments.

escalation:
  command: [/home/you/bin/notify-escalation, --channel, training]

It is run directly, with no shell, so there is no quoting or expansion in the list. The program runs on the host that runs the supervisor, as you, with the supervisor's environment.

The key is read from configuration when the supervisor starts. Change it and restart the supervisor (see Keep a job supervised) for it to take effect.

What the command receives¶

The escalation arrives as one JSON object on stdin. The values here are illustrative:

{
  "job": "01J9ZQ6E0KXV8M2T4N7P3R5A1B",
  "id": "esc-1",
  "detail": "rule 0 exhausted its budget of 3",
  "at": "2026-10-12T09:14:03Z"
}

The command's own stdout and stderr are captured and shown only if it fails.

A script that reads the object and posts it:

#!/bin/sh
body=$(cat)
curl -fsS -X POST -H 'Content-Type: application/json' -d "$body" "$WEBHOOK_URL"

Exit 0 when delivered. Any other exit, or failing to start, counts as a delivery failure.

When delivery fails¶

dunscaith prints escalation <id> not delivered, retrying on the next loop: <error> and tries again on the next loop of the supervisor. A failing sink never stops supervision. dunscaith records a successful delivery in the job's record, so each escalation is delivered once.

The command can therefore run again for the same escalation after a failure that partly succeeded, so make it safe to repeat.

After the job ends¶

Once the job reaches an outcome, the supervisor stays for supervise.wait (two minutes by default) to finish delivering, retrying every poll interval, including an escalation raised by the step that settled the record. An escalation still undelivered at the end of that time makes submit or supervise exit 5, and prints which one. dunscaith reconcile on the owning host makes one more delivery attempt for every escalation a settled record still holds undelivered; run it again, for example from cron, until the command succeeds.

Escalations that need you to act, such as an unmatched attempt that must be closed with abandon, are described in the exit code reference.

Test the command¶

Run it by hand with a sample object:

echo '{"job":"01J9ZQ6E0KXV8M2T4N7P3R5A1B","id":"esc-1","detail":"test","at":"2026-10-12T09:14:03Z"}' \
  | /home/you/bin/notify-escalation --channel training