Keep a job supervised¶
dunscaith submit supervises the job in your terminal until it reaches an outcome. There is no
daemon. If the terminal goes, so does the supervisor. This guide shows how to stop that happening,
and how to recover when it does.
Run submit under a multiplexer¶
Start tmux or screen first, then submit inside it:
Detach with your multiplexer's key (Ctrl-b d in tmux). The supervisor keeps running. Reattach with
tmux attach -t train.
nohup also works, but you lose the live output unless you redirect it:
Under nohup, SIGHUP is ignored when the process starts, and dunscaith does not subscribe to a
signal that was ignored at start. A dropped session therefore neither drains nor kills the
supervisor. To stop it, send SIGTERM (kill <pid>), which dunscaith always handles.
What happens if the terminal closes anyway¶
Without a multiplexer, closing the terminal sends SIGHUP. dunscaith treats SIGHUP like SIGINT and SIGTERM: it drains and exits.
- It stops starting new steps.
- It finishes the step in flight, within
supervise.drain(30 seconds by default). - It commits every response it has received. A submit attempt whose reply never came is committed as unknown, not dropped.
- It releases the lock and exits.
The job keeps running on the cluster. It is now Unattended: nothing is watching it, so nothing
will recover it if it is preempted.
Some sites kill a user's processes when a session ends (KillUserProcesses=yes or a reaper). That
ends the supervisor without a signal it can drain on. A multiplexer started under the same session
may be killed too. Ask your administrators, or test with a short job.
A second signal during the drain exits immediately and leaves only what is already committed.
Find unattended jobs¶
Every job command prints a line for each of your jobs that is Unattended, before it does its own
work:
list marks them too, and status <job> says supervision Unattended for the one job:
How Unattended is judged depends on where you ask:
- On the job's owning host, dunscaith checks whether anything holds the job's lock.
- On any other host, it reads the pointer. A job not at an outcome whose pointer has no
supervised-attime, or one older than twenty poll intervals (ten minutes by default), is shown asUnattended.
The second can lag by up to ten minutes with the defaults. A job whose supervisor just died can look supervised from another node for that long.
Reattach¶
Log in to the host that owns the job (the host list shows) and run:
supervise takes the job's lock and reconciles with the scheduler first, so it picks up from the
committed record. It ends the same way submit does, with the exit code in the
exit code reference.
supervise is never forwarded over SSH, because a forwarded supervisor would last only as long as
the SSH session. Run on the wrong host it refuses and names the right one:
command is never forwarded: supervise runs only on the job's owning host; log in to login-0 and run it there
If the job is already supervised, supervise fails with job <id> is already supervised.
Settle what a reboot left behind¶
A supervisor exits once the job's record is settled. If the owning host loses power soon after,
the settlement can be lost with the filesystem's unsynced writes, and the record comes back
unsettled. The pointer still says the outcome was reached, so the job is not shown Unattended,
and nothing settles it again until someone runs a command that does.
On the owning host, commands that take a job, submit and prune name such a job:
01J... reached its outcome but its record is not settled, so nothing supervises it: `dunscaith reconcile` or `dunscaith supervise 01J...` settles it
When the job's record lives under a root other than the one the command uses, the line adds
--root <path> to reconcile.
To settle these without waiting for someone to run a command, run dunscaith reconcile at boot on
each login node you submit from. A systemd user unit does it. Save it as
~/.config/systemd/user/dunscaith-reconcile.service, with the binary's absolute path:
[Unit]
Description=Settle dunscaith records a reboot left open
[Service]
Type=oneshot
ExecStart=/opt/dunscaith/bin/dunscaith reconcile
[Install]
WantedBy=default.target
Then enable it:
A user unit under default.target starts with your user manager. Without lingering that is your
first login, not the boot, so loginctl enable-linger is what makes it run at boot. Some sites do
not allow lingering; ask your administrators.
Where systemd user units are not available, an @reboot line in your crontab (crontab -e) is the
alternative:
cron runs it with a minimal environment, so give the absolute path. In both, add --root <path>
if your records are not under the configured root. reconcile talks to the scheduler, so if the
scheduler is not reachable that early it fails, and the notice above still names the job the next
time you run a command. A root declared record.ephemeral does not survive a reboot, so there is
nothing to reconcile on it.
Submit without supervising¶
--detach submits and returns. Rules in the job's recovery policy that only the supervisor could
enforce become unenforced, so the submit fails unless you accept that with --accept-unenforced N
for each such rule (rule numbers count from 0). The plan records that you did. Attach a supervisor
later with supervise.
Tune the timings¶
The poll interval, the drain and the post-outcome wait are configuration, under supervise:
See the configuration reference.