How dunscaith hosts the supervisor¶

dunscaith does not decide how to recover a job. That is the supervisor in go/dispatch. dunscaith's part is to host it: to give it a process to run in, a record to commit to, a way to be reached from other machines, and a way to stop cleanly. Most of what you notice about the job commands follows from the choices made there.

The supervisor runs in your terminal¶

submit starts the job and then supervises it in the same process, in the foreground, until the job has an outcome. There is no background service. This is deliberate for a shared login node: a daemon would be a long-lived process owned by one user on a machine many people share, with its own ports and its own lifecycle to manage.

The cost is that the supervisor lasts only as long as the process. If you close the terminal, it ends, and the job carries on unwatched. dunscaith does not hide that. It marks the job Unattended, prints a notice on every later command, and offers supervise to reattach. The practical advice, to run submit under tmux, screen or nohup, is in Keep a job supervised.

A supervisor that can no longer act does not linger looking healthy. If its record stays unreadable, or its loop's health stops advancing (the store or the scheduler unreachable), for longer than supervise.giveup, it stops, exits with code 4 and leaves the job Unattended.

There is no HTTP health port, because on a shared login node ports collide and even loopback is visible to every user. Health is instead the supervisor's own loop: it writes the time its last loop completed into the job's lock file, and status calls the loop unhealthy when that is more than three poll intervals old. This is display only. Nothing about safety reads it.

The record lives on one host¶

Each job has a record: a durable history of what the supervisor decided and what the scheduler said. It is protected by a lock so that only one supervisor acts on a job at a time. That lock is only sound on a filesystem that one host controls, so dunscaith refuses a record root on a network filesystem. Login-node home directories are usually NFS, Lustre, GPFS or BeeGFS, which is why the default is a local directory under /var/tmp rather than your home.

Where records live is configuration because no single default works everywhere. /var/tmp is a local disk on a conventional login node, an overlay in a container, and a network mount on some Kubernetes-based systems. See Choose a record backing.

The root is bound to its host by a HOST file holding the machine ID and hostname. A record root copied or mounted onto another machine is refused there rather than used.

Records are local, but you want to ask about a job from whichever login node you are on. So at submit, dunscaith writes a small pointer file into your home directory, where every login node sees it: ~/.config/dunscaith/jobs/<job>.

The pointer names the owning host, the record root and its HOST identity, the absolute path of the dunscaith binary on that host, the cluster name and the adapter that reaches it, and the job's last known state with a time the supervisor last confirmed it was running.

A command for a job reads the pointer and compares the root's HOST identity with this machine's. If they match, it runs locally. If not, it re-runs itself on the owning host over SSH, so every change is still committed on that host under its lock. No file is shared between hosts for coordination.

Everything in a pointer is for display. The last state and supervised-at are written only on the owning host: the state by the supervisor on every change and by any command that commits there (hold, release, requeue, resubmit, give-up, cancel, abandon, run locally or forwarded), and supervised-at at most once every ten poll intervals to keep NFS writes rare. Nothing about safety reads a pointer, which is why an out-of-date one can mislead you but cannot make dunscaith do the wrong thing to a job. The one exception is routing: the owning host's name and binary path decide where a command goes.

Two consequences:

Off the owning host, Unattended is a judgement from the pointer and can lag by the refresh period. On the owning host it is exact, from the lock.
The absolute binary path is stored because ssh host dunscaith fails wherever PATH is set only by a login profile.

stop --emergency is the exception to everything above. It reads no record and uses no SSH. It opens the cluster with no store and stops every object that carries the job's name, so it works when the owning host is down.

How forwarding stays safe¶

Forwarding runs a command on another machine, as you, so the details matter.

The arguments travel as one base64 token, not as a command line. Joining arguments for ssh allowed an injected command, and per-argument quoting broke under tcsh.
The forwarded process carries a hidden flag and refuses to forward again, so an alias that points back at a forwarder cannot recurse.
The token carries a protocol version and the caller's dunscaith version. A protocol the owning host does not know is refused, naming both. The same protocol with different versions runs, with a warning.
Output is framed, because a shell's startup files can print into the stream. Anything outside the frames is discarded.
SSH failures are classified from ssh's own log. Only failures that prove the command never ran (authentication, host key, connection) are reported as such. A dropped connection is reported as "may have run", and dunscaith never retries it by itself.
dunscaith never relaxes host key checking, and prompts only after an authentication failure, only at a terminal. A running supervisor is never forwarded, so it never prompts.

The user-facing side is in Reach a job from another login node.

Signals drain the supervisor¶

go-tool-base's root owns signal handling and dunscaith registers none of its own. The root takes SIGTERM always, and SIGINT and SIGHUP unless the process started with them ignored, as under nohup. The first signal cancels the command's context. The supervisor then drains:

it stops starting new steps;
it finishes the step in flight, within supervise.drain;
it commits every response received;
it commits uncertainty: a submit attempt whose reply never came is recorded as unknown, never left out;
it releases the lock and exits.

A second signal ends the process at once, keeping what is committed. After a drain, the root re-raises the signal so the process dies by it, and your shell reports 128 + n.

A forwarded command drains too. With no terminal, the remote process is never sent a signal when the local ssh goes away. So the forwarded side treats end-of-input on its stdin as SIGHUP. For that to mean only "the caller went away", dunscaith gives ssh a stdin it holds open: your own stdin is copied into it while it lasts, but its end (an empty stdin, < /dev/null, a closed pipe in a script) is not passed on. It closes only when the local command is cancelled. With a terminal, dunscaith hands ssh the terminal itself and asks for one on the remote (-t), so Ctrl-C reaches the remote as SIGINT.

The wait after the outcome¶

A job reaching its outcome is not the end of the work. There may be objects still stopping, escalations not yet delivered, or an attempt of unknown outcome that blocks settlement. So the foreground supervisor keeps working for one bounded period, supervise.wait, on everything outstanding. At the end it prints what is left and exits 5, and reconcile carries it on later. The exit codes are in Exit codes.

Escalations are committed by the supervisor and delivered by dunscaith: printed in the terminal, and passed to your configured command if there is one. A delivery failure is retried and never blocks supervision.

Residual sweeps¶

A submit attempt can fail in a way that leaves its outcome unknown: the request went out and the reply never came. The scheduler might have created the job anyway. By design every object is created held and is never opened until the supervisor knows it is the right one, and a scheduler does not purge a held job, so a late object can still be found later by its name.

Under the default residual policy such an attempt does not hold the record open. When the record settles, the supervisor copies its unknown attempts to a per-user list in your home, ~/.config/dunscaith/residual/<job>, where any login node sees it. abandon and prune add to the same list. Under the manual policy the attempt escalates instead and the record waits for you.

A sweep looks up each job on the list by name. Every object of a job shares that name, so a sweep stops only an object that has not started running and is not foreign: for an unknown or abandoned attempt, only that attempt's object; for a pruned job, any such object. A running object is never stopped by a sweep. Every command that already talks to the scheduler sweeps: submit and supervise as supervision starts, hold, release, requeue, resubmit, give-up, cancel and abandon after they act, and status and reconcile. In this build a sweep asks the scheduler once per job on the list. Entries expire 30 days after they were added; an expiry is reported (by the command that drops it, and in list), never silent.

What is not here yet¶

No scheduler adapter. This build links none, so nothing that touches a scheduler runs end to end. The Slurm adapter is a separate module still to come.
Single host stores only. Postgres and Kubernetes stores, which would remove the one-host limit, are go/dispatch work.
No daemon mode. It would be the same supervisor holding many per-job locks.