Reach a job from another login node¶

A job's record lives on the login node that submitted it. From any other login node that shares your home directory you can still run status, watch, cancel and the other job commands on it. dunscaith finds the owning host from a pointer file and runs the command there over SSH.

What you need¶

• A home directory shared between login nodes (usually NFS), so the pointer is visible everywhere.
• SSH from one login node to another, as yourself, without a password prompt you cannot answer. An agent, a key, or an open ControlMaster connection all work.
• The same dunscaith binary at the same absolute path on the owning host. The pointer records the path the binary had when you submitted.

The owning host is named by its own hostname¶

The pointer records the owning host by its own hostname, never by a login alias such as login.cluster.example that round-robins across several nodes. An alias would send the SSH hop to whichever node it picked, not to the one holding the record. Where the OS hostname is not the name other login nodes use, a site profile sets hosting.host and dunscaith records and checks that name instead.

So submit checks, before it creates any record, that the name resolves to this host and nothing else. It refuses when the name does not resolve, resolves to no address, or resolves to any address this host does not have, which is what a shared alias looks like:

submit refused before any record was created: this host cannot be reached by its own name: "login" resolves to 10.0.0.6, which this host does not have, so other login nodes could not reach its jobs by the name dunscaith records
  ways out:
  - set hosting.host to a name other login nodes reach this node by (a site profile can set it)
  - submit from a login node whose hostname resolves to that node alone
  - ask the site to make "login" resolve to this host only (DNS or /etc/hosts), not to a shared alias
  - see https://dunscaith.phpboyscout.uk/how-to/reach-a-job-from-another-login-node/

An OS hostname that resolves only to loopback addresses is refused too. Debian and Ubuntu write 127.0.1.1 <hostname> into /etc/hosts, so the name resolves on this host but tells no other host where it is. Set hosting.host to the name the other login nodes use:

hosting:
  host: login-0.cluster.example

A name set in hosting.host that resolves only to loopback here passes, because the site has said it is the right one. It must still not resolve to another host's address.

supersede makes the same check, since it writes a pointer too. There is no setting to skip it. The check resolves the name from this host, so it cannot prove that every other login node resolves it the same way.

Run the command¶

Nothing changes in how you type it:

dunscaith status 01J...
dunscaith watch 01J...
dunscaith cancel 01J...

dunscaith reads ~/.config/dunscaith/jobs/<job>. If the pointer says the record belongs to this host, the command runs here. Otherwise it runs ssh <owning host> <binary> --x-forwarded=<token> and relays the output and exit status back.

Three commands never forward:

list, prune and reconcile take no job and act on the host you run them on. list reads only pointers, so it works with the owning host down.

A forwarded command is not forwarded again. If it lands on a host that does not match the record root's HOST identity (machine-id and hostname together), it refuses. The owning host also refuses a forwarded submit or supervise, which only ever run where they are invoked, whatever global flags such as --config come before the command name.

The record root follows the job¶

A job command uses the record root the job was submitted under, which its pointer names. A forwarded command carries that root to the owning host, which checks its HOST identity against that root and runs the command there. On the owning host itself the pointer's root is used too, unless you pass --root, which then wins. So a job submitted with --root /somewhere is found by status, cancel and the rest without repeating --root, from any login node.

Commands that take no job (reconcile, prune) still act on the root resolved from --root, DUNSCAITH_ROOT, record.root and the default. See Choose a record backing.

The authentication ladder¶

dunscaith tries SSH in a fixed order and stops at the first that works.

1. Non-interactive. ssh -o BatchMode=yes. This covers an agent, your keys and an open ControlMaster socket, and never waits for input.
2. Interactive, only after an authentication failure. If step 1 failed with Permission denied and dunscaith is attached to a terminal, it runs ssh again without BatchMode and lets ssh ask for your passphrase, password or one-time code itself. dunscaith writes no prompt of its own, so two-factor setups work unchanged.
3. Nothing else falls back. Any other failure ends the command with an error.

Passing --ci, setting CI=true or the ci configuration key turns step 2 off. A running supervisor never forwards, so it never prompts. dunscaith does not use the OS keychain for SSH.

Understand the error¶

dunscaith classifies a failure from ssh's own log. The class decides whether the command might have run on the remote.

For a command that changes things, the dropped message adds it may have run, check with dunscaith status. dunscaith never retries a dropped write. Check the job before you repeat it.

When a read command (status, watch) finds the host unreachable, it also prints the job's last known state from the pointer, and whether it is supervised, judged from the pointer's supervised-at:

owning host login-0 unreachable; last known state of 01J...: running (none), outcome reached false
supervision Unattended, judged from the pointer off the owning host, which can lag by up to 5m0s

The state is whatever the owning host last wrote, and may be stale; the supervision judgement can lag by one refresh period (ten poll intervals). An authentication or host-key failure prints only its own error, since the host may be up.

If the owning host is down and you need the job stopped, use stop --emergency. It talks to the scheduler directly, records no cancel, and says so: recovery may still occur.

dunscaith stop --emergency 01J...

Fix a host key failure¶

dunscaith never passes StrictHostKeyChecking=no or accept-new, and it never falls back after a host-key failure. With =no, a changed key could still authenticate and run the command. Connect once by hand, check the fingerprint with your administrators, and accept it:

ssh login-0

Then repeat the dunscaith command. If the key has genuinely changed, remove the old entry (ssh-keygen -R login-0) only once you know why.

Connection reuse and ControlPath¶

dunscaith asks ssh to share one connection between commands (ControlMaster=auto, kept open for 60 seconds after the last use), so watch then cancel authenticates once. The socket directory is on local disk and never in your home, where ssh deletes a socket it cannot connect to:

• $XDG_RUNTIME_DIR/dunscaith, when XDG_RUNTIME_DIR is an absolute path outside $HOME;
• otherwise <tmp>/dunscaith-<uid> (normally /tmp/dunscaith-<uid>), where <tmp> follows $TMPDIR.

The directory is created with mode 0700. If it exists with another mode or owner, or if <directory>/<40-character hash> would exceed 90 bytes, dunscaith turns reuse off (ControlMaster=no), says why on stderr (ssh connection reuse is off, ...), and carries on. If every forwarded command asks for authentication again, check the length of $XDG_RUNTIME_DIR and $TMPDIR first.

ssh's own log (-E) normally goes in the same directory. When reuse is off for either reason, dunscaith puts the log in a fresh private directory (mode 0700) under <tmp> instead, and removes it when the command ends.

dunscaith adds -t when it is attached to a terminal, so Ctrl-C reaches the remote command. With -t, the remote's stderr merges into stdout.

Other things that can go wrong¶

• Different versions. The two binaries must speak the same forwarding protocol, or the owning host refuses and names both versions. If the protocol matches but the versions differ, the command runs with a warning naming both.
• Output noise. Anything your shell startup files print on the remote is discarded; only dunscaith's own framed output is relayed. If the remote shell cannot find the binary, you see the tail of its stderr in the error.
• Closing the connection. If your local ssh goes away, or you cancel the local command, the remote command sees its stdin close and drains as if it had received SIGHUP. Running a forwarded command with no stdin (from cron, or with < /dev/null) does not count: dunscaith keeps the remote's stdin open until the command ends.