Choose a record backing for your site¶

dunscaith keeps one record per job. The supervisor commits every decision to it, and status, watch and cancel read it. This guide covers where the records live and what to do when dunscaith refuses the place it was given.

For why records are tied to one host, see How dunscaith hosts the supervisor.

Check what you have now¶

With no configuration, dunscaith keeps records under /var/tmp/dunscaith-$USER/. It creates the directory with mode 0700 and writes a HOST file in it that binds the root to this host. Any command that opens records reports a refused root before it does anything else, so run one:

dunscaith reconcile

A refusal is described below. If the root is acceptable, the command goes on to the scheduler. In a build with no adapter linked, it then stops with no cluster adapter, which here means the root is fine. With an adapter it prints reconciled <root>. reconcile is safe to run at any time.

Choose a root¶

Only the file store exists in this build, so the choice is the directory it uses. It must be on a filesystem local to the login node. NFS, Lustre, GPFS and BeeGFS are refused, because the record's lock is only sound on one host.

dunscaith picks the root in this order, first match wins:

1. --root <path> on the command
2. the DUNSCAITH_ROOT environment variable
3. record.root in configuration
4. /var/tmp/dunscaith-$USER/

$USER, $HOME and a leading ~/ are expanded.

Set it once in configuration rather than on each command. Every host that reaches your jobs reads the same configuration file, and a forwarded command resolves the root on the owning host from that file. Do not rely on --root for jobs you will reach from another login node. See Reach a job from another login node.

# ~/.dunscaith/config.yaml
record:
  store: file
  root: /var/tmp/dunscaith-${USER}/

When dunscaith refuses the root¶

If the filesystem is not local, or is a RAM-backed one, dunscaith stops with:

record root /dev/shm/dr refused: ...: tmpfs loses every record on reboot; AcceptEphemeral acknowledges that
  records must live on a filesystem local to this host; ways out:
  - pass --root, or set DUNSCAITH_ROOT, to a local path
  - set record.root in configuration (a site profile can set it for everyone)
  - for a root that will not survive a reboot or pod restart, set record.ephemeral: true
  - see https://dunscaith.phpboyscout.uk/how-to/choose-a-record-backing/

The first line names the filesystem type that was found. Pick the way out that matches it:

Declare an ephemeral root¶

record.ephemeral is a declaration, not a detection. dunscaith cannot tell from the filesystem type that a directory lives only as long as a pod, so you say so:

record:
  root: /tmp/dunscaith
  ephemeral: true

Set it only for a root you know will not survive a reboot or pod restart. dunscaith then:

• accepts a RAM-backed root it would otherwise refuse;
• says records under <root> do not survive a pod restart (record.ephemeral) once, in the submit receipt;
• repeats records do not survive a pod restart (record.ephemeral) in every status.

Losing a record loses the supervisor's memory of the job, not the job. The scheduler still has it, and stop --emergency can still stop it by name.

Unknown territory¶

Whether /var/tmp survives on your cluster is not something dunscaith can know for certain. Many sites clean it with systemd-tmpfiles after a set age, and login nodes can be reimaged. If either is true for you, a record can vanish while its job runs. status then reports the record missing and shows the last state from the job's pointer. Ask your administrators for a persistent node-local path and use that as record.root.

The clean-up age warning¶

Where it can, dunscaith reads the clean-up age from the host's systemd-tmpfiles configuration: the .conf files in /etc/tmpfiles.d, /run/tmpfiles.d, /usr/local/lib/tmpfiles.d and /usr/lib/tmpfiles.d, a file in an earlier directory hiding one of the same name in a later one. The rule that counts is the line for the record root or its nearest ancestor whose type cleans (d, D, e, v, q, Q, C) and whose age is set. systemd-tmpfiles skips a path that has an entry of its own, so any line, glob or not, for the root or an ancestor nearer than that cleaning line means there is no clean-up: x, or a d with no age such as d /var/tmp/dunscaith-%u 0700 - - -. X is the exception: it excludes only the directory itself, not what is in it (tmpfiles.d(5)), so it does not protect the records under the root. Boot-only lines (!) do not run on the clean-up timer and are ignored.

dunscaith expands %u, %U and %h as the user running it, %% as a percent sign, and the fixed directory specifiers such as %V (/var/tmp). A line whose path holds a specifier it cannot expand, such as %m or %H, and that might name the root or an ancestor, leaves the age unknown, so dunscaith says nothing.

When a job's record was created longer ago than that age, status, supervise and reconcile print a warning once for that job:

warning: the record of 01J9ZQ6E0KXV8M2T4N7P3R5A1B is 31 days old, older than the clean-up age 30d that /usr/lib/tmpfiles.d/tmp.conf:12 sets for /var/tmp; systemd-tmpfiles may delete it.
  - for new jobs, set record.root (or pass --root, or set DUNSCAITH_ROOT) to a path the site does not clean
  - or ask the site to exclude the root with a line such as `x /var/tmp/dunscaith-ada` in /etc/tmpfiles.d
  - see https://dunscaith.phpboyscout.uk/how-to/choose-a-record-backing/

dunscaith remembers the warning in ~/.config/dunscaith/warned/<job>. Delete that file to see the warning again. If the configuration is missing, unreadable, or holds a line dunscaith cannot parse, it cannot read the age and says nothing.

A line whose type dunscaith does not know, such as one a newer systemd added, is ignored unless its path is the root or an ancestor. If it is, that line might clean or exclude the root, so dunscaith cannot judge the age. It says so once per command, naming the line, and carries on without the warning:

note: whether systemd-tmpfiles cleans /var/tmp/dunscaith-ada could not be judged: /etc/tmpfiles.d/site.conf:3: line type "Y" is not one dunscaith knows

An administrator who keeps the root with its own line, such as x /var/tmp/dunscaith-* or d /var/tmp/dunscaith-%u 0700 - - -, turns the warning off for everyone.

Other backings¶

record.store accepts only file. Other stores (Postgres, Kubernetes) are planned in go/dispatch and will arrive as further values of the same key.