Job commands¶

These are dunscaith's own commands. The commands go-tool-base supplies (config, doctor, init, update, version and others) are listed in dunscaith --help.

A job argument is a job ID, a ULID, as printed by submit and list. Run dunscaith <command> --help for the flags of the build you have.

Global flags¶

Every command accepts:

The --root flag¶

Every command that takes a job, and submit, reconcile and prune, accepts --root <path>, the record root. It overrides DUNSCAITH_ROOT and record.root. A command that takes a job uses the root its pointer names unless --root is given; a forwarded one always uses the pointer's root. list, s3 login and s3 secret do not have it. stop accepts it but does not use it, because the emergency stop never reads records.

Where a command runs¶

supervise run off the owning host refuses and names the host. See Reach a job from another login node.

Commands that take a job, submit and reconcile first print a line for every one of your jobs that is Unattended. On the owning host, commands that take a job, submit and prune also print a line for every job whose pointer says it reached its outcome while its record is not settled, which is what a settlement lost to a power cut leaves behind. See Settle what a reboot left behind.

submit and supervise as supervision starts, status, reconcile, and hold, release, requeue, resubmit, give-up, cancel and abandon after they act, also sweep the residual list for late objects (see Residual sweeps).

submit¶

dunscaith submit <jobfile> [--detach] [--idempotency-key <key>]
                 [--accept-unenforced N,...] [--accept-unverified N:enforcer,...] [--root <path>]

Reads the job file, submits it, writes the pointer, and supervises the job in the foreground until its outcome. The exit code is in Exit codes.

Before it creates the record, submit checks that the name the pointer records, hosting.host or else this host's hostname, resolves to this host alone. If the name does not resolve, resolves to an address this host does not have (as a round-robin login alias does), or is the hostname and resolves only to loopback, it refuses, naming the host and the ways out. See Reach a job from another login node.

If the pointer cannot be written, submit stops before anything touches the scheduler and its error names the job, its root and this host, with the supervise --root and cancel --root commands that reach it from here.

It prints a receipt:

submitted 01J9ZQ6E0KXV8M2T4N7P3R5A1B
  rule 0: supervisor
supervising 01J9ZQ6E0KXV8M2T4N7P3R5A1B in the foreground; it ends with this terminal unless run under tmux, screen or nohup

With --detach the last line is detached: `dunscaith supervise <job>` attaches a supervisor. While supervising it prints each state change as <time> <job>: <from> -> <to> (<cause>), then a closing job <id>: <state> (<cause>). Under a record.ephemeral root the receipt adds that the records do not survive a pod restart.

supervise¶

dunscaith supervise <job> [--root <path>]

Reattaches a foreground supervisor to an existing job. It takes the job's lock, reconciles with the scheduler, and supervises to the outcome. Runs only on the owning host. Exits as submit does. Before it starts, it warns once if the record is older than the site's clean-up age (see the clean-up age warning).

list¶

dunscaith list

Prints every job of yours from its pointer, without reading any record, so it works when the owning host is down. One line per job: ID, owning host, last state, cause, and Unattended when it is. It then reports any residual-list entry that has expired.

status¶

dunscaith status <job> [--root <path>]

Prints the job's ID, state and cause, clean-up state, any hold, each recovery rule with its enforcer, identity weakened: <reason> when the plan records that this job's executions may not each carry a distinct identity (for example a Kubeflow site's allowRankRestart), each submission (canonical object, executions, whether abandoned), a line undelivered escalation <id>: <detail> for each escalation not yet delivered, and the supervision line:

• supervision Unattended: ... reattaches
• supervision supervised; last loop <time>
• supervision supervised, but the loop is unhealthy: last completed <time>
• supervision supervised; loop health unknown

It ends with records do not survive a pod restart (record.ephemeral) under an ephemeral root, and then sweeps for late objects of settled jobs. If the record is older than the site's clean-up age, it warns once for the job (see the clean-up age warning).

If the record is gone, status says the job was pruned and removes its pointer, or says the record is missing, shows the pointer's last state and keeps the pointer.

watch¶

dunscaith watch <job> [--root <path>]

Streams the job's state transitions, one line each, until its outcome, then exits 0. Needs the record: if it is gone, watch says so (pruned or missing, as status does) and exits 1 rather than waiting. Any other failed read is reported and retried on the next poll.

hold¶

dunscaith hold <job> --until <time> --reason <reason> [--root <path>]

Places a backend hold. --until is an RFC 3339 time or a duration from now such as 6h. Both --until and --reason are required. What the scheduler does with it is the adapter's business.

release¶

dunscaith release <job> [--reason <reason>] [--root <path>]

Releases a backend hold. Refused on a recovery hold; use resubmit, give-up or cancel for that.

requeue¶

dunscaith requeue <job> --reason <reason> [--root <path>]

Commits a requeue intent, which the supervisor then carries out. A later cancel revokes it. Fails as unsupported where the adapter cannot requeue. --reason is required.

resubmit¶

dunscaith resubmit <job> [--reason <reason>] [--root <path>]

Leaves a recovery hold with a new submission of the same job.

supersede¶

dunscaith supersede <job> [--reason <reason>] [--accept-unenforced N,...]
                    [--accept-unverified N:enforcer,...] [--root <path>]

Runs a Failed job again as a new job that supersedes it. It prints the new plan and the new job's ID and writes its pointer. It does not supervise it: run dunscaith supervise <new job>. Like submit, it first refuses a host whose recorded name does not resolve to this host alone.

abandon¶

dunscaith abandon <job> <N>[/<attempt>] [--reason <reason>] [--root <path>]
dunscaith abandon <job> --foreign <ref> [--reason <reason>] [--root <path>]

Once an escalation has said so, ends submission N's unmatched attempts, or the one attempt you name, or disposes of a foreign object found under the job's name. Name either <N>[/<attempt>] or --foreign, not both. N and the attempt count from 1.

Each abandoned attempt joins the residual list by its number (for <N> alone, the attempts the record shows it closed), so a sweep never stops the submission's canonical object. abandon warns that a late object of one may still appear. --foreign takes the object's ID.

give-up¶

dunscaith give-up <job> [--reason <reason>] [--root <path>]

Leaves a recovery hold as Failed, with an escalation.

cancel¶

dunscaith cancel <job> [--reason <reason>] [--root <path>]

Commits the cancel, then stops everything found by name. It reports the two separately:

cancel committed on <job>
clean-up complete

or clean-up outstanding: N object(s) still stopping, with a line for each node that was down or not responding, whose processes may live. A job that had already ended says so and reports that the cancel changed nothing. It fails if the store is unavailable.

stop¶

dunscaith stop --emergency <job> [--reason <reason>]

Stops every object of the job by name without recording a cancel. --emergency is required: without it, use cancel. It goes to the scheduler directly, never over SSH and never through the record, so it works with the owning host down and the store unavailable. Recovery may still occur, and the output says so.

stop uses the cluster in your configuration. When the job has a pointer naming another cluster, by cluster.name or by adapter, it refuses and names both, because the pointer does not carry the adapter's settings and a stop by name on the wrong scheduler would find nothing and still report success. Run it again with the configuration of the cluster the job was submitted to.

reconcile¶

dunscaith reconcile [--root <path>]

Visits every record on this host that has not settled, including finished ones with work left, and settles what it can. It copies the residuals of every record settled in the last 30 days to the residual list, so a record it settled is swept for like one a supervisor settled; an entry already on the list keeps the time it was added. It warns once for each record, settled or not, that is older than the site's clean-up age (see the clean-up age warning). It then sweeps for late objects and prints reconciled <root>. It acts on the current host's root only.

prune¶

dunscaith prune [--root <path>]

Tombstones settled records older than 30 days from settlement. For each, it first adds the job's names to the residual list, then tombstones the record, then removes the pointer, and prints pruned <job>. It never runs on its own.

s3 login¶

dunscaith s3 login [--store s3://bucket/prefix] [--profile <name>] [--from-env field=VARIABLE,...]

Writes the file credential source and probes the store with it. --store defaults to checkpoint.s3.root, and one of them is required. --profile defaults to default. Without --from-env it needs a terminal. See Set up S3 checkpoint credentials.

s3 secret¶

dunscaith s3 secret --namespace <ns> [--profile <name>] [--soperator]

Creates or updates the Kubernetes Secret behind a secret source, with your own kubectl, and prints the volume and mount your pods need, then the checkpoint.s3.credentials entry that reads the mounted Secret, as YAML. --soperator adds a warning that the Secret is readable by every job on shared workers.

Hidden entry point¶

Forwarding re-runs dunscaith on the owning host as dunscaith --x-forwarded=<token>. The flag is hidden from help, is for dunscaith's own use, and its format is not a stable interface.