Job file format¶

dunscaith submit <jobfile> reads a job specification from a YAML or JSON file. A file ending .json is read as JSON; any other name is read as YAML.

dunscaith refuses any key the specification does not have, at every level, and reports the offender:

job file bad.yaml: json: cannot unmarshal JSON string into Go model.JobSpec: unknown object member name "foo"

A key is matched exactly, including case. A key given twice in the same object is refused, in YAML and in JSON, and so is a YAML file holding more than one document (a --- followed by more). It then validates the result, so a job that no backend could run is refused before anything is submitted.

Fields¶

name: llama-pretrain
command: [torchrun, --nproc-per-node=8, train.py]
env:
  WANDB_MODE: offline
workDir: /shared/projects/llama
nodes: 4
gpusPerNode: 8
procsPerNode: 8
timeLimit: 36h
checkpoint: s3://my-bucket/llama
recovery:
  rules:
    - on: [preempted, node-lost]
      action: resubmit
      budget: 3
    - on: [out-of-memory]
      action: escalate

Field	Type	Required	Meaning
`name`	string	no	A label for the job.
`command`	list of strings	yes	The program and its arguments. The first must not be empty and no argument may contain a NUL.
`env`	map of string to string	no	Environment variables. A key may not be empty, contain `=`, or begin with `DISPATCH_`, which is reserved.
`workDir`	string	no	The working directory.
`nodes`	integer	yes	Nodes to run on, at least 1.
`gpusPerNode`	integer	no	GPUs per node, not negative.
`procsPerNode`	integer	no	Processes per node, not negative.
`timeLimit`	duration	no	The job's wall-clock limit, not negative. See Durations.
`checkpoint`	string	no	Where checkpoints go: an absolute path or an `s3://bucket/prefix` URL with no query, fragment or credentials in it.
`recovery`	object	no	The recovery policy.
`extensions`	list	no	Backend-specific fields. An adapter refuses a namespace it does not know. None is available in this build.

IDs that dunscaith prints for jobs are ULIDs. You do not choose them.

Durations¶

timeLimit and a rule's for are written as Go duration strings: 90m, 12h, 36h30m. dunscaith converts them. A bare number is refused, because the specification would read it as nanoseconds, not seconds:

job file h.yaml: job file: /timeLimit is 3600; write a duration such as "90m" or "12h" (a bare number would be nanoseconds)

In JSON, write the duration as a string too: "timeLimit": "12h".

`recovery.rules`¶

An ordered list. A rule's position counts from 0 and is how --accept-unenforced and --accept-unverified name it.

Field	Type	Meaning
`on`	list of causes	Causes the rule covers. At least one. No cause may appear in two rules.
`action`	string	One of `resubmit`, `restart`, `requeue`, `hold`, `escalate`.
`budget`	integer	How many times to act. At least 1 for `resubmit`, `restart` and `requeue`. Must be absent for `hold` and `escalate`.
`for`	duration	A hold's deadline, for `hold` only. See Durations.

Causes are preempted, node-lost, out-of-memory, time-limit, non-zero-exit, launch-failed, no-capacity, held, released, requeued, cancelled, unknown. none and duplicate are refused, because neither can end an execution that recovery could act on.

The meaning of the actions and causes belongs to go/dispatch. See its documentation for how each behaves.

JSON¶

The same document in JSON:

{
  "name": "demo",
  "command": ["python", "train.py"],
  "nodes": 2,
  "timeLimit": "12h"
}

Checking a file¶

submit reads and validates the file before it opens the scheduler. In this build, with no adapter linked, that gives you a way to check a file:

a good file stops with no cluster adapter: ..., which means it parsed and validated;
a bad file stops with job file <name>: ....

No job is submitted either way, though the first run creates an empty record root. Values in env must be strings: A: 1 is refused, so quote numbers (A: "1").