dunscaith¶
Submit, supervise and recover multi-node GPU training jobs on Slurm and Kubeflow, from one CLI.
dunscaith takes a training job from submission to a classified outcome. It submits to the scheduler you already have, watches the run, tells preemption from node loss from a genuine failure, and recovers what can be recovered, with every decision visible in its status. It is built on go-tool-base and on the go/dispatch module family. The training itself is scatha's.
Design phase
dunscaith is a go-tool-base skeleton with no features of its own yet. Decisions are being
written as specs in the project wiki,
and nothing is implemented before its spec is APPROVED.
The name¶
Dún Scáith (Scottish Gaelic Dùn Sgàthaich; say "doon SKAH") is the fortress on the Isle of Skye that housed Scáthach's school of arms in the Ulster Cycle. The stronghold that keeps the school supplied and holds it against attack is this tool's job: it provisions the hardware through the schedulers, guards node health, and recovers runs from preemption.
Documentation¶
The documentation follows the Diátaxis framework:
- Getting started: install and run dunscaith for the first time.
- Tutorials: learning-oriented lessons.
- How-to guides: task-oriented recipes for specific goals.
- Explanation: how dunscaith works, and why.
- Reference: commands, flags and configuration.