Set up S3 checkpoint credentials¶

A job whose checkpoint is an s3://bucket/prefix URL publishes its checkpoints with go/dispatch's publish helper. The helper needs credentials. dunscaith keeps them out of the job script and the job's environment: the script carries only a reference to where the credentials are, and the helper reads them there.

Credential sources¶

checkpoint.s3.credentials is an ordered list. The helper tries each source in turn. There are three kinds:

If you set nothing, the list is [{kind: file, profile: default}].

Write a file source with s3 login¶

dunscaith s3 login --store s3://my-bucket/checkpoints

--store is the checkpoint root dunscaith probes with the credentials once they are written. If you always publish to the same root, set it once in configuration and leave --store off:

checkpoint:
  s3:
    root: s3://my-bucket/checkpoints

--store overrides checkpoint.s3.root. With neither, s3 login stops before asking for anything. It prompts for each value, without echoing the secrets:

access_key_id: ...
secret_access_key: 
session_token (optional): 
endpoint (optional): 
region (optional): 

It then writes ~/.config/dunscaith/s3/default/, directory mode 0700, one file per value with mode 0600, and probes the store: a conditional write, a rewrite and a delete under the root. It prints wrote <dir> and probed <store>.

Use --profile <name> for more than one set of credentials. A profile name is letters, digits, ., _ and -, up to 64 characters, starting with a letter or digit.

If an existing profile is readable by anyone but you, or holds a file it should not, s3 login refuses before it asks for anything. Fix the modes (chmod 0700 the directory, chmod 0600 the files) and rotate the keys if others could have read them. If the probe fails, the profile stays written, and the error names the store that refused it.

Without a terminal¶

s3 login needs a terminal to prompt. In a script, name the environment variables that hold each value with --from-env field=VARIABLE:

dunscaith s3 login --store s3://my-bucket/checkpoints \
  --from-env access_key_id=AWS_ACCESS_KEY_ID,secret_access_key=AWS_SECRET_ACCESS_KEY

The fields are access_key_id and secret_access_key (both required), and session_token, endpoint and region (optional). A required variable that is unset or empty is an error.

Put the credentials in Kubernetes with s3 secret¶

If jobs run in pods, copy a file profile into a Secret. dunscaith runs your own kubectl with your own kubeconfig; it links no Kubernetes client.

dunscaith s3 secret --namespace training

This renders the Secret from the profile directory (kubectl create secret generic ... --dry-run) and applies it. The values reach kubectl as file paths, so none is on a command line. The Secret is named dunscaith-s3-<profile>. The command then prints the volume and mount your worker or job pods need, which you must add yourself:

volumes:
  - name: dunscaith-s3-default
    secret:
      secretName: dunscaith-s3-default
      defaultMode: 0400
containers:
  - volumeMounts:
      - name: dunscaith-s3-default
        mountPath: /var/run/secrets/dunscaith/s3/default
        readOnly: true

The mount path is /var/run/secrets/dunscaith/s3/<profile>. The command also prints, as YAML ready to paste into your configuration, the secret source that names it:

checkpoint:
  s3:
    credentials:
      - kind: secret
        path: /var/run/secrets/dunscaith/s3/default

--profile chooses the file profile to copy. On a Soperator site add --soperator: the command then warns that a Secret mounted on shared worker pods is readable by every job there, so treat it as a team credential.

Vault sources¶

With Kubernetes auth, the job's pod presents its service-account token and logs in under role:

checkpoint:
  s3:
    credentials:
      - kind: vault
        addr: https://vault.example.org:8200
        auth: kubernetes
        role: training
        path: s3/training

With AppRole, the helper reads role_id and secret_id from files in approleDir, a directory the compute nodes see. Keep it as private as a file profile (0700, files 0600): on bare-metal Slurm it is no stronger than a file source.

checkpoint:
  s3:
    credentials:
      - kind: vault
        addr: https://vault.example.org:8200
        auth: approle
        approleDir: /home/you/.config/dunscaith/vault
        mount: kv
        path: s3/training

addr is http(s)://host[:port] with no path. mount is the KV v2 mount, secret when left out, and path is the secret's path within it: s3/training, not secret/data/s3/training. kubernetes needs role and takes no approleDir; approle needs approleDir and takes no role.

The job script carries the source as the publish helper's location, for example vault:https://vault.example.org:8200/s3/training?approle=%2Fhome%2Fyou%2F.config%2Fdunscaith%2Fvault&auth=approle&mount=kv.

Reach S3 or Vault through a proxy¶

The publish helper ignores HTTPS_PROXY, HTTP_PROXY and NO_PROXY in the job's environment and connects directly. If your compute nodes reach the store or Vault only through a proxy, name it in configuration, separately for each:

checkpoint:
  s3:
    proxy: http://proxy.example.org:3128
    credentials:
      - kind: vault
        addr: https://vault.example.org:8200
        auth: kubernetes
        role: training
        path: s3/training
        proxy: http://proxy.example.org:3128

checkpoint.s3.proxy is the store's, and reaches the helper as -proxy http://proxy.example.org:3128. A vault source's proxy is for that Vault server only, and joins its location as proxy=http%3A%2F%2Fproxy.example.org%3A3128. Leave either out to connect directly. file and secret sources make no network call and take no proxy.

A proxy is http:// or https://, a host and an optional port. A user name or password in it is refused, because it would be written into the job script; use a proxy that admits the compute nodes by address. There is no exclusion list: a proxy named is used for every request to that server.

s3 login probes the store from the login node, not from a compute node, and ignores the proxy variables in its own environment too. If the login node reaches the store only through a proxy, name that one separately; it is used for the probe and never written into a job:

checkpoint:
  s3:
    loginProxy: http://login-proxy.example.org:3128

It follows the same form rule as the others. Leave it out and the probe connects directly, even when checkpoint.s3.proxy is set.

Rules the setting follows¶

• A file source takes only profile; a secret source only path; a vault source addr, auth, path, mount, proxy, and role or approleDir as its auth needs. Any other key is refused.
• A secret source's path and a vault source's approleDir must be absolute and contain no ,. Each is the path as the compute node sees it.
• A source may not repeat. An empty list is refused; leave the key out for the default.

Run the files and checks above on a machine where $HOME is the same directory the compute nodes see, or the file source will point at a path they cannot read.

Check the result¶

ls -ld ~/.config/dunscaith/s3/default      # drwx------
ls -l  ~/.config/dunscaith/s3/default      # -rw------- for each file