---
title: "Submitting jobs"
image: "https://docs.virdx.dev/img/virdx-social-card.png"
---

> Documentation Index
> Fetch the complete documentation index at: https://docs.virdx.dev/llms.txt
> Use this file to discover all available pages before exploring further.

# Submitting jobs

{/* Vendored from infra_k8s/SUBMITTING_JOBS.md @ ff0b56e by scripts/sync-docs.sh. Do not edit here. */}

This is the canonical guide for users and agents submitting workloads to this
cluster. It covers `kuberun`, the shared `run-image` WorkflowTemplate, and custom
Argo Workflows. Ordinary jobs belong in namespace `argo-workflows`, using the
submitter's authenticated Kubernetes identity. Start with the existing project
configuration and the workload's actual resource and restart requirements.

## Choose a scheduling mode

Personal jobs are guaranteed RTX 6000 Ada GPUs; team jobs stay within their
RTX 6000 PRO entitlement; preemptible jobs borrow whatever is idle, PRO
first. All three modes share the same Kueue cohort.

| Mode | kuberun options | Workflow `metadata.labels` | Placement | Borrowing |
|---|---|---|---|---|
| Personal | `--no-team --no-preemptible` | Neither mode label needed | Ada (compute-002) only, within the 2-GPU allowance | None; `gpu-class: rtx-6000-pro` is rejected |
| Team | `--team <name> --no-preemptible` | `virdx.dev/team: <name>` | PRO nodes (compute-001/003) only; members of the named team (`ve2e`, `histo`, `diffsim`) | None |
| Preemptible | `--no-team --preemptible` | `virdx.dev/preemptible: "true"` | PRO nodes first, then Ada, subject to explicit hardware constraints | Idle GPUs, fair-shared between users; evicted whenever a guaranteed Pod needs the node |

Scheduling teams `ve2e`, `histo` and `diffsim` are defined in infra_iam
(quotas 12 / 5 / 4 PRO, 192 / 128 / 128 CPU, 2Ti / 800Gi / 800Gi). A user may
belong to several teams and names the one to draw on per job: `--team histo`
or `team: histo` in the project config. A submission naming a team the
submitter is not in (or no such team, including the legacy value `"true"`
from older clients) is rejected, as are team + preemptible and team +
`gpu-class: rtx-6000` (team queues are PRO-only).

Personal defaults are **36 CPU / 264Gi / 2 Ada GPUs across admitted Pods**,
including sidecars, all on Ada. RTX 6000 PRO GPUs are team quota: use
`--team <name>` as a member, or `--preemptible` (PRO first, evictable) otherwise;
a personal `--gpu-class rtx-6000-pro` is rejected by kuberun and by the
cluster. Full-GPU Coder workspaces using the personal queue
share this allowance; `nvidia.com/gpu.shared` slices do not count toward it.
Two of the usual **16 CPU / 128Gi** containers fit alongside their Argo wait sidecars.
Authoritative quotas and any overrides live in `infra_iam/data/scheduling/`
and its rendered `k8s/scheduling/` manifests.

Personal GPU quotas deliberately exceed physical capacity. They are admission
caps, not reserved cards: an admitted personal Pod can still wait for a GPU
behind other personal work. It does not wait behind preemptible work: when no
GPU is free, the scheduler evicts preemptible Pods to place it. The same
applies on the PRO nodes: a team Pod evicts preemptible Pods holding its
cards.
Coordinate long personal GPU runs. Admission also cannot guarantee that a Pod
fits on an individual node.

Preemptible capacity has no per-user cap, but it is **fair-shared**: each
submitter has their own `user-<name>-preemptible` LocalQueue into the shared
pool, and Kueue admits pending preemptible work in order of each user's
decayed historical GPU usage (48-hour half-life). Alone, you can use every
idle GPU; as soon as someone else queues preemptible work, their steps go
ahead of yours until usage evens out. Check your standing with
`kubectl get localqueue -n argo-workflows user-<name>-preemptible -o jsonpath='{.status.fairSharing}'`.

Preemptible jobs must tolerate being killed and restarted. The policy supplies
template defaults of a **12-hour deadline per Pod attempt and 10 retries**;
these are not a checkpoint mechanism or a total Workflow runtime guarantee.
Applications must persist progress, resume explicitly, and tolerate repeated
execution. Do not opt into preemption merely to obtain more resources unless
the workload can handle that behavior.

## Choose GPU hardware

| Hardware | Workflow `virdx.dev/gpu-class` (client setting: `gpu_class`) | Backend modes |
|---|---|---|
| RTX 6000 Ada, 48 GB VRAM | `rtx-6000` | Personal, preemptible |
| RTX PRO 6000 Blackwell, 96 GB VRAM | `rtx-6000-pro` | Personal, preemptible |
| Either type | Omit the label; see client compatibility below for `auto` | Personal, preemptible |

For raw Workflows, omit the GPU-class label to allow either type. Use an
explicit type when VRAM, architecture, or software compatibility requires it:
for example, a job needing 64 GB of VRAM requires `rtx-6000-pro`.
`auto` is **not a valid Kubernetes GPU-class label**.

**Client compatibility:** vxrun 0.2 and later resolve personal and
preemptible `gpu_class: auto` to "either type" (no label) and accept an
explicit `rtx-6000-pro` in personal mode; only `--team` with `rtx-6000` is
rejected. vxrun 0.4 and later take a team name (`--team histo`); older clients
send `virdx.dev/team: "true"`, which the cluster rejects. Older kuberun/vxrun
clients still resolve personal `auto` to Ada and
reject personal PRO: upgrade, or submit `run-image` or a custom Workflow
directly as shown below. Use `--no-team` to clear an inherited team.

For raw Workflows, put any required GPU-class label in `metadata.labels`;
Kyverno propagates it to Pods and supplies matching selectors. Prefer this over
handwriting `nodeSelector` or node affinity. A manually supplied GPU product
selector is a hard requirement, not a preference. The shared flavor accounts
for either type without injecting a conflicting product or node class;
kube-scheduler enforces the supplied selector. Do not strip a hardware
requirement to make a job start.

For CPU-only work, use `gpu: 0` in kuberun/run-image; custom containers should
omit `nvidia.com/gpu` resources and the NVIDIA runtime patch. Do not add a
GPU-class label or hardware selector. The queue still determines the pool;
Kueue adds `nvidia.com/gpu.present: "true"` at admission, even for CPU jobs,
so ordinary workflows use the GPU compute nodes, not CPU-only infra nodes.

## Submit a project with kuberun

Use kuberun when building and running code from an application repository. It
builds the image and submits a Workflow referencing `run-image`.

Example `.kuberun.yaml` for a personal one-GPU job:

```yaml
cpu: "16"
memory: "128Gi"
gpu: 1
gpu_class: auto
team: ""          # or a team name such as histo
preemptible: false
request_ratio: 1.0
```

Use the project's Dockerfile command, or specify `entrypoint` / `cmd` in the
config or on the command line. Inspect the effective settings with
`kuberun config`. CLI flags override project `.kuberun.yaml`, which overrides
global `~/.kuberun/config.yaml` (with its `KUBERUN_*` environment overrides).
Review the working tree and kuberun's `autocommit` setting before running:
building may commit and push local changes.

```sh
# Personal quota, either GPU type (older clients select Ada here)
vxrun run --no-team --no-preemptible --gpu 1 --gpu-class auto

# Borrow either type, with interruption/restart accepted
vxrun run --no-team --preemptible --gpu 1 --gpu-class auto

# Borrow PRO specifically, for a workload requiring that hardware
vxrun run --no-team --preemptible --gpu 1 --gpu-class rtx-6000-pro
```

The explicit negative flags clear inherited scheduling modes. Likewise,
`--gpu-class auto` clears an inherited GPU preference (older clients still
select Ada in personal mode). For ordinary CPU jobs, change the GPU count to
zero and choose the appropriate mode.

`cpu` and `memory` are container limits. Kuberun's `request_ratio` defaults to 0.5;
values from 0.1 to 1.0 reduce CPU/memory requests relative to those limits, with
rounding and a minimum of 1 CPU / 1Gi. For example, 16 CPU / 128Gi at 0.5
requests 8 CPU / 64Gi. GPU requests are unchanged. Lower requests consume less
admission quota but permit more contention; choose them from expected usage,
not simply to fit more jobs. Ratio 1.0 is a predictable starting point.
Direct `run-image` submissions keep requests equal to limits unless
`cpu_request` / `memory_request` are explicitly supplied.

### Multiple projects in one repository

With the monorepo-aware kuberun release, `kuberun -C apps/worker run` selects the
nearest `.kuberun.yaml` up to the Git root. `kuberun --config PATH run` selects an
explicit config relative to `-C` (or the caller's directory). These global
options precede the command. Parent configs are not merged; an invalid selected
config fails without falling back. YAML paths are relative to their config;
CLI Dockerfile/context paths remain repo-relative. Paths must stay within the
repo, including symlink targets, and the Dockerfile must be within the build
context. For example, `apps/worker/.kuberun.yaml` can contain:

```yaml
project: worker
context_dir: ../..
dockerfile: Dockerfile
autocommit: true
```

This config uses the whole repo as build context. Without an explicit context,
the default is also the repo root unless global config overrides it. A nested
config discovers its own Dockerfile when none is specified. The config
directory, Docker build context, and container WORKDIR are separate concepts.

The source repository is still `mono`. Project identity controls job names,
labels, and `/mnt/artifacts/<user>/<image>/<workflow>` output paths (vxrun passes
the project identity as the `image` parameter). The image
repository is derived from the source repo and full config-directory path:
`mono/apps/worker` here, distinct from `mono/tools/worker` or a `worker` repo.
There is no `image_name` config override; changing the project label or build
context does not change the image name. Configs in the same directory share an
image repository; moving the config changes it. Invalid OCI paths are rejected.
The image name also participates in the cache key.

Kuberun passes this derived name through the backend's `image-name` (Tekton) or
`image_name` (Argo buildah) parameter. Admission requires the source repo name
or a path beneath it, rejecting a submission from `mono` to `foo` or `other/foo`.
Tekton additionally allows centrally approved legacy aliases only for their
exact repo/context/Dockerfile mapping in
[the image-name policy](https://github.com/virdx/infra_k8s/blob/ff0b56e44d5354ab6a403ed2751da3351e8aa751/current/tekton-builds/policies/image-name.yaml). For
example, `mono` + `.` + `apps/vxdata-jobs/Dockerfile` may publish `vxdata-jobs`.
The policy supplies the alias for `buildkit-image` when the parameter is
omitted. Root-config kuberun builds omit it for compatibility; the nested
`apps/vxdata-jobs/.kuberun.yaml` config publishes `mono/apps/vxdata-jobs` instead.
Argo buildah also checks the namespace before building, including templateRef
calls. These checks govern the supplied build pipelines/templates; registry
credential authorization remains a separate boundary.

Autocommit commits only the selected config directory and its descendants,
including additions/deletions and hook fixes. Staged changes elsewhere remain
staged. Uncommitted shared packages outside that scope are excluded; builds use
their committed versions. Commit those changes separately when required. A
selected Dockerfile outside the commit scope must be clean, so kuberun's parsed
command matches the version used remotely. A symlinked config keeps its selected
directory as the commit scope, rather than the target file's directory.
Without a config, scope is the repository root. Use `--no-autocommit` to require
manual commits; the built-in default is `autocommit: true`, which project/global
settings may override. Pushing still publishes existing commits on the branch.

Jobs carry separate repo and project labels. Use `kuberun status --repo mono`
or `kuberun status --repo mono --project worker` to filter them. Older jobs
without a project label remain visible under the repo filter.

Use `kuberun config` for setting sources, `run --dry-run` / `build --dry-run` for
a local plan, and `kuberun init` to create files explicitly. Config inspection
and dry runs do not commit, push, authenticate to the cluster, or generate files.

Deploy the companion templates/tasks/policy before releasing the CLI. Tekton
cache lookup, build, and archive keys now include a hash of build arguments;
empty arguments retain legacy keys. Build and cache-hit tasks export the actual
image repository/digest. Argo buildah exports the pushed `image-ref`, and
run-image accepts `image_ref` for immutable execution while retaining `image`
for the output directory and backwards-compatible tagged submissions.

## Submit an existing image with run-image

Use the installed `run-image` WorkflowTemplate when the job is one container
with standard credentials and storage. Image builds normally use kuberun's
Tekton backend; build manifests are under
[current/tekton-builds/examples](https://github.com/virdx/infra_k8s/tree/ff0b56e44d5354ab6a403ed2751da3351e8aa751/current/tekton-builds/examples).
For Pixi Dockerfiles, see [Dockerfile advice](https://github.com/virdx/infra_k8s/blob/ff0b56e44d5354ab6a403ed2751da3351e8aa751/DOCKERFILE_ADVICE.md) for the shared
cache mount and layer ordering, and [BuildKit package caches](https://github.com/virdx/infra_k8s/blob/ff0b56e44d5354ab6a403ed2751da3351e8aa751/docs/buildkit-cache.md)
for retention behaviour.

Save this as `run-image-job.yaml`. Replace the example image name, tag, and
command with an image already present in Zot and its actual command.
Optional `image_ref` (`repo@sha256:...`) pins an exact digest and overrides
`image`/`image_tag`; kuberun sets it for every build:

```yaml
apiVersion: argoproj.io/v1alpha1
kind: Workflow
metadata:
  generateName: run-example-
  namespace: argo-workflows
spec:
  workflowTemplateRef:
name: run-image
  arguments:
parameters:
  - name: image
    value: example-job
  - name: image_tag
    value: replace-with-built-tag
  - name: cmd
    value: "python -u /app/job.py"
  - name: cpu
    value: "16"
  - name: memory
    value: "128Gi"
  - name: gpu
    value: "1"
```

This example uses personal quota and can run on either GPU type. To require
PRO, add `metadata.labels: {virdx.dev/gpu-class: rtx-6000-pro}`. For Ada, use
`rtx-6000`. To borrow instead, add `virdx.dev/preemptible: "true"`. The same
labels apply to custom Workflows below. Kubernetes label values must be
strings, so quote `"true"`.

```sh
kubectl create --dry-run=server -f run-image-job.yaml -o yaml
kubectl create -f run-image-job.yaml
```

The image name is relative to `zot.fra.virdx.dev:5000`, not a full registry URL.
Prefer a specific built tag over `latest`. Set **at least one of `cmd` or
`entrypoint`** for manual submissions: the template uses an Argo script wrapper
and does not discover the image's Dockerfile defaults. Kuberun resolves those
defaults before submitting. `cmd` alone is executed through `sh -c`;
`entrypoint` plus `cmd` uses shell word splitting, so put complex launch logic
in an executable script in the image.

The template defaults to 16 CPU / 128Gi / 0 GPUs, with requests equal to limits.
Optional `cpu_request` and `memory_request` parameters override requests;
they must be positive and no greater than the corresponding limits. Supported
limit values are:

| Parameter | Accepted values |
|---|---|
| `cpu` | 1, 2, 4, 8, 16, 32, 64, 128 |
| `memory` | 2Gi, 4Gi, 8Gi, 16Gi, 32Gi, 64Gi, 128Gi, 192Gi, 256Gi, 512Gi |
| `gpu` | Integers 0–8; quota and per-node capacity still apply |
| `local_cache` | 0 (disabled), 16Gi, 32Gi, 64Gi, 128Gi, 256Gi, 512Gi, 1024Gi |

`run_environment` selects `production` (default) or `staging` credentials.
The template provides ClearML/S3 configuration, environment-specific credentials,
registry authentication, the internal CA, `/dev/shm`, read-only `/mnt/storage`,
and `/output` backed by `/mnt/artifacts/<username>/<image>/<workflow-name>`.
S3 credentials are exposed through the standard shared credentials file
(`AWS_SHARED_CREDENTIALS_FILE`), with the same profiles as Coder workspaces:
`[scratch]` (read/write scratch bucket), `[workflow-output]` (read-only
published batch results) and `[default]` (the same read-only identity as
`[workflow-output]`); every profile carries the internal RGW `endpoint_url`,
and none is real AWS. See the
[template](https://github.com/virdx/infra_k8s/blob/ff0b56e44d5354ab6a403ed2751da3351e8aa751/current/argo-workflows/templates/run-image.yaml) for exact mounts and
environment variables; never copy secret values into job manifests.

`local_cache` provisions node-local scratch at `/cache` and exports
`VXRUN_CACHE_DIR=/cache` and `VXS3_CACHE_DIR=/cache/.vxs3` (vxs3's default
download/unpack root). Directory variables follow `VXRUN_<NAME>_DIR`
(`VXRUN_OUTPUT_DIR` is always set); `KUBERUN_OUTPUT` and `KUBERUN_CACHE` are
legacy aliases with the same values, kept until no worker code reads them.
Treat `/cache`, `emptyDir`,
and the container filesystem as disposable across retries and deletion. Store
checkpoints in `/output`, shared artifacts, object storage, or an appropriate
persistent volume, and explicitly reload them after a restart. A newly submitted
Workflow has a different `/output` directory, so resuming a previous submission
requires its checkpoint location.

For temporary development checkpoints, prefer the `scratch` S3 bucket over
`/output`: the NFS export behind `/mnt/artifacts` is a poor fit for frequent
large rewrites. See [Scratch storage for development](#scratch-storage-for-development)
for retention, access and publishing data you want to keep.

The Argo UI can submit the same template with these parameters and Workflow
labels. For manifests, use `kubectl create` with an authenticated user
kubeconfig; there is no need to supply creator or username labels yourself.
Kyverno derives identity from the API request; through Argo Server, it uses the
creator identity set by the server.

## Scratch storage for development

The `scratch` S3 bucket facilitates development: use it for experiments,
intermediate data and temporary checkpoints while working out what you need.
It has a **two-week (14-day) garbage-collection policy**. Treat its contents as
disposable; scratch is not for permanent storage or inputs and outputs that
must remain available for repeatable workflows.

Once you are happy with the data, model or result, **publish it to vxdata and
create a permanent resource**. Use that resource in subsequent workflows.
The retained bytes must live in vxdata storage; a resource that only points
at `s3://scratch/...` will still lose its data to scratch GC. See
[Vxoptimise service workflows](#vxoptimise-service-workflows) for publishing
an immutable S3 prefix with `vxrun storage upload` and `--retention permanent`.
Finish publication before the scratch objects expire.

### Access from Coder and jobs

Coder workspaces, `run-image` and `run-batch` jobs expose
the `[scratch]` profile through `AWS_SHARED_CREDENTIALS_FILE`. It carries
`endpoint_url = https://s3.fra.virdx.dev`. Use
`boto3.Session(profile_name="scratch")`, `s3fs.S3FileSystem(profile="scratch")`
or `AWS_PROFILE=scratch aws s3 ...`. For example, this read-only boto3 check
uses the workspace or job's internal CA bundle:

```python
import os
import boto3

s3 = boto3.Session(profile_name="scratch").client(
"s3",
endpoint_url="https://s3.fra.virdx.dev",
verify=os.environ["SSL_CERT_FILE"],
)
result = s3.list_objects_v2(Bucket="scratch", MaxKeys=5)
print("S3 OK — objects returned:", result["KeyCount"])
```

The Coder template keeps your persistent local home PVC, `/mnt/storage`
read-only and `/mnt/artifacts` read/write. Startup does not create directories
or change ownership on NFS, so it requires no writes to a full or read-only
export. Application writes to `/mnt/artifacts` can still fail when the export
is full or read-only. Write development data to your home and upload results to S3.
Existing workspace Pods pick up this change after the updated
`kubernetes-dev` template is published and the workspace is rebuilt.

Write under `s3://scratch/<username>/<experiment-or-workflow>/`. The scratch
identity is shared, so this prefix is an organizational convention, not an
access boundary. The bucket also has a shared size quota; delete temporary
data you no longer need. Retrying or resuming an experiment does not exempt
its objects from the 14-day GC policy.

### Keep scratch request load modest

Scratch shares Ceph with other workloads. Before uploading a directory, inspect
the local source with the advisory analyzer:

```sh
vxrun data inspect ./data --scratch
```

This uses `vxs3` to report object counts, sizes, the share below 4 MiB and
format-specific layout findings. It suggests changes without uploading data or
blocking submission. `vxrun review` also flags scratch references for review;
a reference alone does not mean the workload is inefficient.

- Consolidate related small records into fewer native files, such as Parquet,
  when the readers support it. For independent ordinary files, use `vxs3` tar
  bundles targeting 1 GiB shards and keep the manifest with the shards.
- Keep Zarr native: review chunking and sharding instead of tarring the store.
  A few large files or occasional checkpoints may need no layout change.
- Stage inputs once per job onto local NVMe (`--local-cache`, mounted at `/cache`)
  and reuse them across epochs and workers. Batch small writes, avoid repeated
  S3 listings, and keep transfer concurrency modest across workers.

Prefer inspection before upload. For remote inspection, select only your
experiment's prefix; do not repeatedly scan scratch during training. Scratch
mode refuses whole-bucket scans and defaults to 10,000 objects, 64 metadata
probes, 4 MiB of metadata and a 30-second cooperative time budget. LIST/HEAD
requests are outside the metadata-probe budget, so these are not hard limits
on total S3 requests or elapsed time. Remote analysis uses the analyzer's
environment-based S3 credentials and endpoint; selecting the boto3 `[scratch]`
profile alone does not configure it.

Check coverage: a partial or failed scan is not a clean bill of health. Layout
analysis does **not** measure IOPS; also review how often the job issues
GET/LIST/PUT requests and whether each worker repeats the same operations.
See vxcluster's `docs://storage-practices` for packing and inspection examples.

## Submit an image batch with run-batch

Use `run-batch` for a finite set of independent commands sharing one image and
resource allocation. It runs one DAG task per JSON item with bounded parallelism.
One failed item does not stop the remaining items; the overall Workflow reports
failure when an item exhausts its attempts. Outputs are written to temporary
node-local storage and, by default, published to the `workflow-output` bucket as
one vxs3 bundle per batch (see [publication](#batch-output-publication)).

The matching CLI exposes `vxrun batch init --items items.json`, then
`plan`, `render`, `build`, `submit` and `status`, alongside `vxrun workflow`.
Deploy [run-batch.yaml](https://github.com/virdx/infra_k8s/blob/ff0b56e44d5354ab6a403ed2751da3351e8aa751/current/argo-workflows/templates/run-batch.yaml) through
ArgoCD before releasing/using that CLI feature. The `namespaces`, `rbac`, and
`argo-workflows-config` applications must all reconcile the batch changes:
the output-volume steps require the `batch-output-volumes` ServiceAccount,
Role, and RoleBinding in `argo-workflows`. Verify a small batch reaches a
terminal phase with a persisted node tree; a finaliser log alone does not prove
the controller could save workflow status. An `items.json` file contains:

```json
[
  {"id": "trial-0", "argv": ["python", "train.py", "lr=0.01", "seed=1"]},
  {"id": "trial-1", "argv": ["python", "train.py", "lr=0.001", "seed=2"]}
]
```

Each item contains `id` and `argv`, optionally `inputs` (below; older clients
that omit it keep working). IDs must be unique, 1–100 letters,
digits, dots, underscores or hyphens; `.` and `..` are invalid. `argv` is the full
command array, including the executable; whitespace and shell metacharacters stay
literal. For Pixi images, start it with `["pixi", "run", ...]`. The image needs
`sh`, `mkdir` and `cat`; it does not need vxrun/vxs3. The CLI rejects NUL and Argo
template delimiters in arguments and caps inputs at 1,000 items / 512 KB per
Workflow. Apply these same limits when submitting manually.

Direct native submissions use the same template:

```yaml
apiVersion: argoproj.io/v1alpha1
kind: Workflow
metadata:
  generateName: training-batch-
  namespace: argo-workflows
spec:
  workflowTemplateRef:
name: run-batch
  parallelism: 2
  arguments:
parameters:
  - name: vxrun-image
    value: zot.fra.virdx.dev:5000/example-job@sha256:REPLACE_WITH_DIGEST
  - name: items
    value: '[{"id":"trial-0","argv":["python","train.py","seed=1"]},{"id":"trial-1","argv":["python","train.py","seed=2"]}]'
  - name: cpu
    value: "8"
  - name: memory
    value: "32Gi"
  - name: gpu
    value: "1"
```

Set a real immutable image digest before submitting. Standard personal/team/
preemptible scheduling labels apply to the whole Workflow, as for `run-image`.
Default per-item resources are 2 CPU / 4Gi / 0 GPUs, with equal requests and
limits. Resource enums match `run-image`; `spec.parallelism` is 1–25.
`run_environment` chooses production/staging credentials. `retries` defaults to
2 (0–10); `retry_policy` defaults to `OnError` and also supports `OnFailure` or
`Always`. `task_deadline` defaults to 43200 seconds (CLI range 1–604800).
Application failures do not retry under the default; checkpoint/resume logic
remains the application's responsibility.

Each item has a temporary TopoLVM PVC. Each attempt writes to a separate
directory on that claim, mounted at `/output` (`VXRUN_OUTPUT_DIR`). The workflow
UID, item ID and Pod name isolate reused workflow names, items and retries.
`VXRUN_ITEM_KEY`, `VXRUN_ATTEMPT`, `VXRUN_WORKFLOW` and `VXRUN_WORKFLOW_UID` identify
the execution. Each worker node exports `item-id` and `output-path` parameters;
inspect its status to distinguish successful outputs from failed attempts.
`output-path` is an internal path on the item's claim, not a shared filesystem
path accessible from Coder. Use `vxrun batch results` for published results.

The same claim carries one `/checkpoints` directory per item
(`VXRUN_CHECKPOINTS_DIR`), shared by every attempt of that item: write resumable
state there and, on start, resume from whatever a previous attempt left. It is
never packed into the published bundle and is deleted with the claim, so it
covers retries of an item, not a resubmitted batch or a lost node (retries are
pinned to the claim's node, see below). Checkpoints that must survive either go
to the `scratch` bucket, keyed by run ID. The claim always exists (16Gi when
`local_cache` is `0`, `local_cache` otherwise) and is shared by inputs, every
attempt's output and checkpoints. The claim layout is:

```text
<user>/batches/<workflow-name>-<workflow-uid>/items/<item-id>/
  attempts/<pod-name>/   /output of one attempt; the successful one is packed
  inputs/                declared inputs, read-only
  checkpoints/           /checkpoints, shared by the item's attempts
```

Mounts include the standard credentials/CA, `/output`, `/dev/shm` and `/cache`.
There are no NFS or hostPath mounts: read inputs from S3/vxdata into `/cache`.
`local_cache: "0"` uses a Pod-local emptyDir at `/cache` and a **16Gi output PVC
per item**. A nonzero `local_cache` sets both the per-Pod scratch capacity and
the per-item output PVC capacity. These are separate allocations: allow space
for cached inputs, output across all attempts, and packing shards.
`VXS3_CACHE_DIR=/cache/.vxs3` in both cases.

The first worker binds its item's claim to a node. Retries and its publisher
must use that node; different items can run on different nodes. Local storage
cannot fail over to another node during a node outage. The claim survives worker
Pod garbage collection while publication is pending. An item exit hook requests
deletion on success, failure or Stop; Kubernetes releases the volume once its
worker/publisher Pods are removed (normally after the one-hour log retention).
Terminate skips hooks; Workflow deletion/TTL is the ownership cleanup backstop.
S3 is the durable output. With `publish: "false"`, `/output` is temporary too:
the application must upload anything it needs to retain. `run-image` keeps its
existing mounts and storage behaviour.

### Declared inputs

A batch can declare the data it reads instead of fetching it from inside the
command. The `inputs` parameter (shared by every item) and each item's `inputs`
list vxs3 bundle selections:

```json
[{"name": "weights", "manifest": "s3://vxdata/models/grader/v3/manifest.parquet",
  "sha256": "<manifest digest>", "paths": ["fold2/checkpoint.pt", "config.yaml"]},
 {"name": "case", "manifest": "s3://vxdata/cohorts/bpmri/2026-10/manifest.parquet",
  "sha256": "<manifest digest>", "prefix": "cases/0017/"}]
```

`manifest` + `sha256` pin the bundle (a vxs3 `BundleRef`); one of `prefix`
(a directory), `paths` (exact members) or `pattern` (globs) selects from it, and
no selector takes the whole bundle. Names are unique across shared and item
inputs and become `/inputs/<name>` in the worker, read-only (`VXRUN_INPUTS_DIR`).
The worker's `fetch-inputs` init container (vxrun-tools image) extracts them
onto the item's volume before the command
starts, verified against the pinned digest; a retried attempt reuses what is
already there and a changed pin for the same name fails. Items without
declared inputs skip the fetch. `s3://vxdata` bundles are read through
vxdata API presigned URLs with the run environment's `VXDATA_TOKEN`; other
buckets use the read-only `[default]` profile. Count inputs in `local_cache`: they share the
item volume with outputs. `env` is a JSON list of `{name, value}` merged into
the worker environment; cluster-owned names (`VXRUN_*`, `AWS_*`, `VXS3_*`,
`CLEARML_*`, ...) are refused by the CLI.

`vxrun spec` (mono) authors all of this as one YAML file: image, resources,
named bundles, inputs, items and outputs. `vxrun spec resolve` pins catalog
resources and sizes every selection, so the file states the minimum
`local_cache`; `vxrun spec compile` produces the `run-batch` Workflow above.
The command keeps cluster network access in this version; a network-isolated
mode is declared in the spec format but not enforced yet.

### Batch output publication

With `publish: "true"` (the default) every item is a two-step sequence: the
command runs as above, then a `pack` step in `tools-image` (the vxrun CLI with
vxs3) packs the successful attempt's `/output` into a vxs3 bundle and publishes
it. An exit handler then completes the batch:

```text
s3://workflow-output/<username>/<workflow-name>-<workflow-uid>/
  workflow.json                 name, uid, user, status, expected item ids (written first)
  items.parquet                 one row per expected item: status, manifest reference, files, bytes
  manifest.parquet              written last, only when every item succeeded: the batch is complete
  items/<id>/manifest.parquet   written last by that item's pack step: the item is complete
  items/<id>/shard-*.tar
```

Shards never move. The root manifest references each item's shards where its
pack step published them (a vxs3 >= 3.1 composed bundle), so one `open_bundle`
on the root sees `items/<id>/...` for every item, and each item's own manifest
stays a complete bundle. The root exists only for a fully successful batch, like
a multipart upload that becomes an object only on completion: a batch with a
failed item keeps `workflow.json`, `items.parquet` and the successful items'
bundles for inspection and re-runs, but gets no root manifest and no catalog
entry. `register: "true"` registers a complete root as a vxdata Bundle with
`items.parquet` as its records table, using the `run_environment` vxdata
credentials. (Finalisers before vxrun 0.4.7 wrote a root manifest for partial
batches too; `items.parquet` tells them apart.)

Collect with the CLI, from a Coder workspace or a job; both carry the read-only
`[workflow-output]` profile:

```sh
vxrun batch results <workflow-name>                      # prefix, completion, one line per item
vxrun batch results <workflow-name> --download ./out     # complete batch -> ./out/items/<id>/...
vxrun batch results <workflow-name> --download ./out --item trial-0   # one item, also when incomplete
vxrun batch results s3://workflow-output/<user>/<workflow-name>-<uid>  # by prefix
```

Workflows are garbage-collected three days after they finish; the data is not.
`results` reads the live Workflow when it exists, otherwise finds
`<user>/<name>-<uid>/` in the bucket and takes name, uid and expected items from
`workflow.json`, so results stay readable after the Workflow is gone.

The pack step mounts the item's output PVC read-only and assembles all of
that item's shards on disk in its own `/cache` before uploading. It does not
buffer whole shards in RAM. `/cache` is a separate TopoLVM scratch volume when
`local_cache` is nonzero, otherwise a disk-backed emptyDir. With publishing on,
`local_cache` should cover an item's outputs and packing space. A pack retry after a partial
upload fails, because the exact destination must be empty and the writer identity
cannot delete; the item is then reported as failed. Terminate skips exit
handlers, so a terminated Workflow has neither `items.parquet` nor a root
manifest; prefer Stop, which records the items that did finish.

Published outputs are immutable from a user's point of view. The writer identity
(RGW user `workflow-writer`: put, get, list, no delete) lives in the
`workflow-output-writer` secret, which only the pack and finalise pods mount;
user containers get the bucket read-only through the `[workflow-output]` profile
of `aws-credentials`. The Kyverno policy `argo-restrict-workflow-output-writer`
rejects any submission that references the writer secret, any run-batch
submission that adds `spec.templates` or `spec.podSpecPatch`, any pod mounting
the secret that is not a controller-created vxrun-tools pod, and in-cluster
edits of ArgoCD-managed WorkflowTemplates. Only the fixed PVC allocation and
cleanup steps use the `batch-output-volumes` service account; user workers
have no PVC lifecycle permissions. `publish: "false"` skips publishing and
finalisation, but still deletes the temporary output claim after each item.

The pack and finalise steps run the `vxrun-tools` image: the released vxrun
CLI with vxs3 and the vxdata SDK from the conda channel, built from
[builds/vxrun-tools](https://github.com/virdx/infra_k8s/blob/ff0b56e44d5354ab6a403ed2751da3351e8aa751/builds/vxrun-tools/README.md) with the Tekton example
`pipelinerun-vxrun-tools.yaml`. It is pinned by digest inside the template and
is deliberately not a parameter, so no user-chosen image ever runs with the
writer identity. Each item node exports `manifest` (JSON `{manifest_uri,
sha256, files, bytes}` from pack, empty when publishing is off) alongside
`output-path`; the inner worker node exports `item-id` and `output-path`.

## Write a custom Workflow

Use a custom Workflow for a DAG, multiple steps, or container/volume settings
that `run-image` does not expose. This is a complete one-GPU **personal** example;
replace the image and command before submitting as `custom-job.yaml`:

```yaml
apiVersion: argoproj.io/v1alpha1
kind: Workflow
metadata:
  generateName: custom-example-
  namespace: argo-workflows
spec:
  entrypoint: main
  parallelism: 1
  imagePullSecrets:
- name: zot-registry-credentials
  podSpecPatch: '{"runtimeClassName":"nvidia"}'
  templates:
- name: main
  container:
    image: zot.fra.virdx.dev:5000/example-job:replace-with-built-tag
    imagePullPolicy: Always
    command: [python, -u, /app/job.py]
    resources:
      requests:
        cpu: "16"
        memory: 128Gi
        nvidia.com/gpu: "1"
      limits:
        cpu: "16"
        memory: 128Gi
        nvidia.com/gpu: "1"
```

CPU/memory requests and limits must be explicit for each application container.
GPU requests and limits must be equal whole numbers. GPU Pods require
`runtimeClassName: nvidia`, set through `podSpecPatch` here; apply it only to
GPU templates when mixing CPU and GPU steps. Standard Argo sidecar requests are
injected by policy. Custom Workflows must explicitly configure any storage,
credentials, or CA mounts they need; they do not inherit run-image's mounts.
Use `imagePullPolicy: Always` for internal registry images because tags can
change; the example above sets it explicitly.
Before making this example preemptible, add persistent checkpoint/output
storage and application restart behavior.

For human-submitted Workflows, permitted hostPath roots are `/mnt/storage`,
`/mnt/scratch`, and `/mnt/artifacts`; `/mnt/storage` must be mounted read-only.
Prefer the existing run-image mounts when they meet the workload's needs.

Do not set Kueue queue labels, Kueue WorkloadPriorityClass labels,
`priorityClassName`, `preemptionPolicy`, or identity labels yourself, including
inside template metadata or `podSpecPatch`. Kyverno assigns queue, priority,
identity, the Workflow service account, and the per-user concurrency semaphore.
Final Pods must match their owning Workflow's routing. Mode and queue are
immutable after submission; create a new Workflow to change them.

Reusable WorkflowTemplates belong in
`current/argo-workflows/templates/<name>.yaml` and are deployed through ArgoCD
after merge. Ordinary users cannot create them directly. One-off `Workflow`
objects can be submitted with `kubectl create -f custom-job.yaml`.

## D3N routing inventory

All newly created Argo workflow Pods in `argo-workflows`, `autoresearch`, and
`vxoptimise` receive `/etc/virdx/d3n/d3n_instances.json` through the
[`inject-d3n-config` policy](https://github.com/virdx/infra_k8s/blob/ff0b56e44d5354ab6a403ed2751da3351e8aa751/current/kyverno-policies/d3n-configmap.yaml).
This covers `run-image`, custom Workflows, referenced WorkflowTemplates and
multi-container steps, including init containers. No template-specific volume
or mount is needed. Unrelated Pods and already-running Pods are unchanged.

Coder workspace Pods receive the same mount in `coder-workspaces`, selected
by `app.kubernetes.io/name: coder-workspace` (set by the Coder Terraform
template). No workspace image rebuild or template-version update is needed.
Existing workspaces pick it up when their Pods are recreated, for example on
the next stop/start. The policy does not restart running workspaces. Both
regular and init containers receive the mount.
The workspace template also exports `VXS3_D3N_INSTANCES` pointing at that
file, so vxs3 discovery treats the deployed inventory as authoritative and
fails if the ConfigMap is absent instead of falling back to gateway routing.

The file contains stable Ketama node names and HTTPS endpoints. Ansible owns
the namespace-local `d3n-instances` ConfigMaps and generates identical content
from the RGW inventory. Do not add another copy to an ArgoCD application.
Reserve volume name `d3n-instances` and mount path `/etc/virdx/d3n` for this
shared configuration; put application-specific files elsewhere.

The mount is read-only and optional, so jobs that do not use D3N can start
before the ConfigMap is provisioned. A D3N client must explicitly read the file
and fail if it is missing or invalid, without silently using a bundled list:

```python
from vxs3.transport.instances import load_instances
from vxs3.transport.routing import select_instance

instances = load_instances("/etc/virdx/d3n/d3n_instances.json")
instance = select_instance("example", "shard.tar", instances=instances)
```

Python and JavaScript clients consume the same JSON; the hashing contract and
compatibility vectors live in `mono/packages/vxs3/ROUTING.md`. Load one snapshot
per client or operation. Projected files can update later, but an already-loaded
vxs3 tuple stays fixed. Coordinate client reloads when changing ring membership.
Making this file available does not enable Ketama routing in existing download
code; the application must use the routing functions.

For rollout, include `argo-workflows`, `autoresearch`, `vxoptimise`, and
`coder-workspaces` in
`infra_ansible/inventories/group_vars/k3s_cluster/k3s_d3n.yaml`, then run
`pixi run playbook k3s_d3n` there. The mutation policy is deployed through the
`kyverno-policies` ArgoCD application after merge, together with the narrow
Coder mount-path exception in `restrict-volume-mounts`. New workflow namespaces
must be added to both the Ansible list and the policy's namespace match.

## Size jobs for both quota and hardware

- Kueue accounts for **Pod requests**, including sidecars and the effective init
  request, across all admitted Pods in the queue. The standard Argo wait
  container adds 100m CPU / 128Mi per Pod. Two 16 CPU / 128Gi applications fit
  the default personal quota; a third does not, even without a GPU.
- GPUs cannot be split across nodes for one Pod. Current full-GPU capacity is
  **5 Ada on compute-002 and 8 PRO on each of compute-001/003**. Three other Adas
  are time-sliced for Coder and are not full `nvidia.com/gpu` capacity. An
  eight-GPU Pod exceeds personal quota even though a PRO node can fit it.
  One Pod cannot mix Ada and PRO because the types are on different nodes.
  Distributed training needs separate Pods and application support.
- Custom application containers are capped at 192 CPU / 512Gi / 8 GPUs by
  policy; run-image has the narrower CPU enum above. These ceilings are not
  entitlements or guarantees that hardware is available.
- Workflow `spec.parallelism` defaults to 20 and cannot exceed 25. This limits
  simultaneously active Pods, not the number of steps. Select it to suit the
  queue's aggregate quota. Personal GPU work can admit up to two one-GPU Pods
  or one two-GPU Pod, subject to CPU/memory quota and physical capacity.
- The per-identity semaphore defaults to 10 concurrent Workflows across modes;
  excess submissions wait. The controller also caps namespace concurrency at
  200 Workflows. Raising parallelism does not increase entitlement.

## Validate and inspect a submission

Check the Kubernetes context and effective mode/resources first. For a custom
Workflow, run `argo lint --offline --strict custom-job.yaml`. Then use
`kubectl create --dry-run=server -f <file> -o yaml` to inspect Workflow admission
and injected routing without creating the job. A server dry run does not create
the eventual Pods or prove that resource requests are schedulable; those checks
also require the quota and hardware review above.

For run-image, inspect the deployed template with
`kubectl get workflowtemplate run-image -n argo-workflows -o yaml` if it may lag
the repository. Offline lint of a template reference needs that template
available locally and does not emulate Kyverno identity injection.

After an authorized submission, inspect both Workflow and Pod state:

```sh
argo get -n argo-workflows WORKFLOW_NAME
kubectl get pods -n argo-workflows -l workflows.argoproj.io/workflow=WORKFLOW_NAME
kubectl describe pod -n argo-workflows POD_NAME
kubectl logs -n argo-workflows POD_NAME -c main -f
```

A Workflow waiting on its semaphore has not yet started its Pods. A Pod with a
Kueue scheduling gate is waiting for admission; an admitted but unschedulable
Pod may be blocked on node capacity, storage, or a hard selector. Inspect events
before changing the configuration. Operators can use `util/kueue-status` and
`util/gpu-status`; users may need an operator to inspect cluster-scoped queues.
Do not relabel an existing job or remove a required hardware constraint to
work around Pending state.

Completed Pods, including failed attempts, are normally removed after one hour.
Use Workflow status and Loki for older failures; container files and scratch
are not durable output. Workflows normally expire two days after success or
seven days after other completion.

To cancel a job, use `argo stop` (runs exit handlers), `argo terminate` (does
not), or the matching Argo UI buttons; `argo delete` also works but leaves no
record. Those are the only changes a submitter may make to a Workflow after
admission: everything else in `spec`, `status` and the labels is locked by
`argo-restrict-user-workflow-updates`, so `argo retry`, `argo resume` and
`argo suspend` need an operator. Resubmit to change a job.

## Vxoptimise service workflows

Self-service uploads and single-store optimisation use `vxrun storage` after the
backend rollout described in [the deployment notes](https://github.com/virdx/infra_k8s/blob/ff0b56e44d5354ab6a403ed2751da3351e8aa751/docs/vxoptimise.md#self-service-storage-rollout).
These commands submit fixed templates directly to Kubernetes with your Coder
OIDC login; they do not build user code or upload bytes from the client.

```sh
vxrun storage upload /mnt/storage/example/dataset --name example \
  --project example-project --description "Synthetic reference dataset" \
  --data-role reference --retention review-180d --wait
vxrun storage upload /mnt/artifacts/example/dataset --metadata metadata.json --dry-run
vxrun storage upload s3://example-source/dataset/ --metadata metadata.json --wait
vxrun storage optimise s3://vxdata/example/store.zarr/ --wait
vxrun storage status
vxrun storage status --project example-project
vxrun storage status JOB_NAME --json
vxrun logs JOB_NAME --namespace vxoptimise --follow
```

Uploads require a name, project slug, description, data role and retention
intention, supplied as flags or metadata JSON. They accept shared paths beneath
`/mnt/storage` and `/mnt/artifacts`, or `s3://bucket/prefix/` on the cluster's
object store after the S3-capable worker rollout; HOME paths cannot be used directly.
The two shared paths are existing read-only NFS mounts, with files readable by UID 1000. The
upload worker runs as UID/GID 1000 and writes only to its private scratch/output
volumes and vxdata; submitter attribution uses the authenticated workflow identity.
S3 prefixes are downloaded into private scratch using a fixed read-only source
identity, then follow the same Zarr/bundling rules. Size/count limits are checked
before download; conditional reads and a second listing reject source changes
before publication. Prefixes must be below a bucket root; they do not select
individual objects. A trailing slash is added when absent. Eligible source buckets
are controlled by S3 IAM grants. Endpoint/credential overrides are not exposed.
The CLI requires `vxrun.virdx.dev/storage-s3-contract: "1"` on the deployed
template. The repository pins the S3-capable worker with this capability enabled;
both take effect after GitOps sync. Reverting to an NFS-only image must disable
the capability in the same change.
Inputs must remain immutable. The worker accepts valid root Zarr stores,
standalone regular files of at least **100 MiB**, and ordinary directories.
Directories are always bundled into vxs3 tar shards targeting 1 GiB: small files
within a directory cannot become loose S3 objects. Small standalone files are
rejected before copying or allocation; group related files into a directory.
Nested/malformed Zarr, links and special files are rejected. Inputs are limited
to 256 GiB/100,000 entries; bundle members must fit a 4 GiB shard, while standalone
files stay native and are exempt from that shard limit. Uploads allocate a vxdata `storage`
prefix and register a verified manifest with submitter/workflow provenance.
Retention metadata does not trigger automatic deletion. Existing vxdata stores
are mirrored into `vxdata-optimised` without creating another catalog identity.

Workflow labels include `app.kubernetes.io/name=vxoptimise`,
`vxoptimise.virdx.dev/operation=upload|optimise`,
`kuberun.virdx.com/task=storage-upload|storage-optimise`, project on uploads,
and the admitted `virdx.dev/username`. They can also be selected with `kubectl -l`.
Argo reads the worker's `ARGO_PROGRESS_FILE` (`N/M`); `vxrun status` and `--wait`
show these work units, and logs contain stage updates. `vxrun storage status`
defaults to your upload and optimise jobs in the storage namespace and shows the
verified data coordinate on success, plus the vxdata resource identifier and
manifest URI for uploads. It supports a job name, `--project`, `--operation`,
`--all`, and `--json` for full result fields. Upload totals expand after
source listing and optimisation determine the object counts; S3 download logs
include file and byte progress. Mirror progress is coarse. Refresh can
lag by about a minute. The Argo web server currently cannot view this namespace;
use vxrun or `ARGO_SERVER= argo get -n vxoptimise JOB_NAME`.

Storage status reads retained Workflow objects, so results survive the five-minute
pod/scratch cleanup. Successful upload workflows expire after two days; successful
optimise workflows after 20 minutes; failed/errored workflows after seven days.
Once the Workflow is deleted, its status entry disappears. Workflow GC does not
delete published vxdata objects or catalog records. The CLI prints the stable
`vxupload-<workflow-uid>` identifier on submission; save it for later SDK lookup
with the Coder environment's `VXDATA_TOKEN`. Read `file_metadata.data_uri` only
when `file_metadata.state` is `ready`. The record may not exist for pending or
rejected jobs. Automatic catalog history is deferred; v1 status uses Kubernetes.

`--dry-run` prints the request offline and never checks credentials or remote
source existence. `--wait --timeout SECONDS` stops waiting without cancelling the
job. Pending/failed uploads require operator reconciliation before resubmission;
the vxdata record preserves the allocated prefix after Argo garbage collection.

S3 discovery and conversion run in the dedicated `vxoptimise` namespace with
separate read-only dispatcher and read/write worker S3 credentials. See [the deployment notes](https://github.com/virdx/infra_k8s/blob/ff0b56e44d5354ab6a403ed2751da3351e8aa751/docs/vxoptimise.md).
Discovery remains an administrator-operated CPU service. Self-service and
discovery share the namespace quota/semaphore; neither uses the shared
`run-image` template or personal/team queues.

After GitOps sync and Secret creation, submit the namespaced template directly
through Kubernetes (the Argo Server remains scoped to `argo-workflows`):

```sh
ARGO_SERVER= argo submit -n vxoptimise --from workflowtemplate/vxoptimise-discovery \
  -p dry-run=true -p max-stores=10
```

Omit `dry-run=true` to submit independent conversion workflows, initially with
`-p max-stores=1`. Dry and real runs share the default state database and each
observed Zarr root is next due 24 hours later, so a real run within a day of a
dry run reports `idle`; wait the interval, or give probes their own `state-db`.
The template defaults to the released image and real submission.
Upload, store and discovery templates pin the same worker release; discovery
passes its image parameter to child store workflows.
`max-stores=0` removes the submission-count limit. The hourly cron permits
5,000 scan steps or five minutes, whichever comes first, and resumes from its
persistent SQLite checkpoints on the next run. Manual template submissions
default to 10,000 steps and 45 minutes; pass `-p max-scan-steps=5000` and
`-p max-scan-seconds=300` for the same reduced RGW load as the cron.
These budgets limit scan duration and volume, not instantaneous request rate.
Adjust the bounds with `max-scan-steps` / `max-scan-seconds`;
the wrapper rejects `max-scan-seconds` above 2700 because the workflow deadline
is 55 minutes.
At 50 outstanding or 900 retained children, candidates are deferred. Dry runs
update local observations and honour the 24-hour settling delay, so the first
run can report only `settling`; use a separate database for independent probes.
Operator overrides are template parameters: `state-db` (a path on the state
claim), `scan-prefix`, `settle-seconds` and `retry-failed=true`.
Active jobs and matching completed twins are skipped, with weekly worker inventory
checks. Each child has its own pod, 64Gi LVM scratch PVC and result; twelve children
may run concurrently. Placement allows infra nodes or compute-001/003 and excludes
compute-002. The dispatcher has a permanent 10Gi LVM claim; worker pod cleanup
does not delete it. The hourly cron is enabled through GitOps and retains the
24-hour settling interval.

For an individual store (also useful after inspecting/cleaning a failed attempt):

```sh
ARGO_SERVER= argo submit -n vxoptimise --from workflowtemplate/vxoptimise-store \
  -p store=s3://vxdata/group/run/example.zarr/
ARGO_SERVER= argo submit -n vxoptimise --from workflowtemplate/vxoptimise-store-large \
  -p store=s3://vxdata/group/run/large.zarr/   # retry after a full 64Gi scratch disk
```

The template normalises the store URI before labelling it: a missing trailing
slash or a Zarr metadata filename resolves to the same store label. A manual
store run is counted by the dispatcher's dedup and backpressure. Inspect with
`kubectl get workflows,pods -n vxoptimise`. Store pods are removed two minutes after
completion, so read logs from Loki (`{namespace="vxoptimise"}`, 90d) or
OpenSearch (14d) rather than `kubectl logs`. After the self-service rollout,
ordinary `argoworkflows-users` can submit the fixed upload/store templates;
discovery remains administrator-operated.

The enabled hourly `vxoptimise-discovery` cron runs normal Zarr conversions with
the reduced walk budget above and disables the bundle-format census
(`bundle-audit=false`) to reduce RGW reads. Manual/scoped submissions also
default to `bundle-audit=false`; opt in only with the main inventory
or an independently configured audit checkpoint. The `bundles` Workflow output
reports versions and coverage without object paths. Bundle publication is not
enabled by this discovery cron.

The separate `vxoptimise-bundle-upgrades` cron is suspended to reduce RGW traffic.
If re-enabled, it runs hourly at minute 50 and verifies legacy bundles before
creating v3 manifests alongside the originals.
It uses dedicated credentials, 5Gi scratch, a separate checkpoint and mutex.
Catalog references and originals remain unchanged. Exact source/successor
URI/hash mappings remain private on the discovery PVC for an operator's later
cutover; the Workflow exposes only aggregate results. The operator-only
`vxoptimise-bundle-upgrade` template defaults to dry-run; pass `-p dry-run=false`
to publish. See [vxoptimise operations](https://github.com/virdx/infra_k8s/blob/ff0b56e44d5354ab6a403ed2751da3351e8aa751/docs/vxoptimise.md#discovery-and-scheduling)
for report locations, coverage limits and resource bounds.

## Sources and maintenance

Update this guide when the following sources change; other agent instructions
and submission commands should link here instead of repeating these rules:

- [run-image parameters and mounts](https://github.com/virdx/infra_k8s/blob/ff0b56e44d5354ab6a403ed2751da3351e8aa751/current/argo-workflows/templates/run-image.yaml).
- [Identity, queue routing, retries, and concurrency](https://github.com/virdx/infra_k8s/blob/ff0b56e44d5354ab6a403ed2751da3351e8aa751/current/argo-workflows/policies/workflow-identity-injection.yaml)
  and [routing guards](https://github.com/virdx/infra_k8s/blob/ff0b56e44d5354ab6a403ed2751da3351e8aa751/current/argo-workflows/policies/workflow-scheduling.yaml).
- [GPU-class validation](https://github.com/virdx/infra_k8s/blob/ff0b56e44d5354ab6a403ed2751da3351e8aa751/current/kyverno-policies/gpu-class-scheduling.yaml)
  and [Kueue flavors / borrowing limits](https://github.com/virdx/infra_k8s/blob/ff0b56e44d5354ab6a403ed2751da3351e8aa751/current/kueue/resource-flavor.yaml).
- [Resource caps](https://github.com/virdx/infra_k8s/blob/ff0b56e44d5354ab6a403ed2751da3351e8aa751/current/argo-workflows/policies/workflow-validation.yaml),
  [sidecar requests](https://github.com/virdx/infra_k8s/blob/ff0b56e44d5354ab6a403ed2751da3351e8aa751/current/argo-workflows/policies/sidecar-policies.yaml),
  and [controller defaults](https://github.com/virdx/infra_k8s/blob/ff0b56e44d5354ab6a403ed2751da3351e8aa751/current/argo-cd/argo-workflows.yaml).
- `infra_iam`: `data/scheduling/defaults.yaml`, `data/scheduling/teams.yaml`,
  per-user overrides, and generated `k8s/scheduling/` manifests.
- `kuberun`: `src/kuberun/config.py`, `cli.py`, `resources.py`, and `workflow.py`.
  Project selection and image naming live in `project.py` and `resolution.py`.
- [Tekton build pipelines and tasks](https://github.com/virdx/infra_k8s/tree/ff0b56e44d5354ab6a403ed2751da3351e8aa751/current/tekton-builds),
  [image aliases and namespace enforcement](https://github.com/virdx/infra_k8s/blob/ff0b56e44d5354ab6a403ed2751da3351e8aa751/current/tekton-builds/policies/image-name.yaml),
  [Argo buildah](https://github.com/virdx/infra_k8s/blob/ff0b56e44d5354ab6a403ed2751da3351e8aa751/current/argo-workflows/templates/build-image.yaml), and
  [Argo image namespace enforcement](https://github.com/virdx/infra_k8s/blob/ff0b56e44d5354ab6a403ed2751da3351e8aa751/current/argo-workflows/policies/build-image-name.yaml).

Source: https://docs.virdx.dev/vx/cluster/submitting-jobs/index.mdx
