This is the canonical guide for users and agents submitting workloads to this
cluster. It covers kuberun, the shared run-image WorkflowTemplate, and custom
Argo Workflows. Ordinary jobs belong in namespace argo-workflows, using the
submitter’s authenticated Kubernetes identity. Start with the existing project
configuration and the workload’s actual resource and restart requirements.
Choose a scheduling mode
Personal jobs are guaranteed RTX 6000 Ada GPUs; team jobs stay within their RTX 6000 PRO entitlement; preemptible jobs borrow whatever is idle, PRO first. All three modes share the same Kueue cohort.
| Mode | kuberun options | Workflow metadata.labels |
Placement | Borrowing |
|---|---|---|---|---|
| Personal | --no-team --no-preemptible |
Neither mode label needed | Ada (compute-002) only, within the 2-GPU allowance | None; gpu-class: rtx-6000-pro is rejected |
| Team | --team <name> --no-preemptible |
virdx.dev/team: <name> |
PRO nodes (compute-001/003) only; members of the named team (ve2e, histo, diffsim) |
None |
| Preemptible | --no-team --preemptible |
virdx.dev/preemptible: "true" |
PRO nodes first, then Ada, subject to explicit hardware constraints | Idle GPUs, fair-shared between users; evicted whenever a guaranteed Pod needs the node |
Scheduling teams ve2e, histo and diffsim are defined in infra_iam
(quotas 12 / 5 / 4 PRO, 192 / 128 / 128 CPU, 2Ti / 800Gi / 800Gi). A user may
belong to several teams and names the one to draw on per job: --team histo
or team: histo in the project config. A submission naming a team the
submitter is not in (or no such team, including the legacy value "true"
from older clients) is rejected, as are team + preemptible and team +
gpu-class: rtx-6000 (team queues are PRO-only).
Personal defaults are 36 CPU / 264Gi / 2 Ada GPUs across admitted Pods,
including sidecars, all on Ada. RTX 6000 PRO GPUs are team quota: use
--team <name> as a member, or --preemptible (PRO first, evictable) otherwise;
a personal --gpu-class rtx-6000-pro is rejected by kuberun and by the
cluster. Full-GPU Coder workspaces using the personal queue
share this allowance; nvidia.com/gpu.shared slices do not count toward it.
Two of the usual 16 CPU / 128Gi containers fit alongside their Argo wait sidecars.
Authoritative quotas and any overrides live in infra_iam/data/scheduling/
and its rendered k8s/scheduling/ manifests.
Personal GPU quotas deliberately exceed physical capacity. They are admission caps, not reserved cards: an admitted personal Pod can still wait for a GPU behind other personal work. It does not wait behind preemptible work: when no GPU is free, the scheduler evicts preemptible Pods to place it. The same applies on the PRO nodes: a team Pod evicts preemptible Pods holding its cards. Coordinate long personal GPU runs. Admission also cannot guarantee that a Pod fits on an individual node.
Preemptible capacity has no per-user cap, but it is fair-shared: each
submitter has their own user-<name>-preemptible LocalQueue into the shared
pool, and Kueue admits pending preemptible work in order of each user’s
decayed historical GPU usage (48-hour half-life). Alone, you can use every
idle GPU; as soon as someone else queues preemptible work, their steps go
ahead of yours until usage evens out. Check your standing with
kubectl get localqueue -n argo-workflows user-<name>-preemptible -o jsonpath='{.status.fairSharing}'.
Preemptible jobs must tolerate being killed and restarted. The policy supplies template defaults of a 12-hour deadline per Pod attempt and 10 retries; these are not a checkpoint mechanism or a total Workflow runtime guarantee. Applications must persist progress, resume explicitly, and tolerate repeated execution. Do not opt into preemption merely to obtain more resources unless the workload can handle that behavior.
Choose GPU hardware
| Hardware | Workflow virdx.dev/gpu-class (client setting: gpu_class) |
Backend modes |
|---|---|---|
| RTX 6000 Ada, 48 GB VRAM | rtx-6000 |
Personal, preemptible |
| RTX PRO 6000 Blackwell, 96 GB VRAM | rtx-6000-pro |
Personal, preemptible |
| Either type | Omit the label; see client compatibility below for auto |
Personal, preemptible |
For raw Workflows, omit the GPU-class label to allow either type. Use an
explicit type when VRAM, architecture, or software compatibility requires it:
for example, a job needing 64 GB of VRAM requires rtx-6000-pro.
auto is not a valid Kubernetes GPU-class label.
Client compatibility: vxrun 0.2 and later resolve personal and
preemptible gpu_class: auto to “either type” (no label) and accept an
explicit rtx-6000-pro in personal mode; only --team with rtx-6000 is
rejected. vxrun 0.4 and later take a team name (--team histo); older clients
send virdx.dev/team: "true", which the cluster rejects. Older kuberun/vxrun
clients still resolve personal auto to Ada and
reject personal PRO: upgrade, or submit run-image or a custom Workflow
directly as shown below. Use --no-team to clear an inherited team.
For raw Workflows, put any required GPU-class label in metadata.labels;
Kyverno propagates it to Pods and supplies matching selectors. Prefer this over
handwriting nodeSelector or node affinity. A manually supplied GPU product
selector is a hard requirement, not a preference. The shared flavor accounts
for either type without injecting a conflicting product or node class;
kube-scheduler enforces the supplied selector. Do not strip a hardware
requirement to make a job start.
For CPU-only work, use gpu: 0 in kuberun/run-image; custom containers should
omit nvidia.com/gpu resources and the NVIDIA runtime patch. Do not add a
GPU-class label or hardware selector. The queue still determines the pool;
Kueue adds nvidia.com/gpu.present: "true" at admission, even for CPU jobs,
so ordinary workflows use the GPU compute nodes, not CPU-only infra nodes.
Submit a project with kuberun
Use kuberun when building and running code from an application repository. It
builds the image and submits a Workflow referencing run-image.
Example .kuberun.yaml for a personal one-GPU job:
cpu: "16"
memory: "128Gi"
gpu: 1
gpu_class: auto
team: "" # or a team name such as histo
preemptible: false
request_ratio: 1.0Use the project’s Dockerfile command, or specify entrypoint / cmd in the
config or on the command line. Inspect the effective settings with
kuberun config. CLI flags override project .kuberun.yaml, which overrides
global ~/.kuberun/config.yaml (with its KUBERUN_* environment overrides).
Review the working tree and kuberun’s autocommit setting before running:
building may commit and push local changes.
# Personal quota, either GPU type (older clients select Ada here)
vxrun run --no-team --no-preemptible --gpu 1 --gpu-class auto
# Borrow either type, with interruption/restart accepted
vxrun run --no-team --preemptible --gpu 1 --gpu-class auto
# Borrow PRO specifically, for a workload requiring that hardware
vxrun run --no-team --preemptible --gpu 1 --gpu-class rtx-6000-proThe explicit negative flags clear inherited scheduling modes. Likewise,
--gpu-class auto clears an inherited GPU preference (older clients still
select Ada in personal mode). For ordinary CPU jobs, change the GPU count to
zero and choose the appropriate mode.
cpu and memory are container limits. Kuberun’s request_ratio defaults to 0.5;
values from 0.1 to 1.0 reduce CPU/memory requests relative to those limits, with
rounding and a minimum of 1 CPU / 1Gi. For example, 16 CPU / 128Gi at 0.5
requests 8 CPU / 64Gi. GPU requests are unchanged. Lower requests consume less
admission quota but permit more contention; choose them from expected usage,
not simply to fit more jobs. Ratio 1.0 is a predictable starting point.
Direct run-image submissions keep requests equal to limits unless
cpu_request / memory_request are explicitly supplied.
Multiple projects in one repository
With the monorepo-aware kuberun release, kuberun -C apps/worker run selects the
nearest .kuberun.yaml up to the Git root. kuberun --config PATH run selects an
explicit config relative to -C (or the caller’s directory). These global
options precede the command. Parent configs are not merged; an invalid selected
config fails without falling back. YAML paths are relative to their config;
CLI Dockerfile/context paths remain repo-relative. Paths must stay within the
repo, including symlink targets, and the Dockerfile must be within the build
context. For example, apps/worker/.kuberun.yaml can contain:
project: worker
context_dir: ../..
dockerfile: Dockerfile
autocommit: trueThis config uses the whole repo as build context. Without an explicit context, the default is also the repo root unless global config overrides it. A nested config discovers its own Dockerfile when none is specified. The config directory, Docker build context, and container WORKDIR are separate concepts.
The source repository is still mono. Project identity controls job names,
labels, and /mnt/artifacts/<user>/<image>/<workflow> output paths (vxrun passes
the project identity as the image parameter). The image
repository is derived from the source repo and full config-directory path:
mono/apps/worker here, distinct from mono/tools/worker or a worker repo.
There is no image_name config override; changing the project label or build
context does not change the image name. Configs in the same directory share an
image repository; moving the config changes it. Invalid OCI paths are rejected.
The image name also participates in the cache key.
Kuberun passes this derived name through the backend’s image-name (Tekton) or
image_name (Argo buildah) parameter. Admission requires the source repo name
or a path beneath it, rejecting a submission from mono to foo or other/foo.
Tekton additionally allows centrally approved legacy aliases only for their
exact repo/context/Dockerfile mapping in
the image-name policy. For
example, mono + . + apps/vxdata-jobs/Dockerfile may publish vxdata-jobs.
The policy supplies the alias for buildkit-image when the parameter is
omitted. Root-config kuberun builds omit it for compatibility; the nested
apps/vxdata-jobs/.kuberun.yaml config publishes mono/apps/vxdata-jobs instead.
Argo buildah also checks the namespace before building, including templateRef
calls. These checks govern the supplied build pipelines/templates; registry
credential authorization remains a separate boundary.
Autocommit commits only the selected config directory and its descendants,
including additions/deletions and hook fixes. Staged changes elsewhere remain
staged. Uncommitted shared packages outside that scope are excluded; builds use
their committed versions. Commit those changes separately when required. A
selected Dockerfile outside the commit scope must be clean, so kuberun’s parsed
command matches the version used remotely. A symlinked config keeps its selected
directory as the commit scope, rather than the target file’s directory.
Without a config, scope is the repository root. Use --no-autocommit to require
manual commits; the built-in default is autocommit: true, which project/global
settings may override. Pushing still publishes existing commits on the branch.
Jobs carry separate repo and project labels. Use kuberun status --repo mono
or kuberun status --repo mono --project worker to filter them. Older jobs
without a project label remain visible under the repo filter.
Use kuberun config for setting sources, run --dry-run / build --dry-run for
a local plan, and kuberun init to create files explicitly. Config inspection
and dry runs do not commit, push, authenticate to the cluster, or generate files.
Deploy the companion templates/tasks/policy before releasing the CLI. Tekton
cache lookup, build, and archive keys now include a hash of build arguments;
empty arguments retain legacy keys. Build and cache-hit tasks export the actual
image repository/digest. Argo buildah exports the pushed image-ref, and
run-image accepts image_ref for immutable execution while retaining image
for the output directory and backwards-compatible tagged submissions.
Submit an existing image with run-image
Use the installed run-image WorkflowTemplate when the job is one container
with standard credentials and storage. Image builds normally use kuberun’s
Tekton backend; build manifests are under
current/tekton-builds/examples.
For Pixi Dockerfiles, see Dockerfile advice for the shared
cache mount and layer ordering, and BuildKit package caches
for retention behaviour.
Save this as run-image-job.yaml. Replace the example image name, tag, and
command with an image already present in Zot and its actual command.
Optional image_ref (repo@sha256:...) pins an exact digest and overrides
image/image_tag; kuberun sets it for every build:
apiVersion: argoproj.io/v1alpha1
kind: Workflow
metadata:
generateName: run-example-
namespace: argo-workflows
spec:
workflowTemplateRef:
name: run-image
arguments:
parameters:
- name: image
value: example-job
- name: image_tag
value: replace-with-built-tag
- name: cmd
value: "python -u /app/job.py"
- name: cpu
value: "16"
- name: memory
value: "128Gi"
- name: gpu
value: "1"This example uses personal quota and can run on either GPU type. To require
PRO, add metadata.labels: {virdx.dev/gpu-class: rtx-6000-pro}. For Ada, use
rtx-6000. To borrow instead, add virdx.dev/preemptible: "true". The same
labels apply to custom Workflows below. Kubernetes label values must be
strings, so quote "true".
kubectl create --dry-run=server -f run-image-job.yaml -o yaml
kubectl create -f run-image-job.yamlThe image name is relative to zot.fra.virdx.dev:5000, not a full registry URL.
Prefer a specific built tag over latest. Set at least one of cmd or
entrypoint for manual submissions: the template uses an Argo script wrapper
and does not discover the image’s Dockerfile defaults. Kuberun resolves those
defaults before submitting. cmd alone is executed through sh -c;
entrypoint plus cmd uses shell word splitting, so put complex launch logic
in an executable script in the image.
The template defaults to 16 CPU / 128Gi / 0 GPUs, with requests equal to limits.
Optional cpu_request and memory_request parameters override requests;
they must be positive and no greater than the corresponding limits. Supported
limit values are:
| Parameter | Accepted values |
|---|---|
cpu |
1, 2, 4, 8, 16, 32, 64, 128 |
memory |
2Gi, 4Gi, 8Gi, 16Gi, 32Gi, 64Gi, 128Gi, 192Gi, 256Gi, 512Gi |
gpu |
Integers 0–8; quota and per-node capacity still apply |
local_cache |
0 (disabled), 16Gi, 32Gi, 64Gi, 128Gi, 256Gi, 512Gi, 1024Gi |
run_environment selects production (default) or staging credentials.
The template provides ClearML/S3 configuration, environment-specific credentials,
registry authentication, the internal CA, /dev/shm, read-only /mnt/storage,
and /output backed by /mnt/artifacts/<username>/<image>/<workflow-name>.
S3 credentials are exposed through the standard shared credentials file
(AWS_SHARED_CREDENTIALS_FILE), with the same profiles as Coder workspaces:
[scratch] (read/write scratch bucket), [workflow-output] (read-only
published batch results) and [default] (the same read-only identity as
[workflow-output]); every profile carries the internal RGW endpoint_url,
and none is real AWS. See the
template for exact mounts and
environment variables; never copy secret values into job manifests.
local_cache provisions node-local scratch at /cache and exports
VXRUN_CACHE_DIR=/cache and VXS3_CACHE_DIR=/cache/.vxs3 (vxs3’s default
download/unpack root). Directory variables follow VXRUN_<NAME>_DIR
(VXRUN_OUTPUT_DIR is always set); KUBERUN_OUTPUT and KUBERUN_CACHE are
legacy aliases with the same values, kept until no worker code reads them.
Treat /cache, emptyDir,
and the container filesystem as disposable across retries and deletion. Store
checkpoints in /output, shared artifacts, object storage, or an appropriate
persistent volume, and explicitly reload them after a restart. A newly submitted
Workflow has a different /output directory, so resuming a previous submission
requires its checkpoint location.
For temporary development checkpoints, prefer the scratch S3 bucket over
/output: the NFS export behind /mnt/artifacts is a poor fit for frequent
large rewrites. See Scratch storage for development
for retention, access and publishing data you want to keep.
The Argo UI can submit the same template with these parameters and Workflow
labels. For manifests, use kubectl create with an authenticated user
kubeconfig; there is no need to supply creator or username labels yourself.
Kyverno derives identity from the API request; through Argo Server, it uses the
creator identity set by the server.
Scratch storage for development
The scratch S3 bucket facilitates development: use it for experiments,
intermediate data and temporary checkpoints while working out what you need.
It has a two-week (14-day) garbage-collection policy. Treat its contents as
disposable; scratch is not for permanent storage or inputs and outputs that
must remain available for repeatable workflows.
Once you are happy with the data, model or result, publish it to vxdata and
create a permanent resource. Use that resource in subsequent workflows.
The retained bytes must live in vxdata storage; a resource that only points
at s3://scratch/... will still lose its data to scratch GC. See
Vxoptimise service workflows for publishing
an immutable S3 prefix with vxrun storage upload and --retention permanent.
Finish publication before the scratch objects expire.
Access from Coder and jobs
Coder workspaces, run-image and run-batch jobs expose
the [scratch] profile through AWS_SHARED_CREDENTIALS_FILE. It carries
endpoint_url = https://s3.fra.virdx.dev. Use
boto3.Session(profile_name="scratch"), s3fs.S3FileSystem(profile="scratch")
or AWS_PROFILE=scratch aws s3 .... For example, this read-only boto3 check
uses the workspace or job’s internal CA bundle:
import os
import boto3
s3 = boto3.Session(profile_name="scratch").client(
"s3",
endpoint_url="https://s3.fra.virdx.dev",
verify=os.environ["SSL_CERT_FILE"],
)
result = s3.list_objects_v2(Bucket="scratch", MaxKeys=5)
print("S3 OK — objects returned:", result["KeyCount"])The Coder template keeps your persistent local home PVC, /mnt/storage
read-only and /mnt/artifacts read/write. Startup does not create directories
or change ownership on NFS, so it requires no writes to a full or read-only
export. Application writes to /mnt/artifacts can still fail when the export
is full or read-only. Write development data to your home and upload results to S3.
Existing workspace Pods pick up this change after the updated
kubernetes-dev template is published and the workspace is rebuilt.
Write under s3://scratch/<username>/<experiment-or-workflow>/. The scratch
identity is shared, so this prefix is an organizational convention, not an
access boundary. The bucket also has a shared size quota; delete temporary
data you no longer need. Retrying or resuming an experiment does not exempt
its objects from the 14-day GC policy.
Keep scratch request load modest
Scratch shares Ceph with other workloads. Before uploading a directory, inspect the local source with the advisory analyzer:
vxrun data inspect ./data --scratchThis uses vxs3 to report object counts, sizes, the share below 4 MiB and
format-specific layout findings. It suggests changes without uploading data or
blocking submission. vxrun review also flags scratch references for review;
a reference alone does not mean the workload is inefficient.
- Consolidate related small records into fewer native files, such as Parquet,
when the readers support it. For independent ordinary files, use
vxs3tar bundles targeting 1 GiB shards and keep the manifest with the shards. - Keep Zarr native: review chunking and sharding instead of tarring the store. A few large files or occasional checkpoints may need no layout change.
- Stage inputs once per job onto local NVMe (
--local-cache, mounted at/cache) and reuse them across epochs and workers. Batch small writes, avoid repeated S3 listings, and keep transfer concurrency modest across workers.
Prefer inspection before upload. For remote inspection, select only your
experiment’s prefix; do not repeatedly scan scratch during training. Scratch
mode refuses whole-bucket scans and defaults to 10,000 objects, 64 metadata
probes, 4 MiB of metadata and a 30-second cooperative time budget. LIST/HEAD
requests are outside the metadata-probe budget, so these are not hard limits
on total S3 requests or elapsed time. Remote analysis uses the analyzer’s
environment-based S3 credentials and endpoint; selecting the boto3 [scratch]
profile alone does not configure it.
Check coverage: a partial or failed scan is not a clean bill of health. Layout
analysis does not measure IOPS; also review how often the job issues
GET/LIST/PUT requests and whether each worker repeats the same operations.
See vxcluster’s docs://storage-practices for packing and inspection examples.
Submit an image batch with run-batch
Use run-batch for a finite set of independent commands sharing one image and
resource allocation. It runs one DAG task per JSON item with bounded parallelism.
One failed item does not stop the remaining items; the overall Workflow reports
failure when an item exhausts its attempts. Outputs are written to temporary
node-local storage and, by default, published to the workflow-output bucket as
one vxs3 bundle per batch (see publication).
The matching CLI exposes vxrun batch init --items items.json, then
plan, render, build, submit and status, alongside vxrun workflow.
Deploy run-batch.yaml through
ArgoCD before releasing/using that CLI feature. The namespaces, rbac, and
argo-workflows-config applications must all reconcile the batch changes:
the output-volume steps require the batch-output-volumes ServiceAccount,
Role, and RoleBinding in argo-workflows. Verify a small batch reaches a
terminal phase with a persisted node tree; a finaliser log alone does not prove
the controller could save workflow status. An items.json file contains:
[
{"id": "trial-0", "argv": ["python", "train.py", "lr=0.01", "seed=1"]},
{"id": "trial-1", "argv": ["python", "train.py", "lr=0.001", "seed=2"]}
]Each item contains id and argv, optionally inputs (below; older clients
that omit it keep working). IDs must be unique, 1–100 letters,
digits, dots, underscores or hyphens; . and .. are invalid. argv is the full
command array, including the executable; whitespace and shell metacharacters stay
literal. For Pixi images, start it with ["pixi", "run", ...]. The image needs
sh, mkdir and cat; it does not need vxrun/vxs3. The CLI rejects NUL and Argo
template delimiters in arguments and caps inputs at 1,000 items / 512 KB per
Workflow. Apply these same limits when submitting manually.
Direct native submissions use the same template:
apiVersion: argoproj.io/v1alpha1
kind: Workflow
metadata:
generateName: training-batch-
namespace: argo-workflows
spec:
workflowTemplateRef:
name: run-batch
parallelism: 2
arguments:
parameters:
- name: vxrun-image
value: zot.fra.virdx.dev:5000/example-job@sha256:REPLACE_WITH_DIGEST
- name: items
value: '[{"id":"trial-0","argv":["python","train.py","seed=1"]},{"id":"trial-1","argv":["python","train.py","seed=2"]}]'
- name: cpu
value: "8"
- name: memory
value: "32Gi"
- name: gpu
value: "1"Set a real immutable image digest before submitting. Standard personal/team/
preemptible scheduling labels apply to the whole Workflow, as for run-image.
Default per-item resources are 2 CPU / 4Gi / 0 GPUs, with equal requests and
limits. Resource enums match run-image; spec.parallelism is 1–25.
run_environment chooses production/staging credentials. retries defaults to
2 (0–10); retry_policy defaults to OnError and also supports OnFailure or
Always. task_deadline defaults to 43200 seconds (CLI range 1–604800).
Application failures do not retry under the default; checkpoint/resume logic
remains the application’s responsibility.
Each item has a temporary TopoLVM PVC. Each attempt writes to a separate
directory on that claim, mounted at /output (VXRUN_OUTPUT_DIR). The workflow
UID, item ID and Pod name isolate reused workflow names, items and retries.
VXRUN_ITEM_KEY, VXRUN_ATTEMPT, VXRUN_WORKFLOW and VXRUN_WORKFLOW_UID identify
the execution. Each worker node exports item-id and output-path parameters;
inspect its status to distinguish successful outputs from failed attempts.
output-path is an internal path on the item’s claim, not a shared filesystem
path accessible from Coder. Use vxrun batch results for published results.
The same claim carries one /checkpoints directory per item
(VXRUN_CHECKPOINTS_DIR), shared by every attempt of that item: write resumable
state there and, on start, resume from whatever a previous attempt left. It is
never packed into the published bundle and is deleted with the claim, so it
covers retries of an item, not a resubmitted batch or a lost node (retries are
pinned to the claim’s node, see below). Checkpoints that must survive either go
to the scratch bucket, keyed by run ID. The claim always exists (16Gi when
local_cache is 0, local_cache otherwise) and is shared by inputs, every
attempt’s output and checkpoints. The claim layout is:
<user>/batches/<workflow-name>-<workflow-uid>/items/<item-id>/
attempts/<pod-name>/ /output of one attempt; the successful one is packed
inputs/ declared inputs, read-only
checkpoints/ /checkpoints, shared by the item's attemptsMounts include the standard credentials/CA, /output, /dev/shm and /cache.
There are no NFS or hostPath mounts: read inputs from S3/vxdata into /cache.
local_cache: "0" uses a Pod-local emptyDir at /cache and a 16Gi output PVC
per item. A nonzero local_cache sets both the per-Pod scratch capacity and
the per-item output PVC capacity. These are separate allocations: allow space
for cached inputs, output across all attempts, and packing shards.
VXS3_CACHE_DIR=/cache/.vxs3 in both cases.
The first worker binds its item’s claim to a node. Retries and its publisher
must use that node; different items can run on different nodes. Local storage
cannot fail over to another node during a node outage. The claim survives worker
Pod garbage collection while publication is pending. An item exit hook requests
deletion on success, failure or Stop; Kubernetes releases the volume once its
worker/publisher Pods are removed (normally after the one-hour log retention).
Terminate skips hooks; Workflow deletion/TTL is the ownership cleanup backstop.
S3 is the durable output. With publish: "false", /output is temporary too:
the application must upload anything it needs to retain. run-image keeps its
existing mounts and storage behaviour.
Declared inputs
A batch can declare the data it reads instead of fetching it from inside the
command. The inputs parameter (shared by every item) and each item’s inputs
list vxs3 bundle selections:
[{"name": "weights", "manifest": "s3://vxdata/models/grader/v3/manifest.parquet",
"sha256": "<manifest digest>", "paths": ["fold2/checkpoint.pt", "config.yaml"]},
{"name": "case", "manifest": "s3://vxdata/cohorts/bpmri/2026-10/manifest.parquet",
"sha256": "<manifest digest>", "prefix": "cases/0017/"}]manifest + sha256 pin the bundle (a vxs3 BundleRef); one of prefix
(a directory), paths (exact members) or pattern (globs) selects from it, and
no selector takes the whole bundle. Names are unique across shared and item
inputs and become /inputs/<name> in the worker, read-only (VXRUN_INPUTS_DIR).
The worker’s fetch-inputs init container (vxrun-tools image) extracts them
onto the item’s volume before the command
starts, verified against the pinned digest; a retried attempt reuses what is
already there and a changed pin for the same name fails. Items without
declared inputs skip the fetch. s3://vxdata bundles are read through
vxdata API presigned URLs with the run environment’s VXDATA_TOKEN; other
buckets use the read-only [default] profile. Count inputs in local_cache: they share the
item volume with outputs. env is a JSON list of {name, value} merged into
the worker environment; cluster-owned names (VXRUN_*, AWS_*, VXS3_*,
CLEARML_*, …) are refused by the CLI.
vxrun spec (mono) authors all of this as one YAML file: image, resources,
named bundles, inputs, items and outputs. vxrun spec resolve pins catalog
resources and sizes every selection, so the file states the minimum
local_cache; vxrun spec compile produces the run-batch Workflow above.
The command keeps cluster network access in this version; a network-isolated
mode is declared in the spec format but not enforced yet.
Batch output publication
With publish: "true" (the default) every item is a two-step sequence: the
command runs as above, then a pack step in tools-image (the vxrun CLI with
vxs3) packs the successful attempt’s /output into a vxs3 bundle and publishes
it. An exit handler then completes the batch:
s3://workflow-output/<username>/<workflow-name>-<workflow-uid>/
workflow.json name, uid, user, status, expected item ids (written first)
items.parquet one row per expected item: status, manifest reference, files, bytes
manifest.parquet written last, only when every item succeeded: the batch is complete
items/<id>/manifest.parquet written last by that item's pack step: the item is complete
items/<id>/shard-*.tarShards never move. The root manifest references each item’s shards where its
pack step published them (a vxs3 >= 3.1 composed bundle), so one open_bundle
on the root sees items/<id>/... for every item, and each item’s own manifest
stays a complete bundle. The root exists only for a fully successful batch, like
a multipart upload that becomes an object only on completion: a batch with a
failed item keeps workflow.json, items.parquet and the successful items’
bundles for inspection and re-runs, but gets no root manifest and no catalog
entry. register: "true" registers a complete root as a vxdata Bundle with
items.parquet as its records table, using the run_environment vxdata
credentials. (Finalisers before vxrun 0.4.7 wrote a root manifest for partial
batches too; items.parquet tells them apart.)
Collect with the CLI, from a Coder workspace or a job; both carry the read-only
[workflow-output] profile:
vxrun batch results <workflow-name> # prefix, completion, one line per item
vxrun batch results <workflow-name> --download ./out # complete batch -> ./out/items/<id>/...
vxrun batch results <workflow-name> --download ./out --item trial-0 # one item, also when incomplete
vxrun batch results s3://workflow-output/<user>/<workflow-name>-<uid> # by prefixWorkflows are garbage-collected three days after they finish; the data is not.
results reads the live Workflow when it exists, otherwise finds
<user>/<name>-<uid>/ in the bucket and takes name, uid and expected items from
workflow.json, so results stay readable after the Workflow is gone.
The pack step mounts the item’s output PVC read-only and assembles all of
that item’s shards on disk in its own /cache before uploading. It does not
buffer whole shards in RAM. /cache is a separate TopoLVM scratch volume when
local_cache is nonzero, otherwise a disk-backed emptyDir. With publishing on,
local_cache should cover an item’s outputs and packing space. A pack retry after a partial
upload fails, because the exact destination must be empty and the writer identity
cannot delete; the item is then reported as failed. Terminate skips exit
handlers, so a terminated Workflow has neither items.parquet nor a root
manifest; prefer Stop, which records the items that did finish.
Published outputs are immutable from a user’s point of view. The writer identity
(RGW user workflow-writer: put, get, list, no delete) lives in the
workflow-output-writer secret, which only the pack and finalise pods mount;
user containers get the bucket read-only through the [workflow-output] profile
of aws-credentials. The Kyverno policy argo-restrict-workflow-output-writer
rejects any submission that references the writer secret, any run-batch
submission that adds spec.templates or spec.podSpecPatch, any pod mounting
the secret that is not a controller-created vxrun-tools pod, and in-cluster
edits of ArgoCD-managed WorkflowTemplates. Only the fixed PVC allocation and
cleanup steps use the batch-output-volumes service account; user workers
have no PVC lifecycle permissions. publish: "false" skips publishing and
finalisation, but still deletes the temporary output claim after each item.
The pack and finalise steps run the vxrun-tools image: the released vxrun
CLI with vxs3 and the vxdata SDK from the conda channel, built from
builds/vxrun-tools with the Tekton example
pipelinerun-vxrun-tools.yaml. It is pinned by digest inside the template and
is deliberately not a parameter, so no user-chosen image ever runs with the
writer identity. Each item node exports manifest (JSON {manifest_uri, sha256, files, bytes} from pack, empty when publishing is off) alongside
output-path; the inner worker node exports item-id and output-path.
Write a custom Workflow
Use a custom Workflow for a DAG, multiple steps, or container/volume settings
that run-image does not expose. This is a complete one-GPU personal example;
replace the image and command before submitting as custom-job.yaml:
apiVersion: argoproj.io/v1alpha1
kind: Workflow
metadata:
generateName: custom-example-
namespace: argo-workflows
spec:
entrypoint: main
parallelism: 1
imagePullSecrets:
- name: zot-registry-credentials
podSpecPatch: '{"runtimeClassName":"nvidia"}'
templates:
- name: main
container:
image: zot.fra.virdx.dev:5000/example-job:replace-with-built-tag
imagePullPolicy: Always
command: [python, -u, /app/job.py]
resources:
requests:
cpu: "16"
memory: 128Gi
nvidia.com/gpu: "1"
limits:
cpu: "16"
memory: 128Gi
nvidia.com/gpu: "1"CPU/memory requests and limits must be explicit for each application container.
GPU requests and limits must be equal whole numbers. GPU Pods require
runtimeClassName: nvidia, set through podSpecPatch here; apply it only to
GPU templates when mixing CPU and GPU steps. Standard Argo sidecar requests are
injected by policy. Custom Workflows must explicitly configure any storage,
credentials, or CA mounts they need; they do not inherit run-image’s mounts.
Use imagePullPolicy: Always for internal registry images because tags can
change; the example above sets it explicitly.
Before making this example preemptible, add persistent checkpoint/output
storage and application restart behavior.
For human-submitted Workflows, permitted hostPath roots are /mnt/storage,
/mnt/scratch, and /mnt/artifacts; /mnt/storage must be mounted read-only.
Prefer the existing run-image mounts when they meet the workload’s needs.
Do not set Kueue queue labels, Kueue WorkloadPriorityClass labels,
priorityClassName, preemptionPolicy, or identity labels yourself, including
inside template metadata or podSpecPatch. Kyverno assigns queue, priority,
identity, the Workflow service account, and the per-user concurrency semaphore.
Final Pods must match their owning Workflow’s routing. Mode and queue are
immutable after submission; create a new Workflow to change them.
Reusable WorkflowTemplates belong in
current/argo-workflows/templates/<name>.yaml and are deployed through ArgoCD
after merge. Ordinary users cannot create them directly. One-off Workflow
objects can be submitted with kubectl create -f custom-job.yaml.
D3N routing inventory
All newly created Argo workflow Pods in argo-workflows, autoresearch, and
vxoptimise receive /etc/virdx/d3n/d3n_instances.json through the
inject-d3n-config policy.
This covers run-image, custom Workflows, referenced WorkflowTemplates and
multi-container steps, including init containers. No template-specific volume
or mount is needed. Unrelated Pods and already-running Pods are unchanged.
Coder workspace Pods receive the same mount in coder-workspaces, selected
by app.kubernetes.io/name: coder-workspace (set by the Coder Terraform
template). No workspace image rebuild or template-version update is needed.
Existing workspaces pick it up when their Pods are recreated, for example on
the next stop/start. The policy does not restart running workspaces. Both
regular and init containers receive the mount.
The workspace template also exports VXS3_D3N_INSTANCES pointing at that
file, so vxs3 discovery treats the deployed inventory as authoritative and
fails if the ConfigMap is absent instead of falling back to gateway routing.
The file contains stable Ketama node names and HTTPS endpoints. Ansible owns
the namespace-local d3n-instances ConfigMaps and generates identical content
from the RGW inventory. Do not add another copy to an ArgoCD application.
Reserve volume name d3n-instances and mount path /etc/virdx/d3n for this
shared configuration; put application-specific files elsewhere.
The mount is read-only and optional, so jobs that do not use D3N can start before the ConfigMap is provisioned. A D3N client must explicitly read the file and fail if it is missing or invalid, without silently using a bundled list:
from vxs3.transport.instances import load_instances
from vxs3.transport.routing import select_instance
instances = load_instances("/etc/virdx/d3n/d3n_instances.json")
instance = select_instance("example", "shard.tar", instances=instances)Python and JavaScript clients consume the same JSON; the hashing contract and
compatibility vectors live in mono/packages/vxs3/ROUTING.md. Load one snapshot
per client or operation. Projected files can update later, but an already-loaded
vxs3 tuple stays fixed. Coordinate client reloads when changing ring membership.
Making this file available does not enable Ketama routing in existing download
code; the application must use the routing functions.
For rollout, include argo-workflows, autoresearch, vxoptimise, and
coder-workspaces in
infra_ansible/inventories/group_vars/k3s_cluster/k3s_d3n.yaml, then run
pixi run playbook k3s_d3n there. The mutation policy is deployed through the
kyverno-policies ArgoCD application after merge, together with the narrow
Coder mount-path exception in restrict-volume-mounts. New workflow namespaces
must be added to both the Ansible list and the policy’s namespace match.
Size jobs for both quota and hardware
- Kueue accounts for Pod requests, including sidecars and the effective init request, across all admitted Pods in the queue. The standard Argo wait container adds 100m CPU / 128Mi per Pod. Two 16 CPU / 128Gi applications fit the default personal quota; a third does not, even without a GPU.
- GPUs cannot be split across nodes for one Pod. Current full-GPU capacity is
5 Ada on compute-002 and 8 PRO on each of compute-001/003. Three other Adas
are time-sliced for Coder and are not full
nvidia.com/gpucapacity. An eight-GPU Pod exceeds personal quota even though a PRO node can fit it. One Pod cannot mix Ada and PRO because the types are on different nodes. Distributed training needs separate Pods and application support. - Custom application containers are capped at 192 CPU / 512Gi / 8 GPUs by policy; run-image has the narrower CPU enum above. These ceilings are not entitlements or guarantees that hardware is available.
- Workflow
spec.parallelismdefaults to 20 and cannot exceed 25. This limits simultaneously active Pods, not the number of steps. Select it to suit the queue’s aggregate quota. Personal GPU work can admit up to two one-GPU Pods or one two-GPU Pod, subject to CPU/memory quota and physical capacity. - The per-identity semaphore defaults to 10 concurrent Workflows across modes; excess submissions wait. The controller also caps namespace concurrency at 200 Workflows. Raising parallelism does not increase entitlement.
Validate and inspect a submission
Check the Kubernetes context and effective mode/resources first. For a custom
Workflow, run argo lint --offline --strict custom-job.yaml. Then use
kubectl create --dry-run=server -f <file> -o yaml to inspect Workflow admission
and injected routing without creating the job. A server dry run does not create
the eventual Pods or prove that resource requests are schedulable; those checks
also require the quota and hardware review above.
For run-image, inspect the deployed template with
kubectl get workflowtemplate run-image -n argo-workflows -o yaml if it may lag
the repository. Offline lint of a template reference needs that template
available locally and does not emulate Kyverno identity injection.
After an authorized submission, inspect both Workflow and Pod state:
argo get -n argo-workflows WORKFLOW_NAME
kubectl get pods -n argo-workflows -l workflows.argoproj.io/workflow=WORKFLOW_NAME
kubectl describe pod -n argo-workflows POD_NAME
kubectl logs -n argo-workflows POD_NAME -c main -fA Workflow waiting on its semaphore has not yet started its Pods. A Pod with a
Kueue scheduling gate is waiting for admission; an admitted but unschedulable
Pod may be blocked on node capacity, storage, or a hard selector. Inspect events
before changing the configuration. Operators can use util/kueue-status and
util/gpu-status; users may need an operator to inspect cluster-scoped queues.
Do not relabel an existing job or remove a required hardware constraint to
work around Pending state.
Completed Pods, including failed attempts, are normally removed after one hour. Use Workflow status and Loki for older failures; container files and scratch are not durable output. Workflows normally expire two days after success or seven days after other completion.
To cancel a job, use argo stop (runs exit handlers), argo terminate (does
not), or the matching Argo UI buttons; argo delete also works but leaves no
record. Those are the only changes a submitter may make to a Workflow after
admission: everything else in spec, status and the labels is locked by
argo-restrict-user-workflow-updates, so argo retry, argo resume and
argo suspend need an operator. Resubmit to change a job.
Vxoptimise service workflows
Self-service uploads and single-store optimisation use vxrun storage after the
backend rollout described in the deployment notes.
These commands submit fixed templates directly to Kubernetes with your Coder
OIDC login; they do not build user code or upload bytes from the client.
vxrun storage upload /mnt/storage/example/dataset --name example \
--project example-project --description "Synthetic reference dataset" \
--data-role reference --retention review-180d --wait
vxrun storage upload /mnt/artifacts/example/dataset --metadata metadata.json --dry-run
vxrun storage upload s3://example-source/dataset/ --metadata metadata.json --wait
vxrun storage optimise s3://vxdata/example/store.zarr/ --wait
vxrun storage status
vxrun storage status --project example-project
vxrun storage status JOB_NAME --json
vxrun logs JOB_NAME --namespace vxoptimise --followUploads require a name, project slug, description, data role and retention
intention, supplied as flags or metadata JSON. They accept shared paths beneath
/mnt/storage and /mnt/artifacts, or s3://bucket/prefix/ on the cluster’s
object store after the S3-capable worker rollout; HOME paths cannot be used directly.
The two shared paths are existing read-only NFS mounts, with files readable by UID 1000. The
upload worker runs as UID/GID 1000 and writes only to its private scratch/output
volumes and vxdata; submitter attribution uses the authenticated workflow identity.
S3 prefixes are downloaded into private scratch using a fixed read-only source
identity, then follow the same Zarr/bundling rules. Size/count limits are checked
before download; conditional reads and a second listing reject source changes
before publication. Prefixes must be below a bucket root; they do not select
individual objects. A trailing slash is added when absent. Eligible source buckets
are controlled by S3 IAM grants. Endpoint/credential overrides are not exposed.
The CLI requires vxrun.virdx.dev/storage-s3-contract: "1" on the deployed
template. The repository pins the S3-capable worker with this capability enabled;
both take effect after GitOps sync. Reverting to an NFS-only image must disable
the capability in the same change.
Inputs must remain immutable. The worker accepts valid root Zarr stores,
standalone regular files of at least 100 MiB, and ordinary directories.
Directories are always bundled into vxs3 tar shards targeting 1 GiB: small files
within a directory cannot become loose S3 objects. Small standalone files are
rejected before copying or allocation; group related files into a directory.
Nested/malformed Zarr, links and special files are rejected. Inputs are limited
to 256 GiB/100,000 entries; bundle members must fit a 4 GiB shard, while standalone
files stay native and are exempt from that shard limit. Uploads allocate a vxdata storage
prefix and register a verified manifest with submitter/workflow provenance.
Retention metadata does not trigger automatic deletion. Existing vxdata stores
are mirrored into vxdata-optimised without creating another catalog identity.
Workflow labels include app.kubernetes.io/name=vxoptimise,
vxoptimise.virdx.dev/operation=upload|optimise,
kuberun.virdx.com/task=storage-upload|storage-optimise, project on uploads,
and the admitted virdx.dev/username. They can also be selected with kubectl -l.
Argo reads the worker’s ARGO_PROGRESS_FILE (N/M); vxrun status and --wait
show these work units, and logs contain stage updates. vxrun storage status
defaults to your upload and optimise jobs in the storage namespace and shows the
verified data coordinate on success, plus the vxdata resource identifier and
manifest URI for uploads. It supports a job name, --project, --operation,
--all, and --json for full result fields. Upload totals expand after
source listing and optimisation determine the object counts; S3 download logs
include file and byte progress. Mirror progress is coarse. Refresh can
lag by about a minute. The Argo web server currently cannot view this namespace;
use vxrun or ARGO_SERVER= argo get -n vxoptimise JOB_NAME.
Storage status reads retained Workflow objects, so results survive the five-minute
pod/scratch cleanup. Successful upload workflows expire after two days; successful
optimise workflows after 20 minutes; failed/errored workflows after seven days.
Once the Workflow is deleted, its status entry disappears. Workflow GC does not
delete published vxdata objects or catalog records. The CLI prints the stable
vxupload-<workflow-uid> identifier on submission; save it for later SDK lookup
with the Coder environment’s VXDATA_TOKEN. Read file_metadata.data_uri only
when file_metadata.state is ready. The record may not exist for pending or
rejected jobs. Automatic catalog history is deferred; v1 status uses Kubernetes.
--dry-run prints the request offline and never checks credentials or remote
source existence. --wait --timeout SECONDS stops waiting without cancelling the
job. Pending/failed uploads require operator reconciliation before resubmission;
the vxdata record preserves the allocated prefix after Argo garbage collection.
S3 discovery and conversion run in the dedicated vxoptimise namespace with
separate read-only dispatcher and read/write worker S3 credentials. See the deployment notes.
Discovery remains an administrator-operated CPU service. Self-service and
discovery share the namespace quota/semaphore; neither uses the shared
run-image template or personal/team queues.
After GitOps sync and Secret creation, submit the namespaced template directly
through Kubernetes (the Argo Server remains scoped to argo-workflows):
ARGO_SERVER= argo submit -n vxoptimise --from workflowtemplate/vxoptimise-discovery \
-p dry-run=true -p max-stores=10Omit dry-run=true to submit independent conversion workflows, initially with
-p max-stores=1. Dry and real runs share the default state database and each
observed Zarr root is next due 24 hours later, so a real run within a day of a
dry run reports idle; wait the interval, or give probes their own state-db.
The template defaults to the released image and real submission.
Upload, store and discovery templates pin the same worker release; discovery
passes its image parameter to child store workflows.
max-stores=0 removes the submission-count limit. The hourly cron permits
5,000 scan steps or five minutes, whichever comes first, and resumes from its
persistent SQLite checkpoints on the next run. Manual template submissions
default to 10,000 steps and 45 minutes; pass -p max-scan-steps=5000 and
-p max-scan-seconds=300 for the same reduced RGW load as the cron.
These budgets limit scan duration and volume, not instantaneous request rate.
Adjust the bounds with max-scan-steps / max-scan-seconds;
the wrapper rejects max-scan-seconds above 2700 because the workflow deadline
is 55 minutes.
At 50 outstanding or 900 retained children, candidates are deferred. Dry runs
update local observations and honour the 24-hour settling delay, so the first
run can report only settling; use a separate database for independent probes.
Operator overrides are template parameters: state-db (a path on the state
claim), scan-prefix, settle-seconds and retry-failed=true.
Active jobs and matching completed twins are skipped, with weekly worker inventory
checks. Each child has its own pod, 64Gi LVM scratch PVC and result; twelve children
may run concurrently. Placement allows infra nodes or compute-001/003 and excludes
compute-002. The dispatcher has a permanent 10Gi LVM claim; worker pod cleanup
does not delete it. The hourly cron is enabled through GitOps and retains the
24-hour settling interval.
For an individual store (also useful after inspecting/cleaning a failed attempt):
ARGO_SERVER= argo submit -n vxoptimise --from workflowtemplate/vxoptimise-store \
-p store=s3://vxdata/group/run/example.zarr/
ARGO_SERVER= argo submit -n vxoptimise --from workflowtemplate/vxoptimise-store-large \
-p store=s3://vxdata/group/run/large.zarr/ # retry after a full 64Gi scratch diskThe template normalises the store URI before labelling it: a missing trailing
slash or a Zarr metadata filename resolves to the same store label. A manual
store run is counted by the dispatcher’s dedup and backpressure. Inspect with
kubectl get workflows,pods -n vxoptimise. Store pods are removed two minutes after
completion, so read logs from Loki ({namespace="vxoptimise"}, 90d) or
OpenSearch (14d) rather than kubectl logs. After the self-service rollout,
ordinary argoworkflows-users can submit the fixed upload/store templates;
discovery remains administrator-operated.
The enabled hourly vxoptimise-discovery cron runs normal Zarr conversions with
the reduced walk budget above and disables the bundle-format census
(bundle-audit=false) to reduce RGW reads. Manual/scoped submissions also
default to bundle-audit=false; opt in only with the main inventory
or an independently configured audit checkpoint. The bundles Workflow output
reports versions and coverage without object paths. Bundle publication is not
enabled by this discovery cron.
The separate vxoptimise-bundle-upgrades cron is suspended to reduce RGW traffic.
If re-enabled, it runs hourly at minute 50 and verifies legacy bundles before
creating v3 manifests alongside the originals.
It uses dedicated credentials, 5Gi scratch, a separate checkpoint and mutex.
Catalog references and originals remain unchanged. Exact source/successor
URI/hash mappings remain private on the discovery PVC for an operator’s later
cutover; the Workflow exposes only aggregate results. The operator-only
vxoptimise-bundle-upgrade template defaults to dry-run; pass -p dry-run=false
to publish. See vxoptimise operations
for report locations, coverage limits and resource bounds.
Sources and maintenance
Update this guide when the following sources change; other agent instructions and submission commands should link here instead of repeating these rules:
- run-image parameters and mounts.
- Identity, queue routing, retries, and concurrency and routing guards.
- GPU-class validation and Kueue flavors / borrowing limits.
- Resource caps, sidecar requests, and controller defaults.
infra_iam:data/scheduling/defaults.yaml,data/scheduling/teams.yaml, per-user overrides, and generatedk8s/scheduling/manifests.kuberun:src/kuberun/config.py,cli.py,resources.py, andworkflow.py. Project selection and image naming live inproject.pyandresolution.py.- Tekton build pipelines and tasks, image aliases and namespace enforcement, Argo buildah, and Argo image namespace enforcement.