Skip to content
VIRDX

Submitting jobs

This is the canonical guide for users and agents submitting workloads to this cluster. It covers kuberun, the shared run-image WorkflowTemplate, and custom Argo Workflows. Ordinary jobs belong in namespace argo-workflows, using the submitter’s authenticated Kubernetes identity. Start with the existing project configuration and the workload’s actual resource and restart requirements.

Choose a scheduling mode

Personal jobs are guaranteed RTX 6000 Ada GPUs; team jobs stay within their RTX 6000 PRO entitlement; preemptible jobs borrow whatever is idle, PRO first. All three modes share the same Kueue cohort.

Mode kuberun options Workflow metadata.labels Placement Borrowing
Personal --no-team --no-preemptible Neither mode label needed Ada (compute-002) only, within the 2-GPU allowance None; gpu-class: rtx-6000-pro is rejected
Team --team <name> --no-preemptible virdx.dev/team: <name> PRO nodes (compute-001/003) only; members of the named team (ve2e, histo, diffsim) None
Preemptible --no-team --preemptible virdx.dev/preemptible: "true" PRO nodes first, then Ada, subject to explicit hardware constraints Idle GPUs, fair-shared between users; evicted whenever a guaranteed Pod needs the node

Scheduling teams ve2e, histo and diffsim are defined in infra_iam (quotas 12 / 5 / 4 PRO, 192 / 128 / 128 CPU, 2Ti / 800Gi / 800Gi). A user may belong to several teams and names the one to draw on per job: --team histo or team: histo in the project config. A submission naming a team the submitter is not in (or no such team, including the legacy value "true" from older clients) is rejected, as are team + preemptible and team + gpu-class: rtx-6000 (team queues are PRO-only).

Personal defaults are 36 CPU / 264Gi / 2 Ada GPUs across admitted Pods, including sidecars, all on Ada. RTX 6000 PRO GPUs are team quota: use --team <name> as a member, or --preemptible (PRO first, evictable) otherwise; a personal --gpu-class rtx-6000-pro is rejected by kuberun and by the cluster. Full-GPU Coder workspaces using the personal queue share this allowance; nvidia.com/gpu.shared slices do not count toward it. Two of the usual 16 CPU / 128Gi containers fit alongside their Argo wait sidecars. Authoritative quotas and any overrides live in infra_iam/data/scheduling/ and its rendered k8s/scheduling/ manifests.

Personal GPU quotas deliberately exceed physical capacity. They are admission caps, not reserved cards: an admitted personal Pod can still wait for a GPU behind other personal work. It does not wait behind preemptible work: when no GPU is free, the scheduler evicts preemptible Pods to place it. The same applies on the PRO nodes: a team Pod evicts preemptible Pods holding its cards. Coordinate long personal GPU runs. Admission also cannot guarantee that a Pod fits on an individual node.

Preemptible capacity has no per-user cap, but it is fair-shared: each submitter has their own user-<name>-preemptible LocalQueue into the shared pool, and Kueue admits pending preemptible work in order of each user’s decayed historical GPU usage (48-hour half-life). Alone, you can use every idle GPU; as soon as someone else queues preemptible work, their steps go ahead of yours until usage evens out. Check your standing with kubectl get localqueue -n argo-workflows user-<name>-preemptible -o jsonpath='{.status.fairSharing}'.

Preemptible jobs must tolerate being killed and restarted. The policy supplies template defaults of a 12-hour deadline per Pod attempt and 10 retries; these are not a checkpoint mechanism or a total Workflow runtime guarantee. Applications must persist progress, resume explicitly, and tolerate repeated execution. Do not opt into preemption merely to obtain more resources unless the workload can handle that behavior.

Choose GPU hardware

Hardware Workflow virdx.dev/gpu-class (client setting: gpu_class) Backend modes
RTX 6000 Ada, 48 GB VRAM rtx-6000 Personal, preemptible
RTX PRO 6000 Blackwell, 96 GB VRAM rtx-6000-pro Personal, preemptible
Either type Omit the label; see client compatibility below for auto Personal, preemptible

For raw Workflows, omit the GPU-class label to allow either type. Use an explicit type when VRAM, architecture, or software compatibility requires it: for example, a job needing 64 GB of VRAM requires rtx-6000-pro. auto is not a valid Kubernetes GPU-class label.

Client compatibility: vxrun 0.2 and later resolve personal and preemptible gpu_class: auto to “either type” (no label) and accept an explicit rtx-6000-pro in personal mode; only --team with rtx-6000 is rejected. vxrun 0.4 and later take a team name (--team histo); older clients send virdx.dev/team: "true", which the cluster rejects. Older kuberun/vxrun clients still resolve personal auto to Ada and reject personal PRO: upgrade, or submit run-image or a custom Workflow directly as shown below. Use --no-team to clear an inherited team.

For raw Workflows, put any required GPU-class label in metadata.labels; Kyverno propagates it to Pods and supplies matching selectors. Prefer this over handwriting nodeSelector or node affinity. A manually supplied GPU product selector is a hard requirement, not a preference. The shared flavor accounts for either type without injecting a conflicting product or node class; kube-scheduler enforces the supplied selector. Do not strip a hardware requirement to make a job start.

For CPU-only work, use gpu: 0 in kuberun/run-image; custom containers should omit nvidia.com/gpu resources and the NVIDIA runtime patch. Do not add a GPU-class label or hardware selector. The queue still determines the pool; Kueue adds nvidia.com/gpu.present: "true" at admission, even for CPU jobs, so ordinary workflows use the GPU compute nodes, not CPU-only infra nodes.

Submit a project with kuberun

Use kuberun when building and running code from an application repository. It builds the image and submits a Workflow referencing run-image.

Example .kuberun.yaml for a personal one-GPU job:

cpu: "16"
memory: "128Gi"
gpu: 1
gpu_class: auto
team: ""          # or a team name such as histo
preemptible: false
request_ratio: 1.0

Use the project’s Dockerfile command, or specify entrypoint / cmd in the config or on the command line. Inspect the effective settings with kuberun config. CLI flags override project .kuberun.yaml, which overrides global ~/.kuberun/config.yaml (with its KUBERUN_* environment overrides). Review the working tree and kuberun’s autocommit setting before running: building may commit and push local changes.

# Personal quota, either GPU type (older clients select Ada here)
vxrun run --no-team --no-preemptible --gpu 1 --gpu-class auto

# Borrow either type, with interruption/restart accepted
vxrun run --no-team --preemptible --gpu 1 --gpu-class auto

# Borrow PRO specifically, for a workload requiring that hardware
vxrun run --no-team --preemptible --gpu 1 --gpu-class rtx-6000-pro

The explicit negative flags clear inherited scheduling modes. Likewise, --gpu-class auto clears an inherited GPU preference (older clients still select Ada in personal mode). For ordinary CPU jobs, change the GPU count to zero and choose the appropriate mode.

cpu and memory are container limits. Kuberun’s request_ratio defaults to 0.5; values from 0.1 to 1.0 reduce CPU/memory requests relative to those limits, with rounding and a minimum of 1 CPU / 1Gi. For example, 16 CPU / 128Gi at 0.5 requests 8 CPU / 64Gi. GPU requests are unchanged. Lower requests consume less admission quota but permit more contention; choose them from expected usage, not simply to fit more jobs. Ratio 1.0 is a predictable starting point. Direct run-image submissions keep requests equal to limits unless cpu_request / memory_request are explicitly supplied.

Multiple projects in one repository

With the monorepo-aware kuberun release, kuberun -C apps/worker run selects the nearest .kuberun.yaml up to the Git root. kuberun --config PATH run selects an explicit config relative to -C (or the caller’s directory). These global options precede the command. Parent configs are not merged; an invalid selected config fails without falling back. YAML paths are relative to their config; CLI Dockerfile/context paths remain repo-relative. Paths must stay within the repo, including symlink targets, and the Dockerfile must be within the build context. For example, apps/worker/.kuberun.yaml can contain:

project: worker
context_dir: ../..
dockerfile: Dockerfile
autocommit: true

This config uses the whole repo as build context. Without an explicit context, the default is also the repo root unless global config overrides it. A nested config discovers its own Dockerfile when none is specified. The config directory, Docker build context, and container WORKDIR are separate concepts.

The source repository is still mono. Project identity controls job names, labels, and /mnt/artifacts/<user>/<image>/<workflow> output paths (vxrun passes the project identity as the image parameter). The image repository is derived from the source repo and full config-directory path: mono/apps/worker here, distinct from mono/tools/worker or a worker repo. There is no image_name config override; changing the project label or build context does not change the image name. Configs in the same directory share an image repository; moving the config changes it. Invalid OCI paths are rejected. The image name also participates in the cache key.

Kuberun passes this derived name through the backend’s image-name (Tekton) or image_name (Argo buildah) parameter. Admission requires the source repo name or a path beneath it, rejecting a submission from mono to foo or other/foo. Tekton additionally allows centrally approved legacy aliases only for their exact repo/context/Dockerfile mapping in the image-name policy. For example, mono + . + apps/vxdata-jobs/Dockerfile may publish vxdata-jobs. The policy supplies the alias for buildkit-image when the parameter is omitted. Root-config kuberun builds omit it for compatibility; the nested apps/vxdata-jobs/.kuberun.yaml config publishes mono/apps/vxdata-jobs instead. Argo buildah also checks the namespace before building, including templateRef calls. These checks govern the supplied build pipelines/templates; registry credential authorization remains a separate boundary.

Autocommit commits only the selected config directory and its descendants, including additions/deletions and hook fixes. Staged changes elsewhere remain staged. Uncommitted shared packages outside that scope are excluded; builds use their committed versions. Commit those changes separately when required. A selected Dockerfile outside the commit scope must be clean, so kuberun’s parsed command matches the version used remotely. A symlinked config keeps its selected directory as the commit scope, rather than the target file’s directory. Without a config, scope is the repository root. Use --no-autocommit to require manual commits; the built-in default is autocommit: true, which project/global settings may override. Pushing still publishes existing commits on the branch.

Jobs carry separate repo and project labels. Use kuberun status --repo mono or kuberun status --repo mono --project worker to filter them. Older jobs without a project label remain visible under the repo filter.

Use kuberun config for setting sources, run --dry-run / build --dry-run for a local plan, and kuberun init to create files explicitly. Config inspection and dry runs do not commit, push, authenticate to the cluster, or generate files.

Deploy the companion templates/tasks/policy before releasing the CLI. Tekton cache lookup, build, and archive keys now include a hash of build arguments; empty arguments retain legacy keys. Build and cache-hit tasks export the actual image repository/digest. Argo buildah exports the pushed image-ref, and run-image accepts image_ref for immutable execution while retaining image for the output directory and backwards-compatible tagged submissions.

Submit an existing image with run-image

Use the installed run-image WorkflowTemplate when the job is one container with standard credentials and storage. Image builds normally use kuberun’s Tekton backend; build manifests are under current/tekton-builds/examples. For Pixi Dockerfiles, see Dockerfile advice for the shared cache mount and layer ordering, and BuildKit package caches for retention behaviour.

Save this as run-image-job.yaml. Replace the example image name, tag, and command with an image already present in Zot and its actual command. Optional image_ref (repo@sha256:...) pins an exact digest and overrides image/image_tag; kuberun sets it for every build:

apiVersion: argoproj.io/v1alpha1
kind: Workflow
metadata:
  generateName: run-example-
  namespace: argo-workflows
spec:
  workflowTemplateRef:
    name: run-image
  arguments:
    parameters:
      - name: image
        value: example-job
      - name: image_tag
        value: replace-with-built-tag
      - name: cmd
        value: "python -u /app/job.py"
      - name: cpu
        value: "16"
      - name: memory
        value: "128Gi"
      - name: gpu
        value: "1"

This example uses personal quota and can run on either GPU type. To require PRO, add metadata.labels: {virdx.dev/gpu-class: rtx-6000-pro}. For Ada, use rtx-6000. To borrow instead, add virdx.dev/preemptible: "true". The same labels apply to custom Workflows below. Kubernetes label values must be strings, so quote "true".

kubectl create --dry-run=server -f run-image-job.yaml -o yaml
kubectl create -f run-image-job.yaml

The image name is relative to zot.fra.virdx.dev:5000, not a full registry URL. Prefer a specific built tag over latest. Set at least one of cmd or entrypoint for manual submissions: the template uses an Argo script wrapper and does not discover the image’s Dockerfile defaults. Kuberun resolves those defaults before submitting. cmd alone is executed through sh -c; entrypoint plus cmd uses shell word splitting, so put complex launch logic in an executable script in the image.

The template defaults to 16 CPU / 128Gi / 0 GPUs, with requests equal to limits. Optional cpu_request and memory_request parameters override requests; they must be positive and no greater than the corresponding limits. Supported limit values are:

Parameter Accepted values
cpu 1, 2, 4, 8, 16, 32, 64, 128
memory 2Gi, 4Gi, 8Gi, 16Gi, 32Gi, 64Gi, 128Gi, 192Gi, 256Gi, 512Gi
gpu Integers 0–8; quota and per-node capacity still apply
local_cache 0 (disabled), 16Gi, 32Gi, 64Gi, 128Gi, 256Gi, 512Gi, 1024Gi

run_environment selects production (default) or staging credentials. The template provides ClearML/S3 configuration, environment-specific credentials, registry authentication, the internal CA, /dev/shm, read-only /mnt/storage, and /output backed by /mnt/artifacts/<username>/<image>/<workflow-name>. S3 credentials are exposed through the standard shared credentials file (AWS_SHARED_CREDENTIALS_FILE), with the same profiles as Coder workspaces: [scratch] (read/write scratch bucket), [workflow-output] (read-only published batch results) and [default] (the same read-only identity as [workflow-output]); every profile carries the internal RGW endpoint_url, and none is real AWS. See the template for exact mounts and environment variables; never copy secret values into job manifests.

local_cache provisions node-local scratch at /cache and exports VXRUN_CACHE_DIR=/cache and VXS3_CACHE_DIR=/cache/.vxs3 (vxs3’s default download/unpack root). Directory variables follow VXRUN_<NAME>_DIR (VXRUN_OUTPUT_DIR is always set); KUBERUN_OUTPUT and KUBERUN_CACHE are legacy aliases with the same values, kept until no worker code reads them. Treat /cache, emptyDir, and the container filesystem as disposable across retries and deletion. Store checkpoints in /output, shared artifacts, object storage, or an appropriate persistent volume, and explicitly reload them after a restart. A newly submitted Workflow has a different /output directory, so resuming a previous submission requires its checkpoint location.

For temporary development checkpoints, prefer the scratch S3 bucket over /output: the NFS export behind /mnt/artifacts is a poor fit for frequent large rewrites. See Scratch storage for development for retention, access and publishing data you want to keep.

The Argo UI can submit the same template with these parameters and Workflow labels. For manifests, use kubectl create with an authenticated user kubeconfig; there is no need to supply creator or username labels yourself. Kyverno derives identity from the API request; through Argo Server, it uses the creator identity set by the server.

Scratch storage for development

The scratch S3 bucket facilitates development: use it for experiments, intermediate data and temporary checkpoints while working out what you need. It has a two-week (14-day) garbage-collection policy. Treat its contents as disposable; scratch is not for permanent storage or inputs and outputs that must remain available for repeatable workflows.

Once you are happy with the data, model or result, publish it to vxdata and create a permanent resource. Use that resource in subsequent workflows. The retained bytes must live in vxdata storage; a resource that only points at s3://scratch/... will still lose its data to scratch GC. See Vxoptimise service workflows for publishing an immutable S3 prefix with vxrun storage upload and --retention permanent. Finish publication before the scratch objects expire.

Access from Coder and jobs

Coder workspaces, run-image and run-batch jobs expose the [scratch] profile through AWS_SHARED_CREDENTIALS_FILE. It carries endpoint_url = https://s3.fra.virdx.dev. Use boto3.Session(profile_name="scratch"), s3fs.S3FileSystem(profile="scratch") or AWS_PROFILE=scratch aws s3 .... For example, this read-only boto3 check uses the workspace or job’s internal CA bundle:

import os
import boto3

s3 = boto3.Session(profile_name="scratch").client(
    "s3",
    endpoint_url="https://s3.fra.virdx.dev",
    verify=os.environ["SSL_CERT_FILE"],
)
result = s3.list_objects_v2(Bucket="scratch", MaxKeys=5)
print("S3 OK — objects returned:", result["KeyCount"])

The Coder template keeps your persistent local home PVC, /mnt/storage read-only and /mnt/artifacts read/write. Startup does not create directories or change ownership on NFS, so it requires no writes to a full or read-only export. Application writes to /mnt/artifacts can still fail when the export is full or read-only. Write development data to your home and upload results to S3. Existing workspace Pods pick up this change after the updated kubernetes-dev template is published and the workspace is rebuilt.

Write under s3://scratch/<username>/<experiment-or-workflow>/. The scratch identity is shared, so this prefix is an organizational convention, not an access boundary. The bucket also has a shared size quota; delete temporary data you no longer need. Retrying or resuming an experiment does not exempt its objects from the 14-day GC policy.

Keep scratch request load modest

Scratch shares Ceph with other workloads. Before uploading a directory, inspect the local source with the advisory analyzer:

vxrun data inspect ./data --scratch

This uses vxs3 to report object counts, sizes, the share below 4 MiB and format-specific layout findings. It suggests changes without uploading data or blocking submission. vxrun review also flags scratch references for review; a reference alone does not mean the workload is inefficient.

  • Consolidate related small records into fewer native files, such as Parquet, when the readers support it. For independent ordinary files, use vxs3 tar bundles targeting 1 GiB shards and keep the manifest with the shards.
  • Keep Zarr native: review chunking and sharding instead of tarring the store. A few large files or occasional checkpoints may need no layout change.
  • Stage inputs once per job onto local NVMe (--local-cache, mounted at /cache) and reuse them across epochs and workers. Batch small writes, avoid repeated S3 listings, and keep transfer concurrency modest across workers.

Prefer inspection before upload. For remote inspection, select only your experiment’s prefix; do not repeatedly scan scratch during training. Scratch mode refuses whole-bucket scans and defaults to 10,000 objects, 64 metadata probes, 4 MiB of metadata and a 30-second cooperative time budget. LIST/HEAD requests are outside the metadata-probe budget, so these are not hard limits on total S3 requests or elapsed time. Remote analysis uses the analyzer’s environment-based S3 credentials and endpoint; selecting the boto3 [scratch] profile alone does not configure it.

Check coverage: a partial or failed scan is not a clean bill of health. Layout analysis does not measure IOPS; also review how often the job issues GET/LIST/PUT requests and whether each worker repeats the same operations. See vxcluster’s docs://storage-practices for packing and inspection examples.

Submit an image batch with run-batch

Use run-batch for a finite set of independent commands sharing one image and resource allocation. It runs one DAG task per JSON item with bounded parallelism. One failed item does not stop the remaining items; the overall Workflow reports failure when an item exhausts its attempts. Outputs are written to temporary node-local storage and, by default, published to the workflow-output bucket as one vxs3 bundle per batch (see publication).

The matching CLI exposes vxrun batch init --items items.json, then plan, render, build, submit and status, alongside vxrun workflow. Deploy run-batch.yaml through ArgoCD before releasing/using that CLI feature. The namespaces, rbac, and argo-workflows-config applications must all reconcile the batch changes: the output-volume steps require the batch-output-volumes ServiceAccount, Role, and RoleBinding in argo-workflows. Verify a small batch reaches a terminal phase with a persisted node tree; a finaliser log alone does not prove the controller could save workflow status. An items.json file contains:

[
  {"id": "trial-0", "argv": ["python", "train.py", "lr=0.01", "seed=1"]},
  {"id": "trial-1", "argv": ["python", "train.py", "lr=0.001", "seed=2"]}
]

Each item contains id and argv, optionally inputs (below; older clients that omit it keep working). IDs must be unique, 1–100 letters, digits, dots, underscores or hyphens; . and .. are invalid. argv is the full command array, including the executable; whitespace and shell metacharacters stay literal. For Pixi images, start it with ["pixi", "run", ...]. The image needs sh, mkdir and cat; it does not need vxrun/vxs3. The CLI rejects NUL and Argo template delimiters in arguments and caps inputs at 1,000 items / 512 KB per Workflow. Apply these same limits when submitting manually.

Direct native submissions use the same template:

apiVersion: argoproj.io/v1alpha1
kind: Workflow
metadata:
  generateName: training-batch-
  namespace: argo-workflows
spec:
  workflowTemplateRef:
    name: run-batch
  parallelism: 2
  arguments:
    parameters:
      - name: vxrun-image
        value: zot.fra.virdx.dev:5000/example-job@sha256:REPLACE_WITH_DIGEST
      - name: items
        value: '[{"id":"trial-0","argv":["python","train.py","seed=1"]},{"id":"trial-1","argv":["python","train.py","seed=2"]}]'
      - name: cpu
        value: "8"
      - name: memory
        value: "32Gi"
      - name: gpu
        value: "1"

Set a real immutable image digest before submitting. Standard personal/team/ preemptible scheduling labels apply to the whole Workflow, as for run-image. Default per-item resources are 2 CPU / 4Gi / 0 GPUs, with equal requests and limits. Resource enums match run-image; spec.parallelism is 1–25. run_environment chooses production/staging credentials. retries defaults to 2 (0–10); retry_policy defaults to OnError and also supports OnFailure or Always. task_deadline defaults to 43200 seconds (CLI range 1–604800). Application failures do not retry under the default; checkpoint/resume logic remains the application’s responsibility.

Each item has a temporary TopoLVM PVC. Each attempt writes to a separate directory on that claim, mounted at /output (VXRUN_OUTPUT_DIR). The workflow UID, item ID and Pod name isolate reused workflow names, items and retries. VXRUN_ITEM_KEY, VXRUN_ATTEMPT, VXRUN_WORKFLOW and VXRUN_WORKFLOW_UID identify the execution. Each worker node exports item-id and output-path parameters; inspect its status to distinguish successful outputs from failed attempts. output-path is an internal path on the item’s claim, not a shared filesystem path accessible from Coder. Use vxrun batch results for published results.

The same claim carries one /checkpoints directory per item (VXRUN_CHECKPOINTS_DIR), shared by every attempt of that item: write resumable state there and, on start, resume from whatever a previous attempt left. It is never packed into the published bundle and is deleted with the claim, so it covers retries of an item, not a resubmitted batch or a lost node (retries are pinned to the claim’s node, see below). Checkpoints that must survive either go to the scratch bucket, keyed by run ID. The claim always exists (16Gi when local_cache is 0, local_cache otherwise) and is shared by inputs, every attempt’s output and checkpoints. The claim layout is:

<user>/batches/<workflow-name>-<workflow-uid>/items/<item-id>/
  attempts/<pod-name>/   /output of one attempt; the successful one is packed
  inputs/                declared inputs, read-only
  checkpoints/           /checkpoints, shared by the item's attempts

Mounts include the standard credentials/CA, /output, /dev/shm and /cache. There are no NFS or hostPath mounts: read inputs from S3/vxdata into /cache. local_cache: "0" uses a Pod-local emptyDir at /cache and a 16Gi output PVC per item. A nonzero local_cache sets both the per-Pod scratch capacity and the per-item output PVC capacity. These are separate allocations: allow space for cached inputs, output across all attempts, and packing shards. VXS3_CACHE_DIR=/cache/.vxs3 in both cases.

The first worker binds its item’s claim to a node. Retries and its publisher must use that node; different items can run on different nodes. Local storage cannot fail over to another node during a node outage. The claim survives worker Pod garbage collection while publication is pending. An item exit hook requests deletion on success, failure or Stop; Kubernetes releases the volume once its worker/publisher Pods are removed (normally after the one-hour log retention). Terminate skips hooks; Workflow deletion/TTL is the ownership cleanup backstop. S3 is the durable output. With publish: "false", /output is temporary too: the application must upload anything it needs to retain. run-image keeps its existing mounts and storage behaviour.

Declared inputs

A batch can declare the data it reads instead of fetching it from inside the command. The inputs parameter (shared by every item) and each item’s inputs list vxs3 bundle selections:

[{"name": "weights", "manifest": "s3://vxdata/models/grader/v3/manifest.parquet",
  "sha256": "<manifest digest>", "paths": ["fold2/checkpoint.pt", "config.yaml"]},
 {"name": "case", "manifest": "s3://vxdata/cohorts/bpmri/2026-10/manifest.parquet",
  "sha256": "<manifest digest>", "prefix": "cases/0017/"}]

manifest + sha256 pin the bundle (a vxs3 BundleRef); one of prefix (a directory), paths (exact members) or pattern (globs) selects from it, and no selector takes the whole bundle. Names are unique across shared and item inputs and become /inputs/<name> in the worker, read-only (VXRUN_INPUTS_DIR). The worker’s fetch-inputs init container (vxrun-tools image) extracts them onto the item’s volume before the command starts, verified against the pinned digest; a retried attempt reuses what is already there and a changed pin for the same name fails. Items without declared inputs skip the fetch. s3://vxdata bundles are read through vxdata API presigned URLs with the run environment’s VXDATA_TOKEN; other buckets use the read-only [default] profile. Count inputs in local_cache: they share the item volume with outputs. env is a JSON list of {name, value} merged into the worker environment; cluster-owned names (VXRUN_*, AWS_*, VXS3_*, CLEARML_*, …) are refused by the CLI.

vxrun spec (mono) authors all of this as one YAML file: image, resources, named bundles, inputs, items and outputs. vxrun spec resolve pins catalog resources and sizes every selection, so the file states the minimum local_cache; vxrun spec compile produces the run-batch Workflow above. The command keeps cluster network access in this version; a network-isolated mode is declared in the spec format but not enforced yet.

Batch output publication

With publish: "true" (the default) every item is a two-step sequence: the command runs as above, then a pack step in tools-image (the vxrun CLI with vxs3) packs the successful attempt’s /output into a vxs3 bundle and publishes it. An exit handler then completes the batch:

s3://workflow-output/<username>/<workflow-name>-<workflow-uid>/
  workflow.json                 name, uid, user, status, expected item ids (written first)
  items.parquet                 one row per expected item: status, manifest reference, files, bytes
  manifest.parquet              written last, only when every item succeeded: the batch is complete
  items/<id>/manifest.parquet   written last by that item's pack step: the item is complete
  items/<id>/shard-*.tar

Shards never move. The root manifest references each item’s shards where its pack step published them (a vxs3 >= 3.1 composed bundle), so one open_bundle on the root sees items/<id>/... for every item, and each item’s own manifest stays a complete bundle. The root exists only for a fully successful batch, like a multipart upload that becomes an object only on completion: a batch with a failed item keeps workflow.json, items.parquet and the successful items’ bundles for inspection and re-runs, but gets no root manifest and no catalog entry. register: "true" registers a complete root as a vxdata Bundle with items.parquet as its records table, using the run_environment vxdata credentials. (Finalisers before vxrun 0.4.7 wrote a root manifest for partial batches too; items.parquet tells them apart.)

Collect with the CLI, from a Coder workspace or a job; both carry the read-only [workflow-output] profile:

vxrun batch results <workflow-name>                      # prefix, completion, one line per item
vxrun batch results <workflow-name> --download ./out     # complete batch -> ./out/items/<id>/...
vxrun batch results <workflow-name> --download ./out --item trial-0   # one item, also when incomplete
vxrun batch results s3://workflow-output/<user>/<workflow-name>-<uid>  # by prefix

Workflows are garbage-collected three days after they finish; the data is not. results reads the live Workflow when it exists, otherwise finds <user>/<name>-<uid>/ in the bucket and takes name, uid and expected items from workflow.json, so results stay readable after the Workflow is gone.

The pack step mounts the item’s output PVC read-only and assembles all of that item’s shards on disk in its own /cache before uploading. It does not buffer whole shards in RAM. /cache is a separate TopoLVM scratch volume when local_cache is nonzero, otherwise a disk-backed emptyDir. With publishing on, local_cache should cover an item’s outputs and packing space. A pack retry after a partial upload fails, because the exact destination must be empty and the writer identity cannot delete; the item is then reported as failed. Terminate skips exit handlers, so a terminated Workflow has neither items.parquet nor a root manifest; prefer Stop, which records the items that did finish.

Published outputs are immutable from a user’s point of view. The writer identity (RGW user workflow-writer: put, get, list, no delete) lives in the workflow-output-writer secret, which only the pack and finalise pods mount; user containers get the bucket read-only through the [workflow-output] profile of aws-credentials. The Kyverno policy argo-restrict-workflow-output-writer rejects any submission that references the writer secret, any run-batch submission that adds spec.templates or spec.podSpecPatch, any pod mounting the secret that is not a controller-created vxrun-tools pod, and in-cluster edits of ArgoCD-managed WorkflowTemplates. Only the fixed PVC allocation and cleanup steps use the batch-output-volumes service account; user workers have no PVC lifecycle permissions. publish: "false" skips publishing and finalisation, but still deletes the temporary output claim after each item.

The pack and finalise steps run the vxrun-tools image: the released vxrun CLI with vxs3 and the vxdata SDK from the conda channel, built from builds/vxrun-tools with the Tekton example pipelinerun-vxrun-tools.yaml. It is pinned by digest inside the template and is deliberately not a parameter, so no user-chosen image ever runs with the writer identity. Each item node exports manifest (JSON {manifest_uri, sha256, files, bytes} from pack, empty when publishing is off) alongside output-path; the inner worker node exports item-id and output-path.

Write a custom Workflow

Use a custom Workflow for a DAG, multiple steps, or container/volume settings that run-image does not expose. This is a complete one-GPU personal example; replace the image and command before submitting as custom-job.yaml:

apiVersion: argoproj.io/v1alpha1
kind: Workflow
metadata:
  generateName: custom-example-
  namespace: argo-workflows
spec:
  entrypoint: main
  parallelism: 1
  imagePullSecrets:
    - name: zot-registry-credentials
  podSpecPatch: '{"runtimeClassName":"nvidia"}'
  templates:
    - name: main
      container:
        image: zot.fra.virdx.dev:5000/example-job:replace-with-built-tag
        imagePullPolicy: Always
        command: [python, -u, /app/job.py]
        resources:
          requests:
            cpu: "16"
            memory: 128Gi
            nvidia.com/gpu: "1"
          limits:
            cpu: "16"
            memory: 128Gi
            nvidia.com/gpu: "1"

CPU/memory requests and limits must be explicit for each application container. GPU requests and limits must be equal whole numbers. GPU Pods require runtimeClassName: nvidia, set through podSpecPatch here; apply it only to GPU templates when mixing CPU and GPU steps. Standard Argo sidecar requests are injected by policy. Custom Workflows must explicitly configure any storage, credentials, or CA mounts they need; they do not inherit run-image’s mounts. Use imagePullPolicy: Always for internal registry images because tags can change; the example above sets it explicitly. Before making this example preemptible, add persistent checkpoint/output storage and application restart behavior.

For human-submitted Workflows, permitted hostPath roots are /mnt/storage, /mnt/scratch, and /mnt/artifacts; /mnt/storage must be mounted read-only. Prefer the existing run-image mounts when they meet the workload’s needs.

Do not set Kueue queue labels, Kueue WorkloadPriorityClass labels, priorityClassName, preemptionPolicy, or identity labels yourself, including inside template metadata or podSpecPatch. Kyverno assigns queue, priority, identity, the Workflow service account, and the per-user concurrency semaphore. Final Pods must match their owning Workflow’s routing. Mode and queue are immutable after submission; create a new Workflow to change them.

Reusable WorkflowTemplates belong in current/argo-workflows/templates/<name>.yaml and are deployed through ArgoCD after merge. Ordinary users cannot create them directly. One-off Workflow objects can be submitted with kubectl create -f custom-job.yaml.

D3N routing inventory

All newly created Argo workflow Pods in argo-workflows, autoresearch, and vxoptimise receive /etc/virdx/d3n/d3n_instances.json through the inject-d3n-config policy. This covers run-image, custom Workflows, referenced WorkflowTemplates and multi-container steps, including init containers. No template-specific volume or mount is needed. Unrelated Pods and already-running Pods are unchanged.

Coder workspace Pods receive the same mount in coder-workspaces, selected by app.kubernetes.io/name: coder-workspace (set by the Coder Terraform template). No workspace image rebuild or template-version update is needed. Existing workspaces pick it up when their Pods are recreated, for example on the next stop/start. The policy does not restart running workspaces. Both regular and init containers receive the mount. The workspace template also exports VXS3_D3N_INSTANCES pointing at that file, so vxs3 discovery treats the deployed inventory as authoritative and fails if the ConfigMap is absent instead of falling back to gateway routing.

The file contains stable Ketama node names and HTTPS endpoints. Ansible owns the namespace-local d3n-instances ConfigMaps and generates identical content from the RGW inventory. Do not add another copy to an ArgoCD application. Reserve volume name d3n-instances and mount path /etc/virdx/d3n for this shared configuration; put application-specific files elsewhere.

The mount is read-only and optional, so jobs that do not use D3N can start before the ConfigMap is provisioned. A D3N client must explicitly read the file and fail if it is missing or invalid, without silently using a bundled list:

from vxs3.transport.instances import load_instances
from vxs3.transport.routing import select_instance

instances = load_instances("/etc/virdx/d3n/d3n_instances.json")
instance = select_instance("example", "shard.tar", instances=instances)

Python and JavaScript clients consume the same JSON; the hashing contract and compatibility vectors live in mono/packages/vxs3/ROUTING.md. Load one snapshot per client or operation. Projected files can update later, but an already-loaded vxs3 tuple stays fixed. Coordinate client reloads when changing ring membership. Making this file available does not enable Ketama routing in existing download code; the application must use the routing functions.

For rollout, include argo-workflows, autoresearch, vxoptimise, and coder-workspaces in infra_ansible/inventories/group_vars/k3s_cluster/k3s_d3n.yaml, then run pixi run playbook k3s_d3n there. The mutation policy is deployed through the kyverno-policies ArgoCD application after merge, together with the narrow Coder mount-path exception in restrict-volume-mounts. New workflow namespaces must be added to both the Ansible list and the policy’s namespace match.

Size jobs for both quota and hardware

  • Kueue accounts for Pod requests, including sidecars and the effective init request, across all admitted Pods in the queue. The standard Argo wait container adds 100m CPU / 128Mi per Pod. Two 16 CPU / 128Gi applications fit the default personal quota; a third does not, even without a GPU.
  • GPUs cannot be split across nodes for one Pod. Current full-GPU capacity is 5 Ada on compute-002 and 8 PRO on each of compute-001/003. Three other Adas are time-sliced for Coder and are not full nvidia.com/gpu capacity. An eight-GPU Pod exceeds personal quota even though a PRO node can fit it. One Pod cannot mix Ada and PRO because the types are on different nodes. Distributed training needs separate Pods and application support.
  • Custom application containers are capped at 192 CPU / 512Gi / 8 GPUs by policy; run-image has the narrower CPU enum above. These ceilings are not entitlements or guarantees that hardware is available.
  • Workflow spec.parallelism defaults to 20 and cannot exceed 25. This limits simultaneously active Pods, not the number of steps. Select it to suit the queue’s aggregate quota. Personal GPU work can admit up to two one-GPU Pods or one two-GPU Pod, subject to CPU/memory quota and physical capacity.
  • The per-identity semaphore defaults to 10 concurrent Workflows across modes; excess submissions wait. The controller also caps namespace concurrency at 200 Workflows. Raising parallelism does not increase entitlement.

Validate and inspect a submission

Check the Kubernetes context and effective mode/resources first. For a custom Workflow, run argo lint --offline --strict custom-job.yaml. Then use kubectl create --dry-run=server -f <file> -o yaml to inspect Workflow admission and injected routing without creating the job. A server dry run does not create the eventual Pods or prove that resource requests are schedulable; those checks also require the quota and hardware review above.

For run-image, inspect the deployed template with kubectl get workflowtemplate run-image -n argo-workflows -o yaml if it may lag the repository. Offline lint of a template reference needs that template available locally and does not emulate Kyverno identity injection.

After an authorized submission, inspect both Workflow and Pod state:

argo get -n argo-workflows WORKFLOW_NAME
kubectl get pods -n argo-workflows -l workflows.argoproj.io/workflow=WORKFLOW_NAME
kubectl describe pod -n argo-workflows POD_NAME
kubectl logs -n argo-workflows POD_NAME -c main -f

A Workflow waiting on its semaphore has not yet started its Pods. A Pod with a Kueue scheduling gate is waiting for admission; an admitted but unschedulable Pod may be blocked on node capacity, storage, or a hard selector. Inspect events before changing the configuration. Operators can use util/kueue-status and util/gpu-status; users may need an operator to inspect cluster-scoped queues. Do not relabel an existing job or remove a required hardware constraint to work around Pending state.

Completed Pods, including failed attempts, are normally removed after one hour. Use Workflow status and Loki for older failures; container files and scratch are not durable output. Workflows normally expire two days after success or seven days after other completion.

To cancel a job, use argo stop (runs exit handlers), argo terminate (does not), or the matching Argo UI buttons; argo delete also works but leaves no record. Those are the only changes a submitter may make to a Workflow after admission: everything else in spec, status and the labels is locked by argo-restrict-user-workflow-updates, so argo retry, argo resume and argo suspend need an operator. Resubmit to change a job.

Vxoptimise service workflows

Self-service uploads and single-store optimisation use vxrun storage after the backend rollout described in the deployment notes. These commands submit fixed templates directly to Kubernetes with your Coder OIDC login; they do not build user code or upload bytes from the client.

vxrun storage upload /mnt/storage/example/dataset --name example \
  --project example-project --description "Synthetic reference dataset" \
  --data-role reference --retention review-180d --wait
vxrun storage upload /mnt/artifacts/example/dataset --metadata metadata.json --dry-run
vxrun storage upload s3://example-source/dataset/ --metadata metadata.json --wait
vxrun storage optimise s3://vxdata/example/store.zarr/ --wait
vxrun storage status
vxrun storage status --project example-project
vxrun storage status JOB_NAME --json
vxrun logs JOB_NAME --namespace vxoptimise --follow

Uploads require a name, project slug, description, data role and retention intention, supplied as flags or metadata JSON. They accept shared paths beneath /mnt/storage and /mnt/artifacts, or s3://bucket/prefix/ on the cluster’s object store after the S3-capable worker rollout; HOME paths cannot be used directly. The two shared paths are existing read-only NFS mounts, with files readable by UID 1000. The upload worker runs as UID/GID 1000 and writes only to its private scratch/output volumes and vxdata; submitter attribution uses the authenticated workflow identity. S3 prefixes are downloaded into private scratch using a fixed read-only source identity, then follow the same Zarr/bundling rules. Size/count limits are checked before download; conditional reads and a second listing reject source changes before publication. Prefixes must be below a bucket root; they do not select individual objects. A trailing slash is added when absent. Eligible source buckets are controlled by S3 IAM grants. Endpoint/credential overrides are not exposed. The CLI requires vxrun.virdx.dev/storage-s3-contract: "1" on the deployed template. The repository pins the S3-capable worker with this capability enabled; both take effect after GitOps sync. Reverting to an NFS-only image must disable the capability in the same change. Inputs must remain immutable. The worker accepts valid root Zarr stores, standalone regular files of at least 100 MiB, and ordinary directories. Directories are always bundled into vxs3 tar shards targeting 1 GiB: small files within a directory cannot become loose S3 objects. Small standalone files are rejected before copying or allocation; group related files into a directory. Nested/malformed Zarr, links and special files are rejected. Inputs are limited to 256 GiB/100,000 entries; bundle members must fit a 4 GiB shard, while standalone files stay native and are exempt from that shard limit. Uploads allocate a vxdata storage prefix and register a verified manifest with submitter/workflow provenance. Retention metadata does not trigger automatic deletion. Existing vxdata stores are mirrored into vxdata-optimised without creating another catalog identity.

Workflow labels include app.kubernetes.io/name=vxoptimise, vxoptimise.virdx.dev/operation=upload|optimise, kuberun.virdx.com/task=storage-upload|storage-optimise, project on uploads, and the admitted virdx.dev/username. They can also be selected with kubectl -l. Argo reads the worker’s ARGO_PROGRESS_FILE (N/M); vxrun status and --wait show these work units, and logs contain stage updates. vxrun storage status defaults to your upload and optimise jobs in the storage namespace and shows the verified data coordinate on success, plus the vxdata resource identifier and manifest URI for uploads. It supports a job name, --project, --operation, --all, and --json for full result fields. Upload totals expand after source listing and optimisation determine the object counts; S3 download logs include file and byte progress. Mirror progress is coarse. Refresh can lag by about a minute. The Argo web server currently cannot view this namespace; use vxrun or ARGO_SERVER= argo get -n vxoptimise JOB_NAME.

Storage status reads retained Workflow objects, so results survive the five-minute pod/scratch cleanup. Successful upload workflows expire after two days; successful optimise workflows after 20 minutes; failed/errored workflows after seven days. Once the Workflow is deleted, its status entry disappears. Workflow GC does not delete published vxdata objects or catalog records. The CLI prints the stable vxupload-<workflow-uid> identifier on submission; save it for later SDK lookup with the Coder environment’s VXDATA_TOKEN. Read file_metadata.data_uri only when file_metadata.state is ready. The record may not exist for pending or rejected jobs. Automatic catalog history is deferred; v1 status uses Kubernetes.

--dry-run prints the request offline and never checks credentials or remote source existence. --wait --timeout SECONDS stops waiting without cancelling the job. Pending/failed uploads require operator reconciliation before resubmission; the vxdata record preserves the allocated prefix after Argo garbage collection.

S3 discovery and conversion run in the dedicated vxoptimise namespace with separate read-only dispatcher and read/write worker S3 credentials. See the deployment notes. Discovery remains an administrator-operated CPU service. Self-service and discovery share the namespace quota/semaphore; neither uses the shared run-image template or personal/team queues.

After GitOps sync and Secret creation, submit the namespaced template directly through Kubernetes (the Argo Server remains scoped to argo-workflows):

ARGO_SERVER= argo submit -n vxoptimise --from workflowtemplate/vxoptimise-discovery \
  -p dry-run=true -p max-stores=10

Omit dry-run=true to submit independent conversion workflows, initially with -p max-stores=1. Dry and real runs share the default state database and each observed Zarr root is next due 24 hours later, so a real run within a day of a dry run reports idle; wait the interval, or give probes their own state-db. The template defaults to the released image and real submission. Upload, store and discovery templates pin the same worker release; discovery passes its image parameter to child store workflows. max-stores=0 removes the submission-count limit. The hourly cron permits 5,000 scan steps or five minutes, whichever comes first, and resumes from its persistent SQLite checkpoints on the next run. Manual template submissions default to 10,000 steps and 45 minutes; pass -p max-scan-steps=5000 and -p max-scan-seconds=300 for the same reduced RGW load as the cron. These budgets limit scan duration and volume, not instantaneous request rate. Adjust the bounds with max-scan-steps / max-scan-seconds; the wrapper rejects max-scan-seconds above 2700 because the workflow deadline is 55 minutes. At 50 outstanding or 900 retained children, candidates are deferred. Dry runs update local observations and honour the 24-hour settling delay, so the first run can report only settling; use a separate database for independent probes. Operator overrides are template parameters: state-db (a path on the state claim), scan-prefix, settle-seconds and retry-failed=true. Active jobs and matching completed twins are skipped, with weekly worker inventory checks. Each child has its own pod, 64Gi LVM scratch PVC and result; twelve children may run concurrently. Placement allows infra nodes or compute-001/003 and excludes compute-002. The dispatcher has a permanent 10Gi LVM claim; worker pod cleanup does not delete it. The hourly cron is enabled through GitOps and retains the 24-hour settling interval.

For an individual store (also useful after inspecting/cleaning a failed attempt):

ARGO_SERVER= argo submit -n vxoptimise --from workflowtemplate/vxoptimise-store \
  -p store=s3://vxdata/group/run/example.zarr/
ARGO_SERVER= argo submit -n vxoptimise --from workflowtemplate/vxoptimise-store-large \
  -p store=s3://vxdata/group/run/large.zarr/   # retry after a full 64Gi scratch disk

The template normalises the store URI before labelling it: a missing trailing slash or a Zarr metadata filename resolves to the same store label. A manual store run is counted by the dispatcher’s dedup and backpressure. Inspect with kubectl get workflows,pods -n vxoptimise. Store pods are removed two minutes after completion, so read logs from Loki ({namespace="vxoptimise"}, 90d) or OpenSearch (14d) rather than kubectl logs. After the self-service rollout, ordinary argoworkflows-users can submit the fixed upload/store templates; discovery remains administrator-operated.

The enabled hourly vxoptimise-discovery cron runs normal Zarr conversions with the reduced walk budget above and disables the bundle-format census (bundle-audit=false) to reduce RGW reads. Manual/scoped submissions also default to bundle-audit=false; opt in only with the main inventory or an independently configured audit checkpoint. The bundles Workflow output reports versions and coverage without object paths. Bundle publication is not enabled by this discovery cron.

The separate vxoptimise-bundle-upgrades cron is suspended to reduce RGW traffic. If re-enabled, it runs hourly at minute 50 and verifies legacy bundles before creating v3 manifests alongside the originals. It uses dedicated credentials, 5Gi scratch, a separate checkpoint and mutex. Catalog references and originals remain unchanged. Exact source/successor URI/hash mappings remain private on the discovery PVC for an operator’s later cutover; the Workflow exposes only aggregate results. The operator-only vxoptimise-bundle-upgrade template defaults to dry-run; pass -p dry-run=false to publish. See vxoptimise operations for report locations, coverage limits and resource bounds.

Sources and maintenance

Update this guide when the following sources change; other agent instructions and submission commands should link here instead of repeating these rules:

Navigation

Type to search…

↑↓ navigate↵ selectEsc close