Skip to content
VIRDX

Image batches

vxrun batch runs a finite list of commands using one image, inside one Argo Workflow. Use it for independent inputs or already-expanded parameter sweeps. Use vxrun workflow when tasks have different dependencies or resource needs.

The cluster must have the run-batch WorkflowTemplate from infra_k8s deployed in argo-workflows before submission. Deploy that template before releasing a CLI that submits to it. Tasks preserve their outputs on shared artifacts storage and, when the template publishes, each item’s output becomes a vxs3 bundle and the batch one root manifest (see publication).

Input and preparation

Create items.json with one complete argv array per item:

[
  {"id": "trial-0", "argv": ["python", "train.py", "lr=0.01", "seed=1"]},
  {"id": "trial-1", "argv": ["python", "train.py", "lr=0.001", "seed=2"]}
]

IDs are unique, stable names using 1–100 letters, digits, dots, underscores or hyphens; . and .. are invalid. Each item contains exactly id and argv. Arguments retain whitespace, quotes and shell metacharacters. They are executed directly; use an explicit sh -c command if shell interpretation is intended. NUL and Argo template delimiters ({{/}}) are rejected in argv. Limits are 1,000 items and 512 KB for the complete workflow; split larger submissions.

From a project containing a Dockerfile and .vxrun.yaml:

vxrun batch init . --items items.json
vxrun batch plan batch.yaml --json
vxrun batch build batch.yaml --output prepared.yaml
vxrun batch submit prepared.yaml --run-id training-001 --dry-run --json
vxrun batch submit prepared.yaml --run-id training-001 --preflight --json
vxrun batch submit prepared.yaml --run-id training-001 --json
vxrun batch status WORKFLOW_NAME --json

init copies the items into a native Argo batch.yaml; subsequent commands use that saved list. Without --items, it also writes a small batch_job.py example. Include the worker in your image. Worker images need sh, mkdir and cat, but need no vxrun or vxs3 installation. For Pixi images, use an argv beginning with pixi run, for example ["pixi", "run", "python", "train.py", "seed=1"].

build uses the existing project commit/push/build policy once, pins its image digest and saves provenance. To use an existing immutable image without building:

vxrun batch render batch.yaml --image REGISTRY/IMAGE@sha256:DIGEST > prepared.yaml

plan, render and submit --dry-run run offline. --preflight performs server dry-run; it checks Workflow admission, not future Pod scheduling or execution. Submission reports output_uri, the s3://workflow-output/<user>/<name>-<uid> prefix that publication fills and vxrun batch results reads. Submission does not require a running client afterward. Repeat the same saved manifest, scheduling flags and --run-id to recover an uncertain submission; changed inputs require a new run ID. Recovery lasts while the Workflow exists.

The Python builder uses the same input and produces an ordinary dictionary:

import json
import yaml
from pathlib import Path
from vxrun.batch import batch

document = batch(
    "training",
    json.loads(Path("items.json").read_text()),
    parallelism=4,
    cpu="8",
    memory="32Gi",
    gpu=1,
)
print(yaml.safe_dump(document, sort_keys=False))

The shared vxrun.api.plan_workflow, prepare_workflow and submit_workflow interfaces accept these batch manifests too. The native DAG validator continues to reject arbitrary template references; run-batch has its own bounded contract.

Run specs: one file per run

vxrun spec authors a batch as one YAML file pinning image, data and commands; see specs.md (read_doc("vxrun/specs") in vx mcp).

A batch spec can also be posted to vxdispatch, the run register and broker (mono/apps/vxdispatch-api; read_doc("vx/cluster/dispatch") in vx mcp), instead of vxrun batch submit. Dispatch submits the same run-batch Workflow as the user, keeps the run’s record beyond Argo’s retention and is the only path to the AWS target (in testing, approval required). vxrun batch submit remains the direct path; dispatch records such runs as observed, with less detail.

Resources, attempts and outputs

Edit spec.parallelism (1–25) and the parameters in batch.yaml. Resources are per worker and shared by every item. Defaults: 2 CPU, 4Gi memory, no GPU, parallelism 2, two retries on infrastructure errors and a 12-hour attempt deadline. CPU/memory/GPU and TopoLVM size choices match run-image; requests equal limits. run_environment chooses production or staging credentials. Scheduling flags --team, --preemptible and --gpu-class apply to the entire workflow.

retry_policy supports OnError (default), OnFailure and Always; retries is 0–10, and task_deadline is 1–604800 seconds. An ordinary nonzero application exit is not retried by the default. One failed item does not stop independent items; the workflow still reports failure if an item exhausts its attempts. Retrying does not implement application checkpoint recovery or undo side effects.

Every attempt gets its own /output (VXRUN_OUTPUT_DIR) on a temporary TopoLVM claim that belongs to the item:

<authenticated-user>/batches/<workflow-name>-<workflow-uid>/items/<item-id>/
  attempts/<pod-name>/   /output of one attempt; the successful one is packed
  inputs/                declared inputs, read-only at /inputs
  checkpoints/           /checkpoints, shared by every attempt of the item

Workflow UID prevents collisions after Argo deletes an older workflow and its name is reused. Attempts never overwrite one another’s output directory.

/checkpoints (VXRUN_CHECKPOINTS_DIR) is keyed by item, not attempt: save resumable state there and, at start, resume from whatever an earlier attempt left. It is never packed and is deleted with the claim, so it covers retries of one item only; a resubmitted batch starts empty, and a lost node takes the claim with it (retries are pinned to its node). State that must survive those goes to the scratch bucket under your run ID. The claim, and so /checkpoints, exists in every batch: it is 16Gi when local_cache is 0 and local_cache otherwise, shared by inputs, every attempt’s output and checkpoints. VXRUN_ITEM_KEY, VXRUN_ATTEMPT, VXRUN_WORKFLOW and VXRUN_WORKFLOW_UID expose identity to the program. Each Argo worker node reports item-id and output-path parameters, visible in vxrun batch status; use node status to identify successful attempts. Directory existence alone does not mean an attempt succeeded.

Standard mounts include /output, /inputs, /checkpoints, /dev/shm, internal CA trust, ClearML/S3 credentials and optional AWS/vxdata credentials; there are no NFS or hostPath mounts, so read inputs from S3/vxdata or declare them. /cache always exists: local_cache: "0" uses Pod-local emptyDir, while a nonzero value selects an ephemeral TopoLVM volume. VXS3_CACHE_DIR points to /cache/.vxs3. Both cache modes are disposable; outputs that must survive go under /output. Path separation is an organization convention, not an access-control boundary. Save output references before Workflow status expires under the cluster TTL.

Directory variables follow VXRUN_<NAME>_DIR: VXRUN_OUTPUT_DIR, VXRUN_INPUTS_DIR, VXRUN_CHECKPOINTS_DIR and VXRUN_CACHE_DIR. KUBERUN_OUTPUT and KUBERUN_CACHE are legacy aliases with the same values; review_job reports them at low severity, and they will be removed once no worker code reads them.

Publication

With the template’s publish parameter on (its default), two more commands run inside the cluster from the tools image; they are not meant for a laptop. After each item’s command, the pack step runs:

vxrun batch pack --source /batch-output/<user>/batches/<wf>-<uid>/items/<id>/attempts/<pod> \
  --destination s3://workflow-output/<user>/<wf>-<uid>/items/<id> \
  --scratch /cache/pack --manifest-out /tmp/manifest.json

It packs the directory into tar shards under --scratch, publishes them to the exact destination with read-back verification, writes the item’s manifest.parquet last, and prints {manifest_uri, sha256, files, bytes}. The destination must be empty; a retry after a partial upload fails and the item is reported as failed by finalisation, since the writer identity cannot delete.

The Workflow’s exit handler then runs:

vxrun batch finalise --prefix s3://workflow-output/<user>/<wf>-<uid> \
  --items-env VXRUN_ITEMS --workflow-status Succeeded --scratch /scratch [--register --title NAME]

It writes workflow.json (name, uid, user, status, expected item ids), then reads every expected item’s manifest and writes items.parquet (one row per item: item, status, manifest_uri, manifest_sha256, files, bytes, detail); an item without a complete manifest is failed. Only when every item succeeded does it merge the item manifests with vxs3 and publish the root manifest.parquet last, so the root’s presence means the whole batch is there, like a multipart upload that becomes an object only on completion. A batch with failures is never completed: it keeps items.parquet and the successful items’ bundles for inspection and re-runs, and gets no root manifest and no catalog entry. The root is a composed vxs3 bundle: items/<id>/... lists an item’s files, shards stay where the items published them, and each item’s own manifest remains a complete bundle. Re-running finalisation is safe: objects already written with identical content are kept, differing ones are an error. With --register, a complete root is registered as a vxdata Bundle with items.parquet as its records table, using VXDATA_TOKEN from the environment.

Credentials come from AWS_PROFILE (default workflow-output) in AWS_SHARED_CREDENTIALS_FILE; the endpoint from VXRUN_S3_ENDPOINT, the profile’s endpoint_url, or the cluster default. Reading a composed bundle needs vxs3 3.1 or later.

Collecting results

vxrun batch results <workflow-name>                     # prefix, completion, one line per item
vxrun batch results <workflow-name> --json
vxrun batch results <workflow-name> --download ./out    # complete batch: ./out/items/<id>/...
vxrun batch results <workflow-name> --download ./out --item trial-0   # one item, works for incomplete batches

results derives s3://workflow-output/<user>/<name>-<uid> from the live Workflow (uid, creator, expected items) and lists that prefix once. Workflows are garbage-collected days after they finish while the data stays, so when the Workflow is gone it finds <user>/<name>-<uid>/ in the bucket instead (--user to look under someone else’s area; an ambiguous reused name lists the candidates), and the prefix itself is accepted in place of the name. Finalise writes workflow.json (name, uid, user, status, expected items) into the prefix first, so a finalised batch describes itself. Before the exit handler has run it reports each item as published or pending; after it, the rows of items.parquet; complete is whether the root manifest exists, and its SHA-256 is reported and pinned on download. Credentials come from the workflow-output profile (--profile or AWS_PROFILE to override) in AWS_SHARED_CREDENTIALS_FILE or ~/.aws/credentials; Coder workspaces and job containers carry the read-only keys under that name. For Python, open manifest_uri values with vxs3.bundle.io.open_bundle, or use client.bundles when the batch was registered.

Navigation

Type to search…

↑↓ navigate↵ selectEsc close