vxrun batch runs a finite list of commands using one image, inside one Argo
Workflow. Use it for independent inputs or already-expanded parameter sweeps.
Use vxrun workflow when tasks have different dependencies or resource needs.
The cluster must have the run-batch WorkflowTemplate from infra_k8s deployed
in argo-workflows before submission. Deploy that template before releasing a
CLI that submits to it. Tasks preserve their outputs on shared artifacts storage
and, when the template publishes, each item’s output becomes a vxs3 bundle and
the batch one root manifest (see publication).
Input and preparation
Create items.json with one complete argv array per item:
[
{"id": "trial-0", "argv": ["python", "train.py", "lr=0.01", "seed=1"]},
{"id": "trial-1", "argv": ["python", "train.py", "lr=0.001", "seed=2"]}
]IDs are unique, stable names using 1–100 letters, digits, dots, underscores or
hyphens; . and .. are invalid. Each item contains exactly id and argv.
Arguments retain whitespace, quotes and shell metacharacters. They are executed
directly; use an explicit sh -c command if shell interpretation is intended.
NUL and Argo template delimiters ({{/}}) are rejected in argv. Limits are
1,000 items and 512 KB for the complete workflow; split larger submissions.
From a project containing a Dockerfile and .vxrun.yaml:
vxrun batch init . --items items.json
vxrun batch plan batch.yaml --json
vxrun batch build batch.yaml --output prepared.yaml
vxrun batch submit prepared.yaml --run-id training-001 --dry-run --json
vxrun batch submit prepared.yaml --run-id training-001 --preflight --json
vxrun batch submit prepared.yaml --run-id training-001 --json
vxrun batch status WORKFLOW_NAME --jsoninit copies the items into a native Argo batch.yaml; subsequent commands use
that saved list. Without --items, it also writes a small batch_job.py example.
Include the worker in your image. Worker images need sh, mkdir and cat,
but need no vxrun or vxs3 installation. For Pixi images, use an argv beginning
with pixi run, for example ["pixi", "run", "python", "train.py", "seed=1"].
build uses the existing project commit/push/build policy once, pins its image
digest and saves provenance. To use an existing immutable image without building:
vxrun batch render batch.yaml --image REGISTRY/IMAGE@sha256:DIGEST > prepared.yamlplan, render and submit --dry-run run offline. --preflight performs server
dry-run; it checks Workflow admission, not future Pod scheduling or execution.
Submission reports output_uri, the s3://workflow-output/<user>/<name>-<uid>
prefix that publication fills and vxrun batch results reads.
Submission does not require a running client afterward. Repeat the same saved
manifest, scheduling flags and --run-id to recover an uncertain submission;
changed inputs require a new run ID. Recovery lasts while the Workflow exists.
The Python builder uses the same input and produces an ordinary dictionary:
import json
import yaml
from pathlib import Path
from vxrun.batch import batch
document = batch(
"training",
json.loads(Path("items.json").read_text()),
parallelism=4,
cpu="8",
memory="32Gi",
gpu=1,
)
print(yaml.safe_dump(document, sort_keys=False))The shared vxrun.api.plan_workflow, prepare_workflow and submit_workflow
interfaces accept these batch manifests too. The native DAG validator continues
to reject arbitrary template references; run-batch has its own bounded contract.
Run specs: one file per run
vxrun spec authors a batch as one YAML file pinning image, data and commands;
see specs.md (read_doc("vxrun/specs") in vx mcp).
A batch spec can also be posted to vxdispatch, the run register and broker
(mono/apps/vxdispatch-api; read_doc("vx/cluster/dispatch") in vx mcp), instead of vxrun batch submit. Dispatch submits the same run-batch Workflow as the user,
keeps the run’s record beyond Argo’s retention and is the only path to the AWS
target (in testing, approval required). vxrun batch submit remains the direct
path; dispatch records such runs as observed, with less detail.
Resources, attempts and outputs
Edit spec.parallelism (1–25) and the parameters in batch.yaml. Resources are
per worker and shared by every item. Defaults: 2 CPU, 4Gi memory, no GPU,
parallelism 2, two retries on infrastructure errors and a 12-hour attempt deadline.
CPU/memory/GPU and TopoLVM size choices match run-image; requests equal limits.
run_environment chooses production or staging credentials. Scheduling flags
--team, --preemptible and --gpu-class apply to the entire workflow.
retry_policy supports OnError (default), OnFailure and Always; retries
is 0–10, and task_deadline is 1–604800 seconds. An ordinary nonzero application
exit is not retried by the default. One failed item does not stop independent
items; the workflow still reports failure if an item exhausts its attempts.
Retrying does not implement application checkpoint recovery or undo side effects.
Every attempt gets its own /output (VXRUN_OUTPUT_DIR) on a temporary TopoLVM
claim that belongs to the item:
<authenticated-user>/batches/<workflow-name>-<workflow-uid>/items/<item-id>/
attempts/<pod-name>/ /output of one attempt; the successful one is packed
inputs/ declared inputs, read-only at /inputs
checkpoints/ /checkpoints, shared by every attempt of the itemWorkflow UID prevents collisions after Argo deletes an older workflow and its name is reused. Attempts never overwrite one another’s output directory.
/checkpoints (VXRUN_CHECKPOINTS_DIR) is keyed by item, not attempt: save
resumable state there and, at start, resume from whatever an earlier attempt
left. It is never packed and is deleted with the claim, so it covers retries of
one item only; a resubmitted batch starts empty, and a lost node takes the claim
with it (retries are pinned to its node). State that must survive those goes to
the scratch bucket under your run ID. The claim, and so /checkpoints, exists
in every batch: it is 16Gi when local_cache is 0 and local_cache otherwise,
shared by inputs, every attempt’s output and checkpoints.
VXRUN_ITEM_KEY, VXRUN_ATTEMPT, VXRUN_WORKFLOW and VXRUN_WORKFLOW_UID expose
identity to the program. Each Argo worker node reports item-id and output-path
parameters, visible in vxrun batch status; use node status to identify successful
attempts. Directory existence alone does not mean an attempt succeeded.
Standard mounts include /output, /inputs, /checkpoints, /dev/shm, internal
CA trust, ClearML/S3 credentials and optional AWS/vxdata credentials; there are
no NFS or hostPath mounts, so read inputs from S3/vxdata or declare them. /cache
always exists: local_cache: "0" uses Pod-local emptyDir, while a nonzero value
selects an ephemeral TopoLVM volume. VXS3_CACHE_DIR points to /cache/.vxs3.
Both cache modes are disposable; outputs that must survive go under /output.
Path separation is an organization convention, not an access-control boundary.
Save output references before Workflow status expires under the cluster TTL.
Directory variables follow VXRUN_<NAME>_DIR: VXRUN_OUTPUT_DIR, VXRUN_INPUTS_DIR,
VXRUN_CHECKPOINTS_DIR and VXRUN_CACHE_DIR. KUBERUN_OUTPUT and KUBERUN_CACHE
are legacy aliases with the same values; review_job reports them at low
severity, and they will be removed once no worker code reads them.
Publication
With the template’s publish parameter on (its default), two more commands run
inside the cluster from the tools image; they are not meant for a laptop. After
each item’s command, the pack step runs:
vxrun batch pack --source /batch-output/<user>/batches/<wf>-<uid>/items/<id>/attempts/<pod> \
--destination s3://workflow-output/<user>/<wf>-<uid>/items/<id> \
--scratch /cache/pack --manifest-out /tmp/manifest.jsonIt packs the directory into tar shards under --scratch, publishes them to the
exact destination with read-back verification, writes the item’s
manifest.parquet last, and prints {manifest_uri, sha256, files, bytes}. The
destination must be empty; a retry after a partial upload fails and the item is
reported as failed by finalisation, since the writer identity cannot delete.
The Workflow’s exit handler then runs:
vxrun batch finalise --prefix s3://workflow-output/<user>/<wf>-<uid> \
--items-env VXRUN_ITEMS --workflow-status Succeeded --scratch /scratch [--register --title NAME]It writes workflow.json (name, uid, user, status, expected item ids), then
reads every expected item’s manifest and writes items.parquet (one row per
item: item, status, manifest_uri, manifest_sha256, files, bytes,
detail); an item without a complete manifest is failed. Only when every item
succeeded does it merge the item manifests with vxs3 and publish the root
manifest.parquet last, so the root’s presence means the whole batch is there,
like a multipart upload that becomes an object only on completion. A batch with
failures is never completed: it keeps items.parquet and the successful items’
bundles for inspection and re-runs, and gets no root manifest and no catalog
entry. The root is a composed vxs3 bundle: items/<id>/... lists an item’s
files, shards stay where the items published them, and each item’s own manifest
remains a complete bundle. Re-running finalisation is safe: objects already
written with identical content are kept, differing ones are an error. With
--register, a complete root is registered as a vxdata Bundle with
items.parquet as its records table, using VXDATA_TOKEN from the environment.
Credentials come from AWS_PROFILE (default workflow-output) in
AWS_SHARED_CREDENTIALS_FILE; the endpoint from VXRUN_S3_ENDPOINT, the
profile’s endpoint_url, or the cluster default. Reading a composed bundle needs
vxs3 3.1 or later.
Collecting results
vxrun batch results <workflow-name> # prefix, completion, one line per item
vxrun batch results <workflow-name> --json
vxrun batch results <workflow-name> --download ./out # complete batch: ./out/items/<id>/...
vxrun batch results <workflow-name> --download ./out --item trial-0 # one item, works for incomplete batchesresults derives s3://workflow-output/<user>/<name>-<uid> from the live
Workflow (uid, creator, expected items) and lists that prefix once. Workflows
are garbage-collected days after they finish while the data stays, so when the
Workflow is gone it finds <user>/<name>-<uid>/ in the bucket instead
(--user to look under someone else’s area; an ambiguous reused name lists the
candidates), and the prefix itself is accepted in place of the name. Finalise
writes workflow.json (name, uid, user, status, expected items) into the prefix
first, so a finalised batch describes itself. Before the exit handler has run
it reports each item as published or pending; after it, the rows of
items.parquet; complete is whether the root manifest exists, and its
SHA-256 is reported and pinned on download.
Credentials come from the workflow-output profile (--profile or
AWS_PROFILE to override) in AWS_SHARED_CREDENTIALS_FILE or
~/.aws/credentials; Coder workspaces and job containers carry the read-only
keys under that name. For Python, open manifest_uri values with
vxs3.bundle.io.open_bundle, or use client.bundles when the batch was
registered.