---
title: "Image batches"
image: "https://docs.virdx.dev/img/virdx-social-card.png"
---

> Documentation Index
> Fetch the complete documentation index at: https://docs.virdx.dev/llms.txt
> Use this file to discover all available pages before exploring further.

# Image batches

`vxrun batch` runs a finite list of commands using one image, inside one Argo
Workflow. Use it for independent inputs or already-expanded parameter sweeps.
Use `vxrun workflow` when tasks have different dependencies or resource needs.

The cluster must have the `run-batch` WorkflowTemplate from `infra_k8s` deployed
in `argo-workflows` before submission. Deploy that template before releasing a
CLI that submits to it. Tasks preserve their outputs on shared artifacts storage
and, when the template publishes, each item's output becomes a vxs3 bundle and
the batch one root manifest (see [publication](#publication)).

## Input and preparation

Create `items.json` with one complete argv array per item:

```json
[
  {"id": "trial-0", "argv": ["python", "train.py", "lr=0.01", "seed=1"]},
  {"id": "trial-1", "argv": ["python", "train.py", "lr=0.001", "seed=2"]}
]
```

IDs are unique, stable names using 1–100 letters, digits, dots, underscores or
hyphens; `.` and `..` are invalid. Each item contains exactly `id` and `argv`.
Arguments retain whitespace, quotes and shell metacharacters. They are executed
directly; use an explicit `sh -c` command if shell interpretation is intended.
NUL and Argo template delimiters (`{{`/`}}`) are rejected in argv. Limits are
1,000 items and 512 KB for the complete workflow; split larger submissions.

From a project containing a Dockerfile and `.vxrun.yaml`:

```sh
vxrun batch init . --items items.json
vxrun batch plan batch.yaml --json
vxrun batch build batch.yaml --output prepared.yaml
vxrun batch submit prepared.yaml --run-id training-001 --dry-run --json
vxrun batch submit prepared.yaml --run-id training-001 --preflight --json
vxrun batch submit prepared.yaml --run-id training-001 --json
vxrun batch status WORKFLOW_NAME --json
```

`init` copies the items into a native Argo `batch.yaml`; subsequent commands use
that saved list. Without `--items`, it also writes a small `batch_job.py` example.
Include the worker in your image. Worker images need `sh`, `mkdir` and `cat`,
but need no vxrun or vxs3 installation. For Pixi images, use an argv beginning
with `pixi run`, for example `["pixi", "run", "python", "train.py", "seed=1"]`.

`build` uses the existing project commit/push/build policy once, pins its image
digest and saves provenance. To use an existing immutable image without building:

```sh
vxrun batch render batch.yaml --image REGISTRY/IMAGE@sha256:DIGEST > prepared.yaml
```

`plan`, `render` and `submit --dry-run` run offline. `--preflight` performs server
dry-run; it checks Workflow admission, not future Pod scheduling or execution.
Submission reports `output_uri`, the `s3://workflow-output/<user>/<name>-<uid>`
prefix that [publication](#publication) fills and `vxrun batch results` reads.
Submission does not require a running client afterward. Repeat the same saved
manifest, scheduling flags and `--run-id` to recover an uncertain submission;
changed inputs require a new run ID. Recovery lasts while the Workflow exists.

The Python builder uses the same input and produces an ordinary dictionary:

```python
import json
import yaml
from pathlib import Path
from vxrun.batch import batch

document = batch(
"training",
json.loads(Path("items.json").read_text()),
parallelism=4,
cpu="8",
memory="32Gi",
gpu=1,
)
print(yaml.safe_dump(document, sort_keys=False))
```

The shared `vxrun.api.plan_workflow`, `prepare_workflow` and `submit_workflow`
interfaces accept these batch manifests too. The native DAG validator continues
to reject arbitrary template references; `run-batch` has its own bounded contract.

## Run specs: one file per run

`vxrun spec` authors a batch as one YAML file pinning image, data and commands;
see [specs.md](specs.md) (`read_doc("vxrun/specs")` in vx mcp).

A batch spec can also be posted to vxdispatch, the run register and broker
(`mono/apps/vxdispatch-api`; `read_doc("vx/cluster/dispatch")` in vx mcp), instead of `vxrun
batch submit`. Dispatch submits the same `run-batch` Workflow as the user,
keeps the run's record beyond Argo's retention and is the only path to the AWS
target (in testing, approval required). `vxrun batch submit` remains the direct
path; dispatch records such runs as observed, with less detail.

## Resources, attempts and outputs

Edit `spec.parallelism` (1–25) and the parameters in `batch.yaml`. Resources are
per worker and shared by every item. Defaults: 2 CPU, 4Gi memory, no GPU,
parallelism 2, two retries on infrastructure errors and a 12-hour attempt deadline.
CPU/memory/GPU and TopoLVM size choices match `run-image`; requests equal limits.
`run_environment` chooses production or staging credentials. Scheduling flags
`--team`, `--preemptible` and `--gpu-class` apply to the entire workflow.

`retry_policy` supports `OnError` (default), `OnFailure` and `Always`; `retries`
is 0–10, and `task_deadline` is 1–604800 seconds. An ordinary nonzero application
exit is not retried by the default. One failed item does not stop independent
items; the workflow still reports failure if an item exhausts its attempts.
Retrying does not implement application checkpoint recovery or undo side effects.

Every attempt gets its own `/output` (`VXRUN_OUTPUT_DIR`) on a temporary TopoLVM
claim that belongs to the item:

```text
<authenticated-user>/batches/<workflow-name>-<workflow-uid>/items/<item-id>/
  attempts/<pod-name>/   /output of one attempt; the successful one is packed
  inputs/                declared inputs, read-only at /inputs
  checkpoints/           /checkpoints, shared by every attempt of the item
```

Workflow UID prevents collisions after Argo deletes an older workflow and its
name is reused. Attempts never overwrite one another's output directory.

`/checkpoints` (`VXRUN_CHECKPOINTS_DIR`) is keyed by item, not attempt: save
resumable state there and, at start, resume from whatever an earlier attempt
left. It is never packed and is deleted with the claim, so it covers retries of
one item only; a resubmitted batch starts empty, and a lost node takes the claim
with it (retries are pinned to its node). State that must survive those goes to
the `scratch` bucket under your run ID. The claim, and so `/checkpoints`, exists
in every batch: it is 16Gi when `local_cache` is `0` and `local_cache` otherwise,
shared by inputs, every attempt's output and checkpoints.
`VXRUN_ITEM_KEY`, `VXRUN_ATTEMPT`, `VXRUN_WORKFLOW` and `VXRUN_WORKFLOW_UID` expose
identity to the program. Each Argo worker node reports `item-id` and `output-path`
parameters, visible in `vxrun batch status`; use node status to identify successful
attempts. Directory existence alone does not mean an attempt succeeded.

Standard mounts include `/output`, `/inputs`, `/checkpoints`, `/dev/shm`, internal
CA trust, ClearML/S3 credentials and optional AWS/vxdata credentials; there are
no NFS or hostPath mounts, so read inputs from S3/vxdata or declare them. `/cache`
always exists: `local_cache: "0"` uses Pod-local `emptyDir`, while a nonzero value
selects an ephemeral TopoLVM volume. `VXS3_CACHE_DIR` points to `/cache/.vxs3`.
Both cache modes are disposable; outputs that must survive go under `/output`.
Path separation is an organization convention, not an access-control boundary.
Save output references before Workflow status expires under the cluster TTL.

Directory variables follow `VXRUN_<NAME>_DIR`: `VXRUN_OUTPUT_DIR`, `VXRUN_INPUTS_DIR`,
`VXRUN_CHECKPOINTS_DIR` and `VXRUN_CACHE_DIR`. `KUBERUN_OUTPUT` and `KUBERUN_CACHE`
are legacy aliases with the same values; `review_job` reports them at low
severity, and they will be removed once no worker code reads them.

## Publication

With the template's `publish` parameter on (its default), two more commands run
inside the cluster from the tools image; they are not meant for a laptop. After
each item's command, the `pack` step runs:

```sh
vxrun batch pack --source /batch-output/<user>/batches/<wf>-<uid>/items/<id>/attempts/<pod> \
  --destination s3://workflow-output/<user>/<wf>-<uid>/items/<id> \
  --scratch /cache/pack --manifest-out /tmp/manifest.json
```

It packs the directory into tar shards under `--scratch`, publishes them to the
exact destination with read-back verification, writes the item's
`manifest.parquet` last, and prints `{manifest_uri, sha256, files, bytes}`. The
destination must be empty; a retry after a partial upload fails and the item is
reported as failed by finalisation, since the writer identity cannot delete.

The Workflow's exit handler then runs:

```sh
vxrun batch finalise --prefix s3://workflow-output/<user>/<wf>-<uid> \
  --items-env VXRUN_ITEMS --workflow-status Succeeded --scratch /scratch [--register --title NAME]
```

It writes `workflow.json` (name, uid, user, status, expected item ids), then
reads every expected item's manifest and writes `items.parquet` (one row per
item: `item`, `status`, `manifest_uri`, `manifest_sha256`, `files`, `bytes`,
`detail`); an item without a complete manifest is `failed`. Only when every item
succeeded does it merge the item manifests with vxs3 and publish the root
`manifest.parquet` last, so the root's presence means the whole batch is there,
like a multipart upload that becomes an object only on completion. A batch with
failures is never completed: it keeps `items.parquet` and the successful items'
bundles for inspection and re-runs, and gets no root manifest and no catalog
entry. The root is a composed vxs3 bundle: `items/<id>/...` lists an item's
files, shards stay where the items published them, and each item's own manifest
remains a complete bundle. Re-running finalisation is safe: objects already
written with identical content are kept, differing ones are an error. With
`--register`, a complete root is registered as a vxdata Bundle with
`items.parquet` as its records table, using `VXDATA_TOKEN` from the environment.

Credentials come from `AWS_PROFILE` (default `workflow-output`) in
`AWS_SHARED_CREDENTIALS_FILE`; the endpoint from `VXRUN_S3_ENDPOINT`, the
profile's `endpoint_url`, or the cluster default. Reading a composed bundle needs
vxs3 3.1 or later.

## Collecting results

```sh
vxrun batch results <workflow-name>                     # prefix, completion, one line per item
vxrun batch results <workflow-name> --json
vxrun batch results <workflow-name> --download ./out    # complete batch: ./out/items/<id>/...
vxrun batch results <workflow-name> --download ./out --item trial-0   # one item, works for incomplete batches
```

`results` derives `s3://workflow-output/<user>/<name>-<uid>` from the live
Workflow (uid, creator, expected items) and lists that prefix once. Workflows
are garbage-collected days after they finish while the data stays, so when the
Workflow is gone it finds `<user>/<name>-<uid>/` in the bucket instead
(`--user` to look under someone else's area; an ambiguous reused name lists the
candidates), and the prefix itself is accepted in place of the name. Finalise
writes `workflow.json` (name, uid, user, status, expected items) into the prefix
first, so a finalised batch describes itself. Before the exit handler has run
it reports each item as `published` or `pending`; after it, the rows of
`items.parquet`; `complete` is whether the root manifest exists, and its
SHA-256 is reported and pinned on download.
Credentials come from the `workflow-output` profile (`--profile` or
`AWS_PROFILE` to override) in `AWS_SHARED_CREDENTIALS_FILE` or
`~/.aws/credentials`; Coder workspaces and job containers carry the read-only
keys under that name. For Python, open `manifest_uri` values with
`vxs3.bundle.io.open_bundle`, or use `client.bundles` when the batch was
registered.

Source: https://docs.virdx.dev/vxrun/batches/index.mdx
