Use a saved Argo Workflow to run a pipeline without an agent scheduling each
step. vxrun.dag supplies small Python builders, vxrun workflow prepares and
submits the manifest, and Argo owns execution once submission succeeds.
For independent items sharing one image and resource allocation, use
vxrun batch. It submits a native reference to the shared
run-batch template and gives each item/attempt a separate /output directory.
Start with the runnable example
From a project with a Dockerfile and .vxrun.yaml:
vxrun workflow init .
vxrun workflow plan workflow.yaml --json
vxrun workflow build workflow.yaml --output prepared.yaml --json
vxrun workflow submit prepared.yaml --run-id example-001 --dry-run --json
vxrun workflow submit prepared.yaml --run-id example-001 --preflight --json
vxrun workflow submit prepared.yaml --run-id example-001 --json
vxrun workflow status WORKFLOW_NAME --jsoninit creates workflow.yaml and workflow_job.py, refusing to overwrite files.
The example prepares a bundle, processes four items with parallelism two, and
aggregates their output references into a final bundle. Its synthetic result is
60. It also exercises checkpoint publication and restore. All steps are CPU by
default. init --gpu requests a GPU for processing to demonstrate a mixed
resource graph; the synthetic arithmetic itself does not use the GPU. Replace
that worker with your real GPU code before spending GPU time on it.
The image must include the worker at its working directory and this version of
vxrun (from the internal conda channel or a local development build). The worker
must run with a Python environment containing vxrun/vxs3. If your image uses
Pixi, set the task command to pixi run python workflow_job.py ... as appropriate.
The container must also provide sh, cat and mkdir for the small bootstrap.
Zarr operations additionally need the numpy/zarr versions supported by vxs3.
build uses the existing scoped commit/push/build policy once for all tasks that
reference the project image. It writes a new manifest with an immutable image
digest and source/build provenance. It does not launch the DAG. Inspect the
project’s build configuration and pending changes before preparation. If the
build is interrupted, prepare again; the existing build cache can reuse a
successful image. Explicitly pinned per-task images need no project build.
If you already have a digest, render a ready manifest without building:
vxrun workflow render workflow.yaml --image REGISTRY/IMAGE@sha256:DIGEST > prepared.yamlplan, render and submit --dry-run are offline and do not authenticate or
write files. --preflight contacts the Kubernetes API using server dry-run; it
checks Workflow admission, not every future Pod, credential or data reference.
The generated manifest also supports argo lint --offline prepared.yaml.
Author a DAG
Builders return ordinary dictionaries, which can be saved with yaml.safe_dump
and edited as native Argo YAML. Commands are argv lists, not shell strings.
import yaml
from vxrun.dag import task, workflow
document = workflow(
"training",
[
task(
"prepare", ["python", "prepare.py"], outputs={"dataset": "/vxrun/outputs/result.json"}
),
task(
"train",
["python", "train.py", "{{inputs.parameters.dataset}}"],
parameters={"dataset": "{{tasks.prepare.outputs.parameters.dataset}}"},
dependencies=["prepare"],
cpu="8",
memory="32Gi",
gpu=1,
local_cache="64Gi",
retries=3,
outputs={"result": "/vxrun/outputs/result.json"},
),
],
parallelism=2,
)
print(yaml.safe_dump(document, sort_keys=False))items=[{"id": "a", ...}, ...] emits withItems; with_param emits Argo’s
withParam. Bind item fields with parameters={"seed": "{{item.seed}}"} and
refer to them in argv as {{inputs.parameters.seed}}. Item IDs must be unique,
stable, path-safe identifiers. Dynamic producers should use write_items to
validate uniqueness before handing the list to Argo. Tasks automatically receive
separate item and attempt namespaces.
Native depends expressions and when are supported. Use depends consistently
within one DAG; Argo does not mix it with dependencies. Consumers must declare
a dependency path to every referenced producer. A skipped or failed producer may
have no output; provide appropriate branch logic and aggregation behavior.
workflow(..., fail_fast=False) lets independent branches continue; it does not
make missing outputs available to downstream tasks.
Keep lists and references small. The helper caps manifests at 512 KB and output
reference files at 64 KB. Batch large workloads into meaningful work units and
use published indexes instead of placing millions of objects in Argo node status.
Concurrency is bounded by workflow parallelism (1-25), then cluster quota and
hardware. Resource previews show per-task requests, not guaranteed capacity.
Scratch and publication
Every managed task has these environment variables:
| Variable | Purpose |
|---|---|
VXRUN_RUN_URI |
s3://scratch/<authenticated-user>/workflows/<workflow-name>-<workflow-uid> |
VXRUN_TASK |
Logical task name |
VXRUN_ITEM_KEY |
Stable item ID, or single |
VXRUN_ATTEMPT |
Unique Pod name |
VXRUN_OUTPUT_FILE |
/vxrun/outputs/result.json, for an Argo parameter |
VXS3_CACHE_DIR |
Local staging/cache directory |
Submission resolves the username through the authenticated Kubernetes identity, overriding the preview placeholder. Username prefixes organise the shared bucket; they are not access boundaries. The workflow UID isolates a new execution even if an older Workflow with the same name was garbage-collected.
<run-uri>/tasks/<task>/<item>/
attempts/<pod-name>/outputs/<output-name>/<fresh-allocation>/...
checkpoints/<pod-name>-<generation>/<fresh-allocation>/...
checkpoints/completed/<pod-name>-<generation>.jsonfrom vxrun.artifacts import TaskStorage, scratch_store, stage, write_output
context = TaskStorage.from_env()
store = scratch_store()
reference = context.publish("/cache/results", kind="bundle", store=store)
write_output(reference) # expose only after successful verified publication
# Downstream: reference = json.loads(the_input_parameter)
local = stage(reference, "/cache/restored", store=store)scratch_store() reads access keys from the mounted [scratch] credentials
profile. Its endpoint defaults to https://s3.fra.virdx.dev when the profile has
no endpoint_url. Override with the profile’s endpoint_url, then
VXRUN_SCRATCH_ENDPOINT, or an explicit scratch_store(endpoint=...) argument
(highest priority). Credential values never belong in the workflow manifest.
Use bundle for collections of ordinary files, zarr for native optimised arrays,
and file for standalone large objects. Bundles use vxs3’s manifest-after-shards
publication and digest pinning. Native Zarr stays native; its completion record
is published after verification. Output allocations are fresh, and stage
requires a new destination. Failed publication exposes no successful output
reference. Some orphaned data may remain for scratch lifecycle cleanup.
Bundling/resharding needs local staging and verification space. Size local_cache
for those operations, not just the original input. With local_cache="0", staging
uses ephemeral Pod disk; use it only for small workloads. Pod-local storage is
never a handoff between tasks. Helpers mount scratch credentials and the internal
CA, merge it with the image’s public trust roots, and set fsGroup for writable
volumes. Other credentials, legacy storage, ClearML and vxdata publication mounts
must be configured explicitly in the native manifest when needed.
Checkpoints and retries
saved = context.latest_checkpoint(compatibility="model-config-data-v1", store=store)
if saved:
checkpoint_file = stage(saved["artifact"], "/cache/resume", store=store)
# Application loads weights, optimiser, RNG state and completed training step.
context.checkpoint(
"/cache/checkpoint.pt", step=100, compatibility="model-config-data-v1", store=store
)Checkpoint generations are immutable. A small completion record is written only after upload and verification. On startup, the helper scans at most 1000 completion records for this task/item and selects the largest numeric step matching the application’s compatibility key. Prune old generations deliberately or raise the bound; hitting it is an error, not a partial result. Missing or expired checkpoint bytes fail restoration. Different attempts share the checkpoint namespace but have separate output namespaces. Do not run simultaneous writers for one item ID.
Tasks default to two OnError retries with exponential backoff (10 s, then 20 s)
and a twelve-hour per-attempt deadline. Do not add backoff.maxDuration: Argo
measures it from the first attempt’s start and stops retrying once it passes. Nonzero application exits
are not automatically retried by this default. Choose retry_policy="Always" or
edit native retryStrategy/expressions when appropriate; checkpoint-aware
preemptible applications generally need to retry interrupted attempts. Explicit
per-template policies take precedence over cluster-injected template defaults.
Retries cannot supply application checkpoint/resume or idempotent side effects.
Scratch objects expire after 14 days even while a workflow is active. Publish retained inputs/results to permanent vxdata storage before expiry; updating a small manifest does not renew its payload objects. Scratch is a temporary handoff, not a permanent run archive. Workflow status/output references also expire with the Workflow unless separately saved. Automatic final reports/notifications and retention-aware promotion are follow-up work.
Submission and recovery
submit requires a prepared manifest and stable --run-id. The Workflow name is
derived from the authenticated identity and run ID. Repeating an identical
request retrieves the existing Workflow; a different manifest, routing or owner
fails. A lost response names the Workflow and instructs you to repeat the same
request. This deduplication lasts while the Workflow object exists. After cluster
TTL cleanup, the same run ID can create a new execution; retain the returned
name/UID and use a new run ID for intentional reruns. The new UID prevents storage
collisions. Recovery of the submission is distinct from retrying failed tasks.
vxdispatch does not accept native DAGs yet (kind: batch specs only), so a
Workflow submitted here goes straight to Argo and the register records it as an
observed run with less detail (read_doc("vx/cluster/dispatch") in vx mcp).
Scheduling is workflow-wide. Pass --team only for a team the user selected;
team and preemptible modes cannot be combined. No per-step mode switching is
implemented. Explicit CPU/memory/GPU requests may vary between tasks. Admission
owns identity, queue and priority fields.
Hydra
The old kuberun add-hydra-launcher branch built once and submitted one Workflow
per override set. It returned JobStatus.COMPLETED at submission time, with a URL
instead of the task’s return value. This implementation carries forward image reuse
and expanded override lists, while keeping completion semantics explicit.
from vxrun.dag import hydra_sweep
document = hydra_sweep(
"training-sweep",
["python", "train.py"],
[
["lr=0.01", "seed=1", "description=first trial"],
["lr=0.001", "seed=2", "description=second trial"],
],
parallelism=2,
initial_job_idx=0,
gpu=1,
cpu="8",
memory="32Gi",
)Each override remains one argv entry, including spaces, commas and quotes. Each
job has its own task/checkpoint namespace, all jobs share the prepared image, and
one failed trial does not stop independent trials. Pass expanded override sets;
the helper does not implement Hydra’s sweep grammar, compose configs, serialise
task functions, or require Hydra to be installed in vxrun. The target application
still needs Hydra in its image. Omit launcher/sweeper options and --multirun so
each container executes one trial.
This is not an installed hydra/launcher=vxrun plugin. A future launcher can use
the same preparation/submission API, but must wait for real task completion and
return real JobReturn values. Optuna/Nevergrad-style adaptive searches also need
their search controller and metric feedback to survive the client session, for
example by running that controller on the cluster. A static export alone does
not provide that behavior. See Hydra’s launcher contract.
Current scope
Supported: flat DAGs with local container templates, static/dynamic fan-out, native conditions, per-task resources/retries, scratch publications/checkpoints, prepared images, offline plans, server preflight, submission recovery and task status/output references. Native manifests remain inspectable and editable.
The validator rejects nested DAGs, arbitrary remote template references, steps/script/resource
templates, exit hooks and templateDefaults rather than silently skipping them.
Those need separate composition and validation work. No new controller or
WorkflowTemplate deployment is required for this first implementation; the
existing scratch secret, CA ConfigMap, registry access and storage class are used.
The separate run-batch workflow reference is supported through its explicit
batch contract and does require that shared template to be deployed.