---
title: "Native Argo workflows"
image: "https://docs.virdx.dev/img/virdx-social-card.png"
---

> Documentation Index
> Fetch the complete documentation index at: https://docs.virdx.dev/llms.txt
> Use this file to discover all available pages before exploring further.

# Native Argo workflows

Use a saved Argo Workflow to run a pipeline without an agent scheduling each
step. `vxrun.dag` supplies small Python builders, `vxrun workflow` prepares and
submits the manifest, and Argo owns execution once submission succeeds.

For independent items sharing one image and resource allocation, use
[`vxrun batch`](batches.md). It submits a native reference to the shared
`run-batch` template and gives each item/attempt a separate `/output` directory.

## Start with the runnable example

From a project with a Dockerfile and `.vxrun.yaml`:

```sh
vxrun workflow init .
vxrun workflow plan workflow.yaml --json
vxrun workflow build workflow.yaml --output prepared.yaml --json
vxrun workflow submit prepared.yaml --run-id example-001 --dry-run --json
vxrun workflow submit prepared.yaml --run-id example-001 --preflight --json
vxrun workflow submit prepared.yaml --run-id example-001 --json
vxrun workflow status WORKFLOW_NAME --json
```

`init` creates `workflow.yaml` and `workflow_job.py`, refusing to overwrite files.
The example prepares a bundle, processes four items with parallelism two, and
aggregates their output references into a final bundle. Its synthetic result is
60. It also exercises checkpoint publication and restore. All steps are CPU by
default. `init --gpu` requests a GPU for processing to demonstrate a mixed
resource graph; the synthetic arithmetic itself does not use the GPU. Replace
that worker with your real GPU code before spending GPU time on it.

The image must include the worker at its working directory and this version of
vxrun (from the internal conda channel or a local development build). The worker
must run with a Python environment containing vxrun/vxs3. If your image uses
Pixi, set the task command to `pixi run python workflow_job.py ...` as appropriate.
The container must also provide `sh`, `cat` and `mkdir` for the small bootstrap.
Zarr operations additionally need the numpy/zarr versions supported by vxs3.

`build` uses the existing scoped commit/push/build policy once for all tasks that
reference the project image. It writes a new manifest with an immutable image
digest and source/build provenance. It does not launch the DAG. Inspect the
project's build configuration and pending changes before preparation. If the
build is interrupted, prepare again; the existing build cache can reuse a
successful image. Explicitly pinned per-task images need no project build.

If you already have a digest, render a ready manifest without building:

```sh
vxrun workflow render workflow.yaml --image REGISTRY/IMAGE@sha256:DIGEST > prepared.yaml
```

`plan`, `render` and `submit --dry-run` are offline and do not authenticate or
write files. `--preflight` contacts the Kubernetes API using server dry-run; it
checks Workflow admission, not every future Pod, credential or data reference.
The generated manifest also supports `argo lint --offline prepared.yaml`.

## Author a DAG

Builders return ordinary dictionaries, which can be saved with `yaml.safe_dump`
and edited as native Argo YAML. Commands are argv lists, not shell strings.

```python
import yaml
from vxrun.dag import task, workflow

document = workflow(
"training",
[
    task(
        "prepare", ["python", "prepare.py"], outputs={"dataset": "/vxrun/outputs/result.json"}
    ),
    task(
        "train",
        ["python", "train.py", "{{inputs.parameters.dataset}}"],
        parameters={"dataset": "{{tasks.prepare.outputs.parameters.dataset}}"},
        dependencies=["prepare"],
        cpu="8",
        memory="32Gi",
        gpu=1,
        local_cache="64Gi",
        retries=3,
        outputs={"result": "/vxrun/outputs/result.json"},
    ),
],
parallelism=2,
)
print(yaml.safe_dump(document, sort_keys=False))
```

`items=[{"id": "a", ...}, ...]` emits `withItems`; `with_param` emits Argo's
`withParam`. Bind item fields with `parameters={"seed": "{{item.seed}}"}` and
refer to them in argv as `{{inputs.parameters.seed}}`. Item IDs must be unique,
stable, path-safe identifiers. Dynamic producers should use `write_items` to
validate uniqueness before handing the list to Argo. Tasks automatically receive
separate item and attempt namespaces.

Native `depends` expressions and `when` are supported. Use `depends` consistently
within one DAG; Argo does not mix it with `dependencies`. Consumers must declare
a dependency path to every referenced producer. A skipped or failed producer may
have no output; provide appropriate branch logic and aggregation behavior.
`workflow(..., fail_fast=False)` lets independent branches continue; it does not
make missing outputs available to downstream tasks.

Keep lists and references small. The helper caps manifests at 512 KB and output
reference files at 64 KB. Batch large workloads into meaningful work units and
use published indexes instead of placing millions of objects in Argo node status.
Concurrency is bounded by workflow `parallelism` (1-25), then cluster quota and
hardware. Resource previews show per-task requests, not guaranteed capacity.

## Scratch and publication

Every managed task has these environment variables:

| Variable | Purpose |
|---|---|
| `VXRUN_RUN_URI` | `s3://scratch/<authenticated-user>/workflows/<workflow-name>-<workflow-uid>` |
| `VXRUN_TASK` | Logical task name |
| `VXRUN_ITEM_KEY` | Stable item ID, or `single` |
| `VXRUN_ATTEMPT` | Unique Pod name |
| `VXRUN_OUTPUT_FILE` | `/vxrun/outputs/result.json`, for an Argo parameter |
| `VXS3_CACHE_DIR` | Local staging/cache directory |

Submission resolves the username through the authenticated Kubernetes identity,
overriding the preview placeholder. Username prefixes organise the shared bucket;
they are not access boundaries. The workflow UID isolates a new execution even
if an older Workflow with the same name was garbage-collected.

```text
<run-uri>/tasks/<task>/<item>/
  attempts/<pod-name>/outputs/<output-name>/<fresh-allocation>/...
  checkpoints/<pod-name>-<generation>/<fresh-allocation>/...
  checkpoints/completed/<pod-name>-<generation>.json
```

```python
from vxrun.artifacts import TaskStorage, scratch_store, stage, write_output

context = TaskStorage.from_env()
store = scratch_store()
reference = context.publish("/cache/results", kind="bundle", store=store)
write_output(reference)  # expose only after successful verified publication
# Downstream: reference = json.loads(the_input_parameter)
local = stage(reference, "/cache/restored", store=store)
```

`scratch_store()` reads access keys from the mounted `[scratch]` credentials
profile. Its endpoint defaults to `https://s3.fra.virdx.dev` when the profile has
no `endpoint_url`. Override with the profile's `endpoint_url`, then
`VXRUN_SCRATCH_ENDPOINT`, or an explicit `scratch_store(endpoint=...)` argument
(highest priority). Credential values never belong in the workflow manifest.

Use `bundle` for collections of ordinary files, `zarr` for native optimised arrays,
and `file` for standalone large objects. Bundles use vxs3's manifest-after-shards
publication and digest pinning. Native Zarr stays native; its completion record
is published after verification. Output allocations are fresh, and `stage`
requires a new destination. Failed publication exposes no successful output
reference. Some orphaned data may remain for scratch lifecycle cleanup.

Bundling/resharding needs local staging and verification space. Size `local_cache`
for those operations, not just the original input. With `local_cache="0"`, staging
uses ephemeral Pod disk; use it only for small workloads. Pod-local storage is
never a handoff between tasks. Helpers mount scratch credentials and the internal
CA, merge it with the image's public trust roots, and set fsGroup for writable
volumes. Other credentials, legacy storage, ClearML and vxdata publication mounts
must be configured explicitly in the native manifest when needed.

## Checkpoints and retries

```python
saved = context.latest_checkpoint(compatibility="model-config-data-v1", store=store)
if saved:
checkpoint_file = stage(saved["artifact"], "/cache/resume", store=store)
# Application loads weights, optimiser, RNG state and completed training step.

context.checkpoint(
"/cache/checkpoint.pt", step=100, compatibility="model-config-data-v1", store=store
)
```

Checkpoint generations are immutable. A small completion record is written only
after upload and verification. On startup, the helper scans at most 1000 completion
records for this task/item and selects the largest numeric step matching the
application's compatibility key. Prune old generations deliberately or raise the
bound; hitting it is an error, not a partial result. Missing or expired checkpoint
bytes fail restoration. Different attempts share the checkpoint namespace but
have separate output namespaces. Do not run simultaneous writers for one item ID.

Tasks default to two `OnError` retries with exponential backoff (10 s, then 20 s)
and a twelve-hour per-attempt deadline. Do not add `backoff.maxDuration`: Argo
measures it from the first attempt's start and stops retrying once it passes. Nonzero application exits
are not automatically retried by this default. Choose `retry_policy="Always"` or
edit native `retryStrategy`/expressions when appropriate; checkpoint-aware
preemptible applications generally need to retry interrupted attempts. Explicit
per-template policies take precedence over cluster-injected template defaults.
Retries cannot supply application checkpoint/resume or idempotent side effects.

Scratch objects expire after 14 days even while a workflow is active. Publish
retained inputs/results to permanent vxdata storage before expiry; updating a
small manifest does not renew its payload objects. Scratch is a temporary handoff,
not a permanent run archive. Workflow status/output references also expire with
the Workflow unless separately saved. Automatic final reports/notifications and
retention-aware promotion are follow-up work.

## Submission and recovery

`submit` requires a prepared manifest and stable `--run-id`. The Workflow name is
derived from the authenticated identity and run ID. Repeating an identical
request retrieves the existing Workflow; a different manifest, routing or owner
fails. A lost response names the Workflow and instructs you to repeat the same
request. This deduplication lasts while the Workflow object exists. After cluster
TTL cleanup, the same run ID can create a new execution; retain the returned
name/UID and use a new run ID for intentional reruns. The new UID prevents storage
collisions. Recovery of the submission is distinct from retrying failed tasks.

vxdispatch does not accept native DAGs yet (`kind: batch` specs only), so a
Workflow submitted here goes straight to Argo and the register records it as an
observed run with less detail (`read_doc("vx/cluster/dispatch")` in vx mcp).

Scheduling is workflow-wide. Pass `--team` only for a team the user selected;
team and preemptible modes cannot be combined. No per-step mode switching is
implemented. Explicit CPU/memory/GPU requests may vary between tasks. Admission
owns identity, queue and priority fields.

## Hydra

The old kuberun `add-hydra-launcher` branch built once and submitted one Workflow
per override set. It returned `JobStatus.COMPLETED` at submission time, with a URL
instead of the task's return value. This implementation carries forward image reuse
and expanded override lists, while keeping completion semantics explicit.

```python
from vxrun.dag import hydra_sweep

document = hydra_sweep(
"training-sweep",
["python", "train.py"],
[
    ["lr=0.01", "seed=1", "description=first trial"],
    ["lr=0.001", "seed=2", "description=second trial"],
],
parallelism=2,
initial_job_idx=0,
gpu=1,
cpu="8",
memory="32Gi",
)
```

Each override remains one argv entry, including spaces, commas and quotes. Each
job has its own task/checkpoint namespace, all jobs share the prepared image, and
one failed trial does not stop independent trials. Pass expanded override sets;
the helper does not implement Hydra's sweep grammar, compose configs, serialise
task functions, or require Hydra to be installed in vxrun. The target application
still needs Hydra in its image. Omit launcher/sweeper options and `--multirun` so
each container executes one trial.

This is not an installed `hydra/launcher=vxrun` plugin. A future launcher can use
the same preparation/submission API, but must wait for real task completion and
return real JobReturn values. Optuna/Nevergrad-style adaptive searches also need
their search controller and metric feedback to survive the client session, for
example by running that controller on the cluster. A static export alone does
not provide that behavior. See Hydra's [launcher contract](https://hydra.cc/docs/1.3/advanced/plugins/overview/).

## Current scope

Supported: flat DAGs with local container templates, static/dynamic fan-out,
native conditions, per-task resources/retries, scratch publications/checkpoints,
prepared images, offline plans, server preflight, submission recovery and task
status/output references. Native manifests remain inspectable and editable.

The validator rejects nested DAGs, arbitrary remote template references, steps/script/resource
templates, exit hooks and templateDefaults rather than silently skipping them.
Those need separate composition and validation work. No new controller or
WorkflowTemplate deployment is required for this first implementation; the
existing scratch secret, CA ConfigMap, registry access and storage class are used.
The separate `run-batch` workflow reference is supported through its explicit
batch contract and does require that shared template to be deployed.

Source: https://docs.virdx.dev/vxrun/workflows/index.mdx
