Skip to content
VIRDX

Native Argo workflows

Use a saved Argo Workflow to run a pipeline without an agent scheduling each step. vxrun.dag supplies small Python builders, vxrun workflow prepares and submits the manifest, and Argo owns execution once submission succeeds.

For independent items sharing one image and resource allocation, use vxrun batch. It submits a native reference to the shared run-batch template and gives each item/attempt a separate /output directory.

Start with the runnable example

From a project with a Dockerfile and .vxrun.yaml:

vxrun workflow init .
vxrun workflow plan workflow.yaml --json
vxrun workflow build workflow.yaml --output prepared.yaml --json
vxrun workflow submit prepared.yaml --run-id example-001 --dry-run --json
vxrun workflow submit prepared.yaml --run-id example-001 --preflight --json
vxrun workflow submit prepared.yaml --run-id example-001 --json
vxrun workflow status WORKFLOW_NAME --json

init creates workflow.yaml and workflow_job.py, refusing to overwrite files. The example prepares a bundle, processes four items with parallelism two, and aggregates their output references into a final bundle. Its synthetic result is 60. It also exercises checkpoint publication and restore. All steps are CPU by default. init --gpu requests a GPU for processing to demonstrate a mixed resource graph; the synthetic arithmetic itself does not use the GPU. Replace that worker with your real GPU code before spending GPU time on it.

The image must include the worker at its working directory and this version of vxrun (from the internal conda channel or a local development build). The worker must run with a Python environment containing vxrun/vxs3. If your image uses Pixi, set the task command to pixi run python workflow_job.py ... as appropriate. The container must also provide sh, cat and mkdir for the small bootstrap. Zarr operations additionally need the numpy/zarr versions supported by vxs3.

build uses the existing scoped commit/push/build policy once for all tasks that reference the project image. It writes a new manifest with an immutable image digest and source/build provenance. It does not launch the DAG. Inspect the project’s build configuration and pending changes before preparation. If the build is interrupted, prepare again; the existing build cache can reuse a successful image. Explicitly pinned per-task images need no project build.

If you already have a digest, render a ready manifest without building:

vxrun workflow render workflow.yaml --image REGISTRY/IMAGE@sha256:DIGEST > prepared.yaml

plan, render and submit --dry-run are offline and do not authenticate or write files. --preflight contacts the Kubernetes API using server dry-run; it checks Workflow admission, not every future Pod, credential or data reference. The generated manifest also supports argo lint --offline prepared.yaml.

Author a DAG

Builders return ordinary dictionaries, which can be saved with yaml.safe_dump and edited as native Argo YAML. Commands are argv lists, not shell strings.

import yaml
from vxrun.dag import task, workflow

document = workflow(
    "training",
    [
        task(
            "prepare", ["python", "prepare.py"], outputs={"dataset": "/vxrun/outputs/result.json"}
        ),
        task(
            "train",
            ["python", "train.py", "{{inputs.parameters.dataset}}"],
            parameters={"dataset": "{{tasks.prepare.outputs.parameters.dataset}}"},
            dependencies=["prepare"],
            cpu="8",
            memory="32Gi",
            gpu=1,
            local_cache="64Gi",
            retries=3,
            outputs={"result": "/vxrun/outputs/result.json"},
        ),
    ],
    parallelism=2,
)
print(yaml.safe_dump(document, sort_keys=False))

items=[{"id": "a", ...}, ...] emits withItems; with_param emits Argo’s withParam. Bind item fields with parameters={"seed": "{{item.seed}}"} and refer to them in argv as {{inputs.parameters.seed}}. Item IDs must be unique, stable, path-safe identifiers. Dynamic producers should use write_items to validate uniqueness before handing the list to Argo. Tasks automatically receive separate item and attempt namespaces.

Native depends expressions and when are supported. Use depends consistently within one DAG; Argo does not mix it with dependencies. Consumers must declare a dependency path to every referenced producer. A skipped or failed producer may have no output; provide appropriate branch logic and aggregation behavior. workflow(..., fail_fast=False) lets independent branches continue; it does not make missing outputs available to downstream tasks.

Keep lists and references small. The helper caps manifests at 512 KB and output reference files at 64 KB. Batch large workloads into meaningful work units and use published indexes instead of placing millions of objects in Argo node status. Concurrency is bounded by workflow parallelism (1-25), then cluster quota and hardware. Resource previews show per-task requests, not guaranteed capacity.

Scratch and publication

Every managed task has these environment variables:

Variable Purpose
VXRUN_RUN_URI s3://scratch/<authenticated-user>/workflows/<workflow-name>-<workflow-uid>
VXRUN_TASK Logical task name
VXRUN_ITEM_KEY Stable item ID, or single
VXRUN_ATTEMPT Unique Pod name
VXRUN_OUTPUT_FILE /vxrun/outputs/result.json, for an Argo parameter
VXS3_CACHE_DIR Local staging/cache directory

Submission resolves the username through the authenticated Kubernetes identity, overriding the preview placeholder. Username prefixes organise the shared bucket; they are not access boundaries. The workflow UID isolates a new execution even if an older Workflow with the same name was garbage-collected.

<run-uri>/tasks/<task>/<item>/
  attempts/<pod-name>/outputs/<output-name>/<fresh-allocation>/...
  checkpoints/<pod-name>-<generation>/<fresh-allocation>/...
  checkpoints/completed/<pod-name>-<generation>.json
from vxrun.artifacts import TaskStorage, scratch_store, stage, write_output

context = TaskStorage.from_env()
store = scratch_store()
reference = context.publish("/cache/results", kind="bundle", store=store)
write_output(reference)  # expose only after successful verified publication
# Downstream: reference = json.loads(the_input_parameter)
local = stage(reference, "/cache/restored", store=store)

scratch_store() reads access keys from the mounted [scratch] credentials profile. Its endpoint defaults to https://s3.fra.virdx.dev when the profile has no endpoint_url. Override with the profile’s endpoint_url, then VXRUN_SCRATCH_ENDPOINT, or an explicit scratch_store(endpoint=...) argument (highest priority). Credential values never belong in the workflow manifest.

Use bundle for collections of ordinary files, zarr for native optimised arrays, and file for standalone large objects. Bundles use vxs3’s manifest-after-shards publication and digest pinning. Native Zarr stays native; its completion record is published after verification. Output allocations are fresh, and stage requires a new destination. Failed publication exposes no successful output reference. Some orphaned data may remain for scratch lifecycle cleanup.

Bundling/resharding needs local staging and verification space. Size local_cache for those operations, not just the original input. With local_cache="0", staging uses ephemeral Pod disk; use it only for small workloads. Pod-local storage is never a handoff between tasks. Helpers mount scratch credentials and the internal CA, merge it with the image’s public trust roots, and set fsGroup for writable volumes. Other credentials, legacy storage, ClearML and vxdata publication mounts must be configured explicitly in the native manifest when needed.

Checkpoints and retries

saved = context.latest_checkpoint(compatibility="model-config-data-v1", store=store)
if saved:
    checkpoint_file = stage(saved["artifact"], "/cache/resume", store=store)
    # Application loads weights, optimiser, RNG state and completed training step.

context.checkpoint(
    "/cache/checkpoint.pt", step=100, compatibility="model-config-data-v1", store=store
)

Checkpoint generations are immutable. A small completion record is written only after upload and verification. On startup, the helper scans at most 1000 completion records for this task/item and selects the largest numeric step matching the application’s compatibility key. Prune old generations deliberately or raise the bound; hitting it is an error, not a partial result. Missing or expired checkpoint bytes fail restoration. Different attempts share the checkpoint namespace but have separate output namespaces. Do not run simultaneous writers for one item ID.

Tasks default to two OnError retries with exponential backoff (10 s, then 20 s) and a twelve-hour per-attempt deadline. Do not add backoff.maxDuration: Argo measures it from the first attempt’s start and stops retrying once it passes. Nonzero application exits are not automatically retried by this default. Choose retry_policy="Always" or edit native retryStrategy/expressions when appropriate; checkpoint-aware preemptible applications generally need to retry interrupted attempts. Explicit per-template policies take precedence over cluster-injected template defaults. Retries cannot supply application checkpoint/resume or idempotent side effects.

Scratch objects expire after 14 days even while a workflow is active. Publish retained inputs/results to permanent vxdata storage before expiry; updating a small manifest does not renew its payload objects. Scratch is a temporary handoff, not a permanent run archive. Workflow status/output references also expire with the Workflow unless separately saved. Automatic final reports/notifications and retention-aware promotion are follow-up work.

Submission and recovery

submit requires a prepared manifest and stable --run-id. The Workflow name is derived from the authenticated identity and run ID. Repeating an identical request retrieves the existing Workflow; a different manifest, routing or owner fails. A lost response names the Workflow and instructs you to repeat the same request. This deduplication lasts while the Workflow object exists. After cluster TTL cleanup, the same run ID can create a new execution; retain the returned name/UID and use a new run ID for intentional reruns. The new UID prevents storage collisions. Recovery of the submission is distinct from retrying failed tasks.

vxdispatch does not accept native DAGs yet (kind: batch specs only), so a Workflow submitted here goes straight to Argo and the register records it as an observed run with less detail (read_doc("vx/cluster/dispatch") in vx mcp).

Scheduling is workflow-wide. Pass --team only for a team the user selected; team and preemptible modes cannot be combined. No per-step mode switching is implemented. Explicit CPU/memory/GPU requests may vary between tasks. Admission owns identity, queue and priority fields.

Hydra

The old kuberun add-hydra-launcher branch built once and submitted one Workflow per override set. It returned JobStatus.COMPLETED at submission time, with a URL instead of the task’s return value. This implementation carries forward image reuse and expanded override lists, while keeping completion semantics explicit.

from vxrun.dag import hydra_sweep

document = hydra_sweep(
    "training-sweep",
    ["python", "train.py"],
    [
        ["lr=0.01", "seed=1", "description=first trial"],
        ["lr=0.001", "seed=2", "description=second trial"],
    ],
    parallelism=2,
    initial_job_idx=0,
    gpu=1,
    cpu="8",
    memory="32Gi",
)

Each override remains one argv entry, including spaces, commas and quotes. Each job has its own task/checkpoint namespace, all jobs share the prepared image, and one failed trial does not stop independent trials. Pass expanded override sets; the helper does not implement Hydra’s sweep grammar, compose configs, serialise task functions, or require Hydra to be installed in vxrun. The target application still needs Hydra in its image. Omit launcher/sweeper options and --multirun so each container executes one trial.

This is not an installed hydra/launcher=vxrun plugin. A future launcher can use the same preparation/submission API, but must wait for real task completion and return real JobReturn values. Optuna/Nevergrad-style adaptive searches also need their search controller and metric feedback to survive the client session, for example by running that controller on the cluster. A static export alone does not provide that behavior. See Hydra’s launcher contract.

Current scope

Supported: flat DAGs with local container templates, static/dynamic fan-out, native conditions, per-task resources/retries, scratch publications/checkpoints, prepared images, offline plans, server preflight, submission recovery and task status/output references. Native manifests remain inspectable and editable.

The validator rejects nested DAGs, arbitrary remote template references, steps/script/resource templates, exit hooks and templateDefaults rather than silently skipping them. Those need separate composition and validation work. No new controller or WorkflowTemplate deployment is required for this first implementation; the existing scratch secret, CA ConfigMap, registry access and storage class are used. The separate run-batch workflow reference is supported through its explicit batch contract and does require that shared template to be deployed.

Navigation

Type to search…

↑↓ navigate↵ selectEsc close