vxdispatch is the one door for finite batch jobs on the Virdx clusters: the
on-prem cluster (fra) and, in testing, AWS (aws). You hand it a run spec
(the same one-file format read_doc("vxrun/specs") describes), it verifies the spec,
decides the cluster, submits the run-batch Workflow as you, and keeps the
record of the run for good.
- UI: https://dispatch.fra.virdx.dev (runs, builds, images, instances, the review queue for admins).
- API: https://dispatch.fra.virdx.dev/api (FastAPI; the endpoints used below are listed at the end).
- Design and deployment:
infra_k8s/docs/vxdispatch.md. Service source:mono/apps/vxdispatch-api(README has the full endpoint and settings reference) andmono/apps/vxdispatch-ui.
Why go through dispatch, also for ordinary FRA runs
Argo deletes a finished Workflow after three days, and with it most of what you
would want to know later. Dispatch keeps the record: the spec as submitted,
the image digest and the commit it was built from, every pod with its host,
attempt, exit reason and the GPU, CPU and memory it was granted and what it
actually used, the outcome, and where the outputs landed
(s3://workflow-output/<user>/<run_id>-<uid>/). Runs are created as you, on
your own queue, so attribution to you and your team is exact and the usage
report (GET /api/reports/usage) adds up per user, team or cluster. Everyone
gets the same view of status, events, logs and stop in the UI and the API,
without kubectl. Every finished run gets an AWS cost estimate and a cloud
compatibility card: eight checks that say what would stop the same job running
in the cloud and how to fix it, so a job can be made cloud-ready before anyone
needs it to be. And cloud runs themselves (approval, image and bundle
replication, output collection) only exist through dispatch. A run submitted
straight to Argo with vxrun/kuberun is still recorded, but as a second-hand
observation with less in it (see the last section).
Use dispatch when the job is a finite list of independent commands on one image
with its data declared as bundles or staged inside the image. Keep the direct
path (build_and_submit_vxrun, submit_run, vxrun batch submit) for what dispatch
does not take yet: native Argo DAGs (kind: workflow), run-image jobs of a
repository without a spec, and anything that still reads /mnt/* at run time.
The spec
Dispatch accepts the run spec and nothing else: apiVersion: vxrun.virdx.dev/v1alpha1, kind: batch. Read read_doc("vxrun/specs") for the full form
(bundles, input selections, resolve-before-size). The model lives in the shared
spec package (mono/packages/vxspec); vxrun, vx and the dispatch API
all validate with the same rules, so a spec that passes validate_run or
vxrun spec validate passes dispatch’s own verification unless the cluster
rules (team membership, quota) reject it.
A minimal batch spec, with the interruption waiver:
apiVersion: vxrun.virdx.dev/v1alpha1
kind: batch
name: histo-qc-sweep
description: QC thresholds, one item per setting.
image:
source: github.com/virdx/histo@<full 40-character commit sha>
dockerfile: Dockerfile
context: .
ref: zot.fra.virdx.dev:5000/histo@sha256:<digest> # from build_image
resources:
cpu: "4"
memory: 16Gi
gpu: 1
local_cache: 32Gi
execution:
parallelism: 4
retries: 2
task_deadline: 7200
# The author's waiver: every item can be killed at any point and restarted,
# from scratch or from its own checkpoints, without harming the result.
# Preemptible scheduling implies it; spot capacity in the cloud needs it.
interruptible: true
scheduling:
preemptible: true
items:
- id: thr-05
argv: [python, -m, histo.qc, --threshold, "0.5", --out, /output]
- id: thr-07
argv: [python, -m, histo.qc, --threshold, "0.7", --out, /output]
outputs:
publish: trueTwo things the spec needs before it can be posted:
image.refmust be a digest.build_imagebuilds a pushed commit and returns it; from a local checkout,vx run workflow builddoes the same. A digest of an earlier build of the same commit is equally good;GET /api/images?repository=<repo>&commit=<sha>lists them.- Declared inputs must be pinned (
manifest+sha256);vx run spec resolvedoes that and sizes them solocal_cachecan be set fromstorage.minimum_local_cache.
execution.interruptible arrived with the shared spec package. A vxrun older
than the release that ships it rejects the key as unknown in vxrun spec validate and validate_run (the spec model forbids unknown fields) while the
dispatch API accepts it. If that happens, update vx (pixi global update vx), or
validate without the key and add it before posting. Declare it only when it is
true: it is a promise about the program, not a scheduling flag.
Submit
Authentication is a Keycloak bearer token whose audience includes dispatch.
The token kubectl already holds for the cluster carries it, and so does the
token of a Coder workspace, so nothing new is issued:
TOKEN=$(kubectl oidc-login get-token \
--oidc-issuer-url=https://keycloak.fra.virdx.dev/realms/virdx \
--oidc-client-id=k3s | python3 -c 'import json,sys; print(json.load(sys.stdin)["status"]["token"])')
API=https://dispatch.fra.virdx.dev/api
curl -sf "$API/me" -H "Authorization: Bearer $TOKEN" # username, team, roleNever write the token into a file, a spec or a report; it expires anyway.
Validate first, then post the spec text with a target and a stable run_id:
vxrun spec validate run.yaml # exit 1 while something blocks
python3 - run.yaml fra qc-sweep-001 > body.json <<'PY'
import json, sys
spec, target, run_id = sys.argv[1:4]
print(json.dumps({"target": target, "run_id": run_id, "spec": open(spec).read()}))
PY
curl -sf "$API/runs" -H "Authorization: Bearer $TOKEN" \
-H 'Content-Type: application/json' -d @body.jsontarget is fra, aws or auto (auto resolves on the server; today that
means fra unless the run was made for the cloud). run_id is also the
Workflow name; posting the same run_id again returns the existing run (HTTP
200 instead of 201) rather than starting a second one, and a run_id owned by
someone else is a 409. The response is the run with its first state:
submitted (on the cluster), pending-approval (cloud target, see below), or
rejected / failed with error listing the reasons; fix the spec and
resubmit under the same run_id.
Then poll and act on the run by its id:
| call | what it gives |
|---|---|
GET /api/runs/{run_id} |
state, cluster, progress (items total / succeeded / failed / running / pending, workers running) once submitted, forecast while pending approval, output_uri and output_state when finished, can_stop, requires_approval |
GET /api/runs/{run_id}/events |
every state transition with time, actor and detail (an admin’s decision carries detail.comment) |
GET /api/runs/{run_id}/logs |
streamed from Argo while running; after that from the log store (14 days) as <timestamp> [<pod>] <message>; ?container=all for sidecars; 409 with the reason when there is nothing to read |
POST /api/runs/{run_id}/stop |
terminates the Workflow (owner or admin); the run ends stopped |
GET /api/runs |
your runs; filters state, cluster, team, origin, cloud; ?all=true for admins |
Poll about every 30 seconds, not faster; states move through submitted -> running -> succeeded | failed | stopped. The UI shows the same run at
https://dispatch.fra.virdx.dev with the events, logs and the cards below.
get_run / get_run_logs still work on the Workflow by name (the run_id)
while it exists on the cluster.
What you get back
Once the run finishes, dispatch fills in the record; nothing in it is guessed
(a value with no basis is null):
GET .../metrics: duration, items counted and attempts, and the pod totals:gpu_seconds,cpu_seconds,memory_mb_secondsgranted,node_hours,preempted_pods,oomkilled_pods, plus what was used, sampled from Prometheus after each pod ended:gpu_util_avg,cpu_used_seconds,mem_max_bytes. Right-size the next spec from these.GET .../pods: every attempt of every pod with template, host, GPU model, phase, exit reason (oomkilled,preempted,deadline), granted and used resources.GET .../costand.../costs: the AWS estimate in USD (spot and on-demand, for the instance type the run maps to) and, where a rate card exists, the on-prem estimate in EUR.GET .../cloud: the cloud compatibility card (next section), also stored on the run ascloud_verdict.GET .../forecast(while a cloud run waits for approval): data size, items, estimated node hours and wall time from the median of earlier succeeded runs of the same spec name and image repository, and the spot / on-demand price that implies.- Outputs:
output_uriis theworkflow-outputprefix, whichever cluster ran the job; AWS results are pulled back (output_state: collecting -> local). Collect them as for any batch (vxrun batch results,read_doc("vx/cluster/storage-practices")“Batch outputs”). GET /api/reports/usage?group=user|team|cluster&since=&until=: runs by outcome, GPU, CPU and wall hours, preempted and OOM-killed pods, AWS and on-prem estimates, per group. Admins see everyone; you see your own.
The cloud target
You may submit with target: aws (or auto). The cloud path is in testing:
every AWS run stops at pending-approval and waits for an admin, and most will
be rejected for now while the path is being proven. Nobody, agent or person,
can promise an AWS run will be approved; do not plan around one. A pending run
expires after seven days without a decision. The reason for a rejection is in
the decision’s comment (GET .../events, detail.comment) and in the cloud
card, so each attempt tells you what to change.
What you can do today is make a run cloud-ready, which the card measures with
eight checks (GET /api/runs/{run_id}/cloud; each pass, warn, fail or
skip with a one-line reason, verdict fail if any check fails):
| check | passes when | what to do otherwise |
|---|---|---|
dispatch |
the run came through dispatch | submit through dispatch; direct Argo runs always fail this one |
instance |
a tracked EC2 type carries the GPU class and count (or is CPU-only for gpu: 0) and some zone offers it now |
pick a GPU class and count that map to a tracked type (GET /api/instances) |
fit |
that type covers the worker’s CPU and memory | lower resources.cpu / memory to a tracked shape |
data |
no inputs declared and no /mnt/ path in any command |
fails by design for now, see below |
image |
the image digest is in the registry inventory | pin image.ref to a digest the registry holds (a warn clears once the inventory has seen it) |
outputs |
outputs.publish: true |
publish; /output on NFS does not exist in the cloud |
local_cache |
none, or at most the NVMe every tracked type has (200 GiB) | shrink local_cache or accept the warning |
interruption |
execution.interruptible: true (preemptible scheduling implies it) |
make items restart cleanly and declare the waiver; spot capacity needs it. Fails outright when spot risk for the type is high, warns otherwise; retried pods of earlier runs of the same image are quoted as evidence |
Be clear with users about data: a run that declares inputs currently fails
this check by design, because bundle replication to the cloud has not been
verified end to end. The dataset gate rejects any declared input on aws
after approval. Until that changes, only runs whose data is inside the image or
that read no data can pass; declaring inputs is still right for FRA runs and
for the record.
A cloud run that is approved goes approved -> replicating (the image is
copied to the cloud registry; GET .../replications) -> submitted, runs
there, and its outputs are pulled back into workflow-output. The same
run_id, events, logs and stop apply.
Runs that bypass dispatch
Dispatch watches the clusters, so a run-batch or run-image Workflow
submitted directly (vxrun, kuberun, build_and_submit_vxrun, submit_run) is
adopted as an observed run (origin: observed) under the user Argo recorded,
with the commit and build where vxrun put them on the Workflow, the image, and
a spec reconstructed from the Workflow’s arguments. From then on it is followed
like any other: metrics and pods when it finishes, the usage report, the
estimates. But it is a thinner record: no declared inputs, no commit for
kuberun submissions, no outputs located, can_stop false (vxrun owns the
Workflow, POST .../stop answers 409; use cancel_run), and the cloud card
fails the dispatch and data checks every time. A Workflow that was never
observed while it existed is lost to the register.
Endpoint summary
All under https://dispatch.fra.virdx.dev/api, bearer token required:
GET /me; POST /runs; GET /runs, /runs/{run_id}, .../events,
.../logs, .../metrics, .../pods, .../cost, .../costs, .../cloud,
.../forecast, .../replications; POST /runs/{run_id}/stop,
.../collect (retry a failed output pull); GET /images, /builds,
/instances, /overview, /reports/usage, /reports/capacity. Admin only:
POST /runs/{run_id}/approve|reject. The API README in
mono/apps/vxdispatch-api is the authoritative list, with response fields.