Skip to content
VIRDX

Dispatch: the register and broker for cluster jobs

vxdispatch is the one door for finite batch jobs on the Virdx clusters: the on-prem cluster (fra) and, in testing, AWS (aws). You hand it a run spec (the same one-file format read_doc("vxrun/specs") describes), it verifies the spec, decides the cluster, submits the run-batch Workflow as you, and keeps the record of the run for good.

  • UI: https://dispatch.fra.virdx.dev (runs, builds, images, instances, the review queue for admins).
  • API: https://dispatch.fra.virdx.dev/api (FastAPI; the endpoints used below are listed at the end).
  • Design and deployment: infra_k8s/docs/vxdispatch.md. Service source: mono/apps/vxdispatch-api (README has the full endpoint and settings reference) and mono/apps/vxdispatch-ui.

Why go through dispatch, also for ordinary FRA runs

Argo deletes a finished Workflow after three days, and with it most of what you would want to know later. Dispatch keeps the record: the spec as submitted, the image digest and the commit it was built from, every pod with its host, attempt, exit reason and the GPU, CPU and memory it was granted and what it actually used, the outcome, and where the outputs landed (s3://workflow-output/<user>/<run_id>-<uid>/). Runs are created as you, on your own queue, so attribution to you and your team is exact and the usage report (GET /api/reports/usage) adds up per user, team or cluster. Everyone gets the same view of status, events, logs and stop in the UI and the API, without kubectl. Every finished run gets an AWS cost estimate and a cloud compatibility card: eight checks that say what would stop the same job running in the cloud and how to fix it, so a job can be made cloud-ready before anyone needs it to be. And cloud runs themselves (approval, image and bundle replication, output collection) only exist through dispatch. A run submitted straight to Argo with vxrun/kuberun is still recorded, but as a second-hand observation with less in it (see the last section).

Use dispatch when the job is a finite list of independent commands on one image with its data declared as bundles or staged inside the image. Keep the direct path (build_and_submit_vxrun, submit_run, vxrun batch submit) for what dispatch does not take yet: native Argo DAGs (kind: workflow), run-image jobs of a repository without a spec, and anything that still reads /mnt/* at run time.

The spec

Dispatch accepts the run spec and nothing else: apiVersion: vxrun.virdx.dev/v1alpha1, kind: batch. Read read_doc("vxrun/specs") for the full form (bundles, input selections, resolve-before-size). The model lives in the shared spec package (mono/packages/vxspec); vxrun, vx and the dispatch API all validate with the same rules, so a spec that passes validate_run or vxrun spec validate passes dispatch’s own verification unless the cluster rules (team membership, quota) reject it.

A minimal batch spec, with the interruption waiver:

apiVersion: vxrun.virdx.dev/v1alpha1
kind: batch
name: histo-qc-sweep
description: QC thresholds, one item per setting.

image:
  source: github.com/virdx/histo@<full 40-character commit sha>
  dockerfile: Dockerfile
  context: .
  ref: zot.fra.virdx.dev:5000/histo@sha256:<digest>   # from build_image

resources:
  cpu: "4"
  memory: 16Gi
  gpu: 1
  local_cache: 32Gi

execution:
  parallelism: 4
  retries: 2
  task_deadline: 7200
  # The author's waiver: every item can be killed at any point and restarted,
  # from scratch or from its own checkpoints, without harming the result.
  # Preemptible scheduling implies it; spot capacity in the cloud needs it.
  interruptible: true

scheduling:
  preemptible: true

items:
  - id: thr-05
    argv: [python, -m, histo.qc, --threshold, "0.5", --out, /output]
  - id: thr-07
    argv: [python, -m, histo.qc, --threshold, "0.7", --out, /output]

outputs:
  publish: true

Two things the spec needs before it can be posted:

  • image.ref must be a digest. build_image builds a pushed commit and returns it; from a local checkout, vx run workflow build does the same. A digest of an earlier build of the same commit is equally good; GET /api/images?repository=<repo>&commit=<sha> lists them.
  • Declared inputs must be pinned (manifest + sha256); vx run spec resolve does that and sizes them so local_cache can be set from storage.minimum_local_cache.

execution.interruptible arrived with the shared spec package. A vxrun older than the release that ships it rejects the key as unknown in vxrun spec validate and validate_run (the spec model forbids unknown fields) while the dispatch API accepts it. If that happens, update vx (pixi global update vx), or validate without the key and add it before posting. Declare it only when it is true: it is a promise about the program, not a scheduling flag.

Submit

Authentication is a Keycloak bearer token whose audience includes dispatch. The token kubectl already holds for the cluster carries it, and so does the token of a Coder workspace, so nothing new is issued:

TOKEN=$(kubectl oidc-login get-token \
  --oidc-issuer-url=https://keycloak.fra.virdx.dev/realms/virdx \
  --oidc-client-id=k3s | python3 -c 'import json,sys; print(json.load(sys.stdin)["status"]["token"])')
API=https://dispatch.fra.virdx.dev/api
curl -sf "$API/me" -H "Authorization: Bearer $TOKEN"     # username, team, role

Never write the token into a file, a spec or a report; it expires anyway.

Validate first, then post the spec text with a target and a stable run_id:

vxrun spec validate run.yaml                             # exit 1 while something blocks
python3 - run.yaml fra qc-sweep-001 > body.json <<'PY'
import json, sys
spec, target, run_id = sys.argv[1:4]
print(json.dumps({"target": target, "run_id": run_id, "spec": open(spec).read()}))
PY
curl -sf "$API/runs" -H "Authorization: Bearer $TOKEN" \
  -H 'Content-Type: application/json' -d @body.json

target is fra, aws or auto (auto resolves on the server; today that means fra unless the run was made for the cloud). run_id is also the Workflow name; posting the same run_id again returns the existing run (HTTP 200 instead of 201) rather than starting a second one, and a run_id owned by someone else is a 409. The response is the run with its first state: submitted (on the cluster), pending-approval (cloud target, see below), or rejected / failed with error listing the reasons; fix the spec and resubmit under the same run_id.

Then poll and act on the run by its id:

call what it gives
GET /api/runs/{run_id} state, cluster, progress (items total / succeeded / failed / running / pending, workers running) once submitted, forecast while pending approval, output_uri and output_state when finished, can_stop, requires_approval
GET /api/runs/{run_id}/events every state transition with time, actor and detail (an admin’s decision carries detail.comment)
GET /api/runs/{run_id}/logs streamed from Argo while running; after that from the log store (14 days) as <timestamp> [<pod>] <message>; ?container=all for sidecars; 409 with the reason when there is nothing to read
POST /api/runs/{run_id}/stop terminates the Workflow (owner or admin); the run ends stopped
GET /api/runs your runs; filters state, cluster, team, origin, cloud; ?all=true for admins

Poll about every 30 seconds, not faster; states move through submitted -> running -> succeeded | failed | stopped. The UI shows the same run at https://dispatch.fra.virdx.dev with the events, logs and the cards below. get_run / get_run_logs still work on the Workflow by name (the run_id) while it exists on the cluster.

What you get back

Once the run finishes, dispatch fills in the record; nothing in it is guessed (a value with no basis is null):

  • GET .../metrics: duration, items counted and attempts, and the pod totals: gpu_seconds, cpu_seconds, memory_mb_seconds granted, node_hours, preempted_pods, oomkilled_pods, plus what was used, sampled from Prometheus after each pod ended: gpu_util_avg, cpu_used_seconds, mem_max_bytes. Right-size the next spec from these.
  • GET .../pods: every attempt of every pod with template, host, GPU model, phase, exit reason (oomkilled, preempted, deadline), granted and used resources.
  • GET .../cost and .../costs: the AWS estimate in USD (spot and on-demand, for the instance type the run maps to) and, where a rate card exists, the on-prem estimate in EUR.
  • GET .../cloud: the cloud compatibility card (next section), also stored on the run as cloud_verdict.
  • GET .../forecast (while a cloud run waits for approval): data size, items, estimated node hours and wall time from the median of earlier succeeded runs of the same spec name and image repository, and the spot / on-demand price that implies.
  • Outputs: output_uri is the workflow-output prefix, whichever cluster ran the job; AWS results are pulled back (output_state: collecting -> local). Collect them as for any batch (vxrun batch results, read_doc("vx/cluster/storage-practices") “Batch outputs”).
  • GET /api/reports/usage?group=user|team|cluster&since=&until=: runs by outcome, GPU, CPU and wall hours, preempted and OOM-killed pods, AWS and on-prem estimates, per group. Admins see everyone; you see your own.

The cloud target

You may submit with target: aws (or auto). The cloud path is in testing: every AWS run stops at pending-approval and waits for an admin, and most will be rejected for now while the path is being proven. Nobody, agent or person, can promise an AWS run will be approved; do not plan around one. A pending run expires after seven days without a decision. The reason for a rejection is in the decision’s comment (GET .../events, detail.comment) and in the cloud card, so each attempt tells you what to change.

What you can do today is make a run cloud-ready, which the card measures with eight checks (GET /api/runs/{run_id}/cloud; each pass, warn, fail or skip with a one-line reason, verdict fail if any check fails):

check passes when what to do otherwise
dispatch the run came through dispatch submit through dispatch; direct Argo runs always fail this one
instance a tracked EC2 type carries the GPU class and count (or is CPU-only for gpu: 0) and some zone offers it now pick a GPU class and count that map to a tracked type (GET /api/instances)
fit that type covers the worker’s CPU and memory lower resources.cpu / memory to a tracked shape
data no inputs declared and no /mnt/ path in any command fails by design for now, see below
image the image digest is in the registry inventory pin image.ref to a digest the registry holds (a warn clears once the inventory has seen it)
outputs outputs.publish: true publish; /output on NFS does not exist in the cloud
local_cache none, or at most the NVMe every tracked type has (200 GiB) shrink local_cache or accept the warning
interruption execution.interruptible: true (preemptible scheduling implies it) make items restart cleanly and declare the waiver; spot capacity needs it. Fails outright when spot risk for the type is high, warns otherwise; retried pods of earlier runs of the same image are quoted as evidence

Be clear with users about data: a run that declares inputs currently fails this check by design, because bundle replication to the cloud has not been verified end to end. The dataset gate rejects any declared input on aws after approval. Until that changes, only runs whose data is inside the image or that read no data can pass; declaring inputs is still right for FRA runs and for the record.

A cloud run that is approved goes approved -> replicating (the image is copied to the cloud registry; GET .../replications) -> submitted, runs there, and its outputs are pulled back into workflow-output. The same run_id, events, logs and stop apply.

Runs that bypass dispatch

Dispatch watches the clusters, so a run-batch or run-image Workflow submitted directly (vxrun, kuberun, build_and_submit_vxrun, submit_run) is adopted as an observed run (origin: observed) under the user Argo recorded, with the commit and build where vxrun put them on the Workflow, the image, and a spec reconstructed from the Workflow’s arguments. From then on it is followed like any other: metrics and pods when it finishes, the usage report, the estimates. But it is a thinner record: no declared inputs, no commit for kuberun submissions, no outputs located, can_stop false (vxrun owns the Workflow, POST .../stop answers 409; use cancel_run), and the cloud card fails the dispatch and data checks every time. A Workflow that was never observed while it existed is lost to the register.

Endpoint summary

All under https://dispatch.fra.virdx.dev/api, bearer token required: GET /me; POST /runs; GET /runs, /runs/{run_id}, .../events, .../logs, .../metrics, .../pods, .../cost, .../costs, .../cloud, .../forecast, .../replications; POST /runs/{run_id}/stop, .../collect (retry a failed output pull); GET /images, /builds, /instances, /overview, /reports/usage, /reports/capacity. Admin only: POST /runs/{run_id}/approve|reject. The API README in mono/apps/vxdispatch-api is the authoritative list, with response fields.

Navigation

Type to search…

↑↓ navigate↵ selectEsc close