---
title: "Dispatch: the register and broker for cluster jobs"
image: "https://docs.virdx.dev/img/virdx-social-card.png"
---

> Documentation Index
> Fetch the complete documentation index at: https://docs.virdx.dev/llms.txt
> Use this file to discover all available pages before exploring further.

# Dispatch: the register and broker for cluster jobs

vxdispatch is the one door for finite batch jobs on the Virdx clusters: the
on-prem cluster (`fra`) and, in testing, AWS (`aws`). You hand it a run spec
(the same one-file format `read_doc("vxrun/specs")` describes), it verifies the spec,
decides the cluster, submits the `run-batch` Workflow as you, and keeps the
record of the run for good.

- UI: https://dispatch.fra.virdx.dev (runs, builds, images, instances, the
  review queue for admins).
- API: https://dispatch.fra.virdx.dev/api (FastAPI; the endpoints used below
  are listed at the end).
- Design and deployment: `infra_k8s/docs/vxdispatch.md`. Service source:
  `mono/apps/vxdispatch-api` (README has the full endpoint and settings
  reference) and `mono/apps/vxdispatch-ui`.

## Why go through dispatch, also for ordinary FRA runs

Argo deletes a finished Workflow after three days, and with it most of what you
would want to know later. Dispatch keeps the record: the spec as submitted,
the image digest and the commit it was built from, every pod with its host,
attempt, exit reason and the GPU, CPU and memory it was granted and what it
actually used, the outcome, and where the outputs landed
(`s3://workflow-output/<user>/<run_id>-<uid>/`). Runs are created as you, on
your own queue, so attribution to you and your team is exact and the usage
report (`GET /api/reports/usage`) adds up per user, team or cluster. Everyone
gets the same view of status, events, logs and stop in the UI and the API,
without kubectl. Every finished run gets an AWS cost estimate and a cloud
compatibility card: eight checks that say what would stop the same job running
in the cloud and how to fix it, so a job can be made cloud-ready before anyone
needs it to be. And cloud runs themselves (approval, image and bundle
replication, output collection) only exist through dispatch. A run submitted
straight to Argo with `vxrun`/`kuberun` is still recorded, but as a second-hand
observation with less in it (see the last section).

Use dispatch when the job is a finite list of independent commands on one image
with its data declared as bundles or staged inside the image. Keep the direct
path (`build_and_submit_vxrun`, `submit_run`, `vxrun batch submit`) for what dispatch
does not take yet: native Argo DAGs (`kind: workflow`), `run-image` jobs of a
repository without a spec, and anything that still reads `/mnt/*` at run time.

## The spec

Dispatch accepts the run spec and nothing else: `apiVersion:
vxrun.virdx.dev/v1alpha1`, `kind: batch`. Read `read_doc("vxrun/specs")` for the full form
(bundles, input selections, resolve-before-size). The model lives in the shared
spec package (`mono/packages/vxspec`); `vxrun`, `vx` and the dispatch API
all validate with the same rules, so a spec that passes `validate_run` or
`vxrun spec validate` passes dispatch's own verification unless the cluster
rules (team membership, quota) reject it.

A minimal batch spec, with the interruption waiver:

```yaml
apiVersion: vxrun.virdx.dev/v1alpha1
kind: batch
name: histo-qc-sweep
description: QC thresholds, one item per setting.

image:
  source: github.com/virdx/histo@<full 40-character commit sha>
  dockerfile: Dockerfile
  context: .
  ref: zot.fra.virdx.dev:5000/histo@sha256:<digest>   # from build_image

resources:
  cpu: "4"
  memory: 16Gi
  gpu: 1
  local_cache: 32Gi

execution:
  parallelism: 4
  retries: 2
  task_deadline: 7200
  # The author's waiver: every item can be killed at any point and restarted,
  # from scratch or from its own checkpoints, without harming the result.
  # Preemptible scheduling implies it; spot capacity in the cloud needs it.
  interruptible: true

scheduling:
  preemptible: true

items:
  - id: thr-05
argv: [python, -m, histo.qc, --threshold, "0.5", --out, /output]
  - id: thr-07
argv: [python, -m, histo.qc, --threshold, "0.7", --out, /output]

outputs:
  publish: true
```

Two things the spec needs before it can be posted:

- `image.ref` must be a digest. `build_image` builds a pushed commit and returns
  it; from a local checkout, `vx run workflow build` does the same. A digest of an earlier build of the same commit is
  equally good; `GET /api/images?repository=<repo>&commit=<sha>` lists them.
- Declared inputs must be pinned (`manifest` + `sha256`); `vx run spec resolve`
  does that and sizes them so `local_cache` can be set from
  `storage.minimum_local_cache`.

`execution.interruptible` arrived with the shared spec package. A `vxrun` older
than the release that ships it rejects the key as unknown in `vxrun spec
validate` and `validate_run` (the spec model forbids unknown fields) while the
dispatch API accepts it. If that happens, update vx (`pixi global update vx`), or
validate without the key and add it before posting. Declare it only when it is
true: it is a promise about the program, not a scheduling flag.

## Submit

Authentication is a Keycloak bearer token whose audience includes `dispatch`.
The token kubectl already holds for the cluster carries it, and so does the
token of a Coder workspace, so nothing new is issued:

```sh
TOKEN=$(kubectl oidc-login get-token \
  --oidc-issuer-url=https://keycloak.fra.virdx.dev/realms/virdx \
  --oidc-client-id=k3s | python3 -c 'import json,sys; print(json.load(sys.stdin)["status"]["token"])')
API=https://dispatch.fra.virdx.dev/api
curl -sf "$API/me" -H "Authorization: Bearer $TOKEN"     # username, team, role
```

Never write the token into a file, a spec or a report; it expires anyway.

Validate first, then post the spec text with a target and a stable `run_id`:

```sh
vxrun spec validate run.yaml                             # exit 1 while something blocks
python3 - run.yaml fra qc-sweep-001 > body.json <<'PY'
import json, sys
spec, target, run_id = sys.argv[1:4]
print(json.dumps({"target": target, "run_id": run_id, "spec": open(spec).read()}))
PY
curl -sf "$API/runs" -H "Authorization: Bearer $TOKEN" \
  -H 'Content-Type: application/json' -d @body.json
```

`target` is `fra`, `aws` or `auto` (`auto` resolves on the server; today that
means `fra` unless the run was made for the cloud). `run_id` is also the
Workflow name; posting the same `run_id` again returns the existing run (HTTP
200 instead of 201) rather than starting a second one, and a `run_id` owned by
someone else is a 409. The response is the run with its first state:
`submitted` (on the cluster), `pending-approval` (cloud target, see below), or
`rejected` / `failed` with `error` listing the reasons; fix the spec and
resubmit under the same `run_id`.

Then poll and act on the run by its id:

| call | what it gives |
|---|---|
| `GET /api/runs/{run_id}` | state, cluster, `progress` (items total / succeeded / failed / running / pending, workers running) once submitted, `forecast` while pending approval, `output_uri` and `output_state` when finished, `can_stop`, `requires_approval` |
| `GET /api/runs/{run_id}/events` | every state transition with time, actor and detail (an admin's decision carries `detail.comment`) |
| `GET /api/runs/{run_id}/logs` | streamed from Argo while running; after that from the log store (14 days) as `<timestamp> [<pod>] <message>`; `?container=all` for sidecars; 409 with the reason when there is nothing to read |
| `POST /api/runs/{run_id}/stop` | terminates the Workflow (owner or admin); the run ends `stopped` |
| `GET /api/runs` | your runs; filters `state`, `cluster`, `team`, `origin`, `cloud`; `?all=true` for admins |

Poll about every 30 seconds, not faster; states move through `submitted ->
running -> succeeded | failed | stopped`. The UI shows the same run at
`https://dispatch.fra.virdx.dev` with the events, logs and the cards below.
`get_run` / `get_run_logs` still work on the Workflow by name (the `run_id`)
while it exists on the cluster.

## What you get back

Once the run finishes, dispatch fills in the record; nothing in it is guessed
(a value with no basis is `null`):

- `GET .../metrics`: duration, items counted and attempts, and the pod
  totals: `gpu_seconds`, `cpu_seconds`, `memory_mb_seconds` granted,
  `node_hours`, `preempted_pods`, `oomkilled_pods`, plus what was used, sampled
  from Prometheus after each pod ended: `gpu_util_avg`, `cpu_used_seconds`,
  `mem_max_bytes`. Right-size the next spec from these.
- `GET .../pods`: every attempt of every pod with template, host, GPU model,
  phase, exit reason (`oomkilled`, `preempted`, `deadline`), granted and used
  resources.
- `GET .../cost` and `.../costs`: the AWS estimate in USD (spot and
  on-demand, for the instance type the run maps to) and, where a rate card
  exists, the on-prem estimate in EUR.
- `GET .../cloud`: the cloud compatibility card (next section), also stored
  on the run as `cloud_verdict`.
- `GET .../forecast` (while a cloud run waits for approval): data size, items,
  estimated node hours and wall time from the median of earlier succeeded runs
  of the same spec name and image repository, and the spot / on-demand price
  that implies.
- Outputs: `output_uri` is the `workflow-output` prefix, whichever cluster ran
  the job; AWS results are pulled back (`output_state: collecting -> local`).
  Collect them as for any batch (`vxrun batch results`,
  `read_doc("vx/cluster/storage-practices")` "Batch outputs").
- `GET /api/reports/usage?group=user|team|cluster&since=&until=`: runs by
  outcome, GPU, CPU and wall hours, preempted and OOM-killed pods, AWS and
  on-prem estimates, per group. Admins see everyone; you see your own.

## The cloud target

You may submit with `target: aws` (or `auto`). The cloud path is in testing:
every AWS run stops at `pending-approval` and waits for an admin, and most will
be rejected for now while the path is being proven. Nobody, agent or person,
can promise an AWS run will be approved; do not plan around one. A pending run
expires after seven days without a decision. The reason for a rejection is in
the decision's comment (`GET .../events`, `detail.comment`) and in the cloud
card, so each attempt tells you what to change.

What you can do today is make a run cloud-ready, which the card measures with
eight checks (`GET /api/runs/{run_id}/cloud`; each `pass`, `warn`, `fail` or
`skip` with a one-line reason, verdict `fail` if any check fails):

| check | passes when | what to do otherwise |
|---|---|---|
| `dispatch` | the run came through dispatch | submit through dispatch; direct Argo runs always fail this one |
| `instance` | a tracked EC2 type carries the GPU class and count (or is CPU-only for `gpu: 0`) and some zone offers it now | pick a GPU class and count that map to a tracked type (`GET /api/instances`) |
| `fit` | that type covers the worker's CPU and memory | lower `resources.cpu` / `memory` to a tracked shape |
| `data` | no inputs declared and no `/mnt/` path in any command | fails by design for now, see below |
| `image` | the image digest is in the registry inventory | pin `image.ref` to a digest the registry holds (a `warn` clears once the inventory has seen it) |
| `outputs` | `outputs.publish: true` | publish; `/output` on NFS does not exist in the cloud |
| `local_cache` | none, or at most the NVMe every tracked type has (200 GiB) | shrink `local_cache` or accept the warning |
| `interruption` | `execution.interruptible: true` (preemptible scheduling implies it) | make items restart cleanly and declare the waiver; spot capacity needs it. Fails outright when spot risk for the type is high, warns otherwise; retried pods of earlier runs of the same image are quoted as evidence |

Be clear with users about `data`: a run that declares inputs currently fails
this check by design, because bundle replication to the cloud has not been
verified end to end. The dataset gate rejects any declared input on `aws`
after approval. Until that changes, only runs whose data is inside the image or
that read no data can pass; declaring inputs is still right for FRA runs and
for the record.

A cloud run that is approved goes `approved -> replicating` (the image is
copied to the cloud registry; `GET .../replications`) `-> submitted`, runs
there, and its outputs are pulled back into `workflow-output`. The same
`run_id`, events, logs and stop apply.

## Runs that bypass dispatch

Dispatch watches the clusters, so a `run-batch` or `run-image` Workflow
submitted directly (`vxrun`, `kuberun`, `build_and_submit_vxrun`, `submit_run`) is
adopted as an observed run (`origin: observed`) under the user Argo recorded,
with the commit and build where vxrun put them on the Workflow, the image, and
a spec reconstructed from the Workflow's arguments. From then on it is followed
like any other: metrics and pods when it finishes, the usage report, the
estimates. But it is a thinner record: no declared inputs, no commit for
kuberun submissions, no outputs located, `can_stop` false (vxrun owns the
Workflow, `POST .../stop` answers 409; use `cancel_run`), and the cloud card
fails the `dispatch` and `data` checks every time. A Workflow that was never
observed while it existed is lost to the register.

## Endpoint summary

All under `https://dispatch.fra.virdx.dev/api`, bearer token required:
`GET /me`; `POST /runs`; `GET /runs`, `/runs/{run_id}`, `.../events`,
`.../logs`, `.../metrics`, `.../pods`, `.../cost`, `.../costs`, `.../cloud`,
`.../forecast`, `.../replications`; `POST /runs/{run_id}/stop`,
`.../collect` (retry a failed output pull); `GET /images`, `/builds`,
`/instances`, `/overview`, `/reports/usage`, `/reports/capacity`. Admin only:
`POST /runs/{run_id}/approve|reject`. The API README in
`mono/apps/vxdispatch-api` is the authoritative list, with response fields.

Source: https://docs.virdx.dev/vx/cluster/dispatch/index.mdx
