---
title: "Virdx cluster design (for agents and people)"
image: "https://docs.virdx.dev/img/virdx-social-card.png"
---

> Documentation Index
> Fetch the complete documentation index at: https://docs.virdx.dev/llms.txt
> Use this file to discover all available pages before exploring further.

# Virdx cluster design (for agents and people)

A k3s cluster in Frankfurt, managed as GitOps from `infra_k8s` via ArgoCD.
Users never touch manifests for ordinary work: `vxrun` (alias `kuberun`)
builds an image and submits an Argo Workflow for them.

## The path of a job

1. `vxrun run` in a Git checkout commits and pushes the selected project
   directory (autocommit is on by default), then asks Tekton (default) or Argo
   to build the image and push it to the registry at `zot.fra.virdx.dev:5000`.
   The build result is an immutable digest.
2. vxrun submits an Argo `Workflow` in namespace `argo-workflows` that
   references the shared `run-image` WorkflowTemplate with that digest, the
   CMD/ENTRYPOINT, resources and labels. `vxrun batch` references `run-batch`
   instead: one DAG task per item of an `items.json`, same image and
   resources, each attempt with its own `/output` directory; by default a
   `pack` step publishes each item's output as a vxs3 bundle and an exit
   handler completes the batch as one bundle in the `workflow-output` bucket
   (see read_doc("vx/cluster/submitting-jobs"), read_doc("vxrun/batches") and read_doc("vxrun/workflows")).
3. Kyverno injects the submitter's identity and queue; Kueue admits the Pod
   against the user's quota; kube-scheduler places it on a GPU node.
4. Inside a `run-image` Pod: `/output` (env `VXRUN_OUTPUT_DIR`; legacy alias
   `KUBERUN_OUTPUT`) is
   `/mnt/artifacts/<user>/<image>/<workflow>` (NFS, legacy) and is the durable
   result, shared by all retries of the same Workflow; `/cache` (env
   `VXRUN_CACHE_DIR`; legacy alias `KUBERUN_CACHE`) is node-local scratch when `local_cache` was requested;
   `/mnt/storage` is mounted read-only; all of `/mnt/artifacts` is mounted
   read-write. A `run-batch` Pod has no NFS mounts: `/output` is a temporary
   per-item TopoLVM claim that the `pack` step publishes to S3, and inputs are
   staged from S3 into `/cache` (see read_doc("vx/cluster/storage-practices")).
   Container scratch is lost. Pods are removed about one hour after completion;
   logs stay in Loki for 90 days. Workflow objects expire 2 days after success,
   7 days after failure.

Argo UI: `https://argo-workflows.fra.virdx.dev/workflows/argo-workflows/<workflow>`.
Kubernetes login for Coder workspaces: `https://coder.fra.virdx.dev/external-auth/k3s`.

## Scheduling: one Kueue cohort, two kinds of queue

- **Personal** (`--no-preemptible`, the default): every user has a ClusterQueue
  and LocalQueue `user-<first>-<last>` rendered from `infra_iam`. Nominal quota
  36 CPU, 264Gi, 2 GPUs (doubled from 18 / 132Gi on 2026-10-05), no borrowing.
  Quota counts Pod requests including the Argo wait sidecar (100m CPU, 128Mi),
  so two 16 CPU / 128Gi containers fit and a third, or one 36 CPU, does not.
  vxrun's `request_ratio` (default 0.5) halves CPU and memory requests relative
  to limits. These caps are shared with the user's
  full-GPU Coder workspace and are admission caps, not reserved cards.
- **Preemptible** (`--preemptible`): one shared ClusterQueue `preemptible` that
  borrows whatever the cohort leaves idle, fed by per-user LocalQueues
  `user-<first>-<last>-preemptible`. Admission order between users is
  usage-based fair sharing with a 48 hour half-life: a user who ran a lot of
  preemptible GPU time recently waits behind one who did not. The old shared
  LocalQueue `preemptible` still exists with pre-cutover workflows on it.
- **Eviction is not Kueue reclaim.** Personal Pods carry priority class
  `guaranteed` (1000); preemptible Pods carry `preemptible` (-1). When a
  personal Pod needs a GPU that a preemptible Pod holds, kube-scheduler evicts
  the preemptible Pod. Each attempt also has a 12 hour `activeDeadlineSeconds`;
  Argo retries up to 10 times, from scratch. Resume logic is the job's job.
- Team queues `team-ve2e`, `team-histo` and `team-diffsim` hold the RTX 6000
  PRO quota (rendered from infra_iam). `--team <name>` sets `virdx.dev/team:
  <name>`; Kyverno routes the workflow to `team-<name>` and rejects it unless
  the submitter is listed under that team in the `team-membership` ConfigMap.
  A user may be in several teams and picks one per job.
- Labels drive routing: `virdx.dev/preemptible: "true"` and
  `virdx.dev/gpu-class`. Kyverno turns them into hard node selectors
  (GPU product plus node class). Do not hand-write nodeSelectors. `gpu_class:
  auto` resolves to Ada for personal jobs, PRO for team jobs and either type
  for preemptible jobs. Since 2026-10-14 PRO is team-or-preemptible only: a
  personal `rtx-6000-pro` is rejected by vxrun and by the cluster.
- Pods wait on the scheduling gate `kueue.x-k8s.io/admission` until admitted;
  the Pod label `kueue.x-k8s.io/queue-name` says which queue they are in.
- The only resource flavor in use is `shared` (any GPU node); `general` and
  `pro` are legacy. Each identity may run 10 Workflows concurrently; Workflow
  parallelism defaults to 20 (max 25); the namespace caps at 200 Workflows.

## Hardware

| Node | GPUs | CPU / memory (approx.) |
|---|---|---|
| compute-002 | 5 × RTX 6000 Ada (48 GB), class `rtx-6000` (3 more Adas time-sliced for Coder) | 382 CPU / 2.2 TiB |
| compute-001, compute-003 | 8 × RTX PRO 6000 Blackwell (96 GB) each, class `rtx-6000-pro` | ~512 CPU / 1.6 TiB each |
| compute-004 to 006 | none (infra nodes) | jobs never land here: Kueue adds `nvidia.com/gpu.present` |

One Pod cannot span nodes or mix GPU types. Custom containers are capped at
192 CPU / 512Gi / 8 GPUs by policy; `run-image` offers a narrower CPU enum.
Nominal cohort GPU quota (68) exceeds the 21 physical GPUs, so admission does
not guarantee a card is free.

## Where things are stuck

| Symptom | Meaning | Look at |
|---|---|---|
| Workflow Pending, no Pod | per-user semaphore (10 concurrent Workflows) | `get_run(workflow)` nodes |
| Pod Pending with gate `kueue.x-k8s.io/admission`, personal queue | your own quota is full, including a GPU Coder workspace | `get_capacity` personal_queue usage |
| Pod Pending with gate, preemptible queue | no idle cohort quota, or others rank ahead on fair sharing | `get_capacity` preemptible_queue (`fair_sharing`) |
| Pod Pending, no gate | admitted, but no node has the GPU free or the class selector excludes nodes; a personal Pod will evict a preemptible one | pod conditions, `get_capacity` gpu inventory |
| Preemptible job restarted | evicted by a guaranteed Pod, or hit the 12 h attempt deadline; after 10 retries it fails | expected; checkpoint/resume is the fix |
| `Not authenticated to Kubernetes` | Coder kube token expired | the Coder external-auth link above |
| `OOMKilled`, `ImagePullBackOff` in `get_run` nodes | memory limit too low; image or registry problem | `get_run_logs`, raise memory, rebuild |
| A process in the workspace died with `Killed` (exit 137) and no traceback, or the whole workspace restarted | the workspace's memory cgroup hit its limit: processes and `/dev/shm` share one ceiling | `coder stat mem`, "Workspace memory" below |

## Workspace memory (Coder)

A Coder workspace is one container with one memory cgroup: the shell, the
coding agent, Jupyter kernels, DataLoader workers and everything written to
`/dev/shm` share the limit set by the workspace's `memory` parameter (GiB,
maximum 96; the default is lower, so check rather than assume). The request is
far below the limit (12Gi at most), so the limit is a hard ceiling, not reserved
memory. Crossing it makes the kernel OOM-kill the largest process in the
cgroup. Two outcomes, both common:

- A child dies (a `python` training script, `pytest -n`, a `bun` build): the
  container survives and the user sees `Killed` (exit 137, signal 9) with no
  traceback. `kubectl get pods` shows nothing.
- The agent or init process dies: the whole workspace restarts and every
  running process is gone.

Check headroom before starting anything heavy in the workspace, and whenever
a process died without a trace:

- `coder stat mem` (`coder stat cpu`; `coder stat` for all of it): usage
  against the cgroup limit, the same numbers the workspace page shows.
- `cat /sys/fs/cgroup/memory.max /sys/fs/cgroup/memory.current`: limit and
  usage in bytes. `grep -E '^(anon|file|shmem) ' /sys/fs/cgroup/memory.stat`
  splits usage into process memory, page cache and tmpfs/shm.
  `grep oom_kill /sys/fs/cgroup/memory.events` counts kills in this container
  so far.
- `env | grep WORKSPACE_`: workspaces built from the current `kubernetes-dev`
  template export `WORKSPACE_MEMORY_LIMIT_GIB` and `WORKSPACE_CPU_LIMIT`.

Memory-backed filesystems are charged to the same limit and the kernel cannot
reclaim them. `/dev/shm` is a 16Gi memory-backed emptyDir: a dataset unpacked
there, or tensors that many DataLoader workers share through it, take that
much away from the processes long before it reports `No space left on
device`. `df -h -t tmpfs` lists what is memory-backed; `du -sh /dev/shm` shows
what is in it. Keep `/dev/shm` for shared memory between the processes of one
job and remove what is left when the job ends. Files go in the home directory
(a PVC, persists across restarts) or `/tmp` (the container's writable layer on
node disk, lost on restart), never on a tmpfs; datasets are not copied into
the workspace at all when a job can stage them from S3 into `/cache`.

When the workspace is tight:

- Run the heavy step as a job (`build_and_submit_vxrun`, `run-batch`). A job has its own
  memory limit, chosen per run, and never competes with the workspace.
- In the workspace, lower `num_workers`, batch size and in-memory caching
  before reaching for more memory.
- If the work really belongs in the workspace, raise the `memory` parameter in
  the workspace settings (up to 96) and rebuild.

## Storage

- `/output` → `/mnt/artifacts/<user>/<image>/<workflow>` (NFS, `run-image`
  only): durable job output; fine for small or infrequent checkpoints. NFS is
  legacy; nothing new should depend on `/mnt/*`.
- `run-batch`: no NFS mounts. `/output` is a temporary per-item claim on the
  node's NVMe, published to `s3://workflow-output` by the `pack` step and read
  back with `vxrun batch results`. Faster than NFS and the path for new work.
- `s3://scratch/<username>/<experiment-or-workflow>/` (Ceph RGW, `[scratch]`
  profile in Coder and jobs): temporary development data, experiments and
  checkpoints, with a two-week (14-day) GC policy and shared size quota.
  Frequent large checkpoint rewrites fit here better than on NFS. Scratch is
  not for permanent storage or repeatable workflow inputs and outputs. Once
  happy with the data, publish it as a permanent vxdata resource and use that
  resource in future workflows.
- `/cache`: node-local scratch (`--local-cache 64Gi`, TopoLVM volume on the
  node's NVMe), lost with the Pod.
- Object storage via `vxdata-sdk` / `vxs3` (Ceph RGW): permanent vxdata resources
  for datasets, model weights and published results. Keep both the resource
  and its data in vxdata; a resource pointing at scratch still loses its data
  to GC. `vxrun storage` uploads and optimises datasets through
  the `vxoptimise` namespace. See read_doc("vx/cluster/storage-practices").

## Builds

Tekton `tag-image` task / Argo `buildah` template produce the image; BuildKit
cache notes live in `infra_k8s/docs/buildkit-cache.md`. The Dockerfile's
CMD/ENTRYPOINT is the default job command.

## Repeatable runs

The Workflow carries `kuberun.virdx.com/commit` and the image digest; together
they are the only record of what a job executed. A run is repeatable when that
record is complete: every line of code in the image, every input immutable.
Three things break it, and the tools treat them differently:

- Code executed from `/mnt/*` or the home directory (`python
  /mnt/storage/users/x/train.py`, `PYTHONPATH=/mnt/scratch/lib`). Whoever edits
  the file changes the next run of the same commit. `vxrun` warns on such a
  command (`build_and_submit_vxrun` dry run and `vxrun plan`: `warnings`, prefix "Not
  repeatable"; DAG validation: a per-task warning). Nothing is blocked yet.
  Fix: move the code into the repository.
- Code pulled in at run time (`sys.path.insert` into a mount, `git clone`,
  `curl | sh`, `pip install` in `cmd`). `review_job` reports these as high
  findings (`code-from-shared-mount`, `code-fetch-at-runtime`,
  `runtime-install`). Fix: vendor or install in the Dockerfile at a pinned
  revision.
- An environment that drifts between builds: untagged or `:latest` base image,
  unpinned fetches or `pip install` in the Dockerfile. `review_job` reports
  these from the Dockerfile (`unpinned-base-image`,
  `dockerfile-fetches-at-build`, `unpinned-dependency-install`).

Data may live outside the image, but only as published, immutable inputs:
vxdata resources staged into `/cache`, not scratch (GC) or a mutable NFS path.

Source: https://docs.virdx.dev/vx/cluster/cluster-design/index.mdx
