Skip to content
VIRDX

Virdx cluster design (for agents and people)

A k3s cluster in Frankfurt, managed as GitOps from infra_k8s via ArgoCD. Users never touch manifests for ordinary work: vxrun (alias kuberun) builds an image and submits an Argo Workflow for them.

The path of a job

  1. vxrun run in a Git checkout commits and pushes the selected project directory (autocommit is on by default), then asks Tekton (default) or Argo to build the image and push it to the registry at zot.fra.virdx.dev:5000. The build result is an immutable digest.
  2. vxrun submits an Argo Workflow in namespace argo-workflows that references the shared run-image WorkflowTemplate with that digest, the CMD/ENTRYPOINT, resources and labels. vxrun batch references run-batch instead: one DAG task per item of an items.json, same image and resources, each attempt with its own /output directory; by default a pack step publishes each item’s output as a vxs3 bundle and an exit handler completes the batch as one bundle in the workflow-output bucket (see read_doc(“vx/cluster/submitting-jobs”), read_doc(“vxrun/batches”) and read_doc(“vxrun/workflows”)).
  3. Kyverno injects the submitter’s identity and queue; Kueue admits the Pod against the user’s quota; kube-scheduler places it on a GPU node.
  4. Inside a run-image Pod: /output (env VXRUN_OUTPUT_DIR; legacy alias KUBERUN_OUTPUT) is /mnt/artifacts/<user>/<image>/<workflow> (NFS, legacy) and is the durable result, shared by all retries of the same Workflow; /cache (env VXRUN_CACHE_DIR; legacy alias KUBERUN_CACHE) is node-local scratch when local_cache was requested; /mnt/storage is mounted read-only; all of /mnt/artifacts is mounted read-write. A run-batch Pod has no NFS mounts: /output is a temporary per-item TopoLVM claim that the pack step publishes to S3, and inputs are staged from S3 into /cache (see read_doc(“vx/cluster/storage-practices”)). Container scratch is lost. Pods are removed about one hour after completion; logs stay in Loki for 90 days. Workflow objects expire 2 days after success, 7 days after failure.

Argo UI: https://argo-workflows.fra.virdx.dev/workflows/argo-workflows/<workflow>. Kubernetes login for Coder workspaces: https://coder.fra.virdx.dev/external-auth/k3s.

Scheduling: one Kueue cohort, two kinds of queue

  • Personal (--no-preemptible, the default): every user has a ClusterQueue and LocalQueue user-<first>-<last> rendered from infra_iam. Nominal quota 36 CPU, 264Gi, 2 GPUs (doubled from 18 / 132Gi on 2026-10-05), no borrowing. Quota counts Pod requests including the Argo wait sidecar (100m CPU, 128Mi), so two 16 CPU / 128Gi containers fit and a third, or one 36 CPU, does not. vxrun’s request_ratio (default 0.5) halves CPU and memory requests relative to limits. These caps are shared with the user’s full-GPU Coder workspace and are admission caps, not reserved cards.
  • Preemptible (--preemptible): one shared ClusterQueue preemptible that borrows whatever the cohort leaves idle, fed by per-user LocalQueues user-<first>-<last>-preemptible. Admission order between users is usage-based fair sharing with a 48 hour half-life: a user who ran a lot of preemptible GPU time recently waits behind one who did not. The old shared LocalQueue preemptible still exists with pre-cutover workflows on it.
  • Eviction is not Kueue reclaim. Personal Pods carry priority class guaranteed (1000); preemptible Pods carry preemptible (-1). When a personal Pod needs a GPU that a preemptible Pod holds, kube-scheduler evicts the preemptible Pod. Each attempt also has a 12 hour activeDeadlineSeconds; Argo retries up to 10 times, from scratch. Resume logic is the job’s job.
  • Team queues team-ve2e, team-histo and team-diffsim hold the RTX 6000 PRO quota (rendered from infra_iam). --team <name> sets virdx.dev/team: <name>; Kyverno routes the workflow to team-<name> and rejects it unless the submitter is listed under that team in the team-membership ConfigMap. A user may be in several teams and picks one per job.
  • Labels drive routing: virdx.dev/preemptible: "true" and virdx.dev/gpu-class. Kyverno turns them into hard node selectors (GPU product plus node class). Do not hand-write nodeSelectors. gpu_class: auto resolves to Ada for personal jobs, PRO for team jobs and either type for preemptible jobs. Since 2026-10-14 PRO is team-or-preemptible only: a personal rtx-6000-pro is rejected by vxrun and by the cluster.
  • Pods wait on the scheduling gate kueue.x-k8s.io/admission until admitted; the Pod label kueue.x-k8s.io/queue-name says which queue they are in.
  • The only resource flavor in use is shared (any GPU node); general and pro are legacy. Each identity may run 10 Workflows concurrently; Workflow parallelism defaults to 20 (max 25); the namespace caps at 200 Workflows.

Hardware

Node GPUs CPU / memory (approx.)
compute-002 5 × RTX 6000 Ada (48 GB), class rtx-6000 (3 more Adas time-sliced for Coder) 382 CPU / 2.2 TiB
compute-001, compute-003 8 × RTX PRO 6000 Blackwell (96 GB) each, class rtx-6000-pro ~512 CPU / 1.6 TiB each
compute-004 to 006 none (infra nodes) jobs never land here: Kueue adds nvidia.com/gpu.present

One Pod cannot span nodes or mix GPU types. Custom containers are capped at 192 CPU / 512Gi / 8 GPUs by policy; run-image offers a narrower CPU enum. Nominal cohort GPU quota (68) exceeds the 21 physical GPUs, so admission does not guarantee a card is free.

Where things are stuck

Symptom Meaning Look at
Workflow Pending, no Pod per-user semaphore (10 concurrent Workflows) get_run(workflow) nodes
Pod Pending with gate kueue.x-k8s.io/admission, personal queue your own quota is full, including a GPU Coder workspace get_capacity personal_queue usage
Pod Pending with gate, preemptible queue no idle cohort quota, or others rank ahead on fair sharing get_capacity preemptible_queue (fair_sharing)
Pod Pending, no gate admitted, but no node has the GPU free or the class selector excludes nodes; a personal Pod will evict a preemptible one pod conditions, get_capacity gpu inventory
Preemptible job restarted evicted by a guaranteed Pod, or hit the 12 h attempt deadline; after 10 retries it fails expected; checkpoint/resume is the fix
Not authenticated to Kubernetes Coder kube token expired the Coder external-auth link above
OOMKilled, ImagePullBackOff in get_run nodes memory limit too low; image or registry problem get_run_logs, raise memory, rebuild
A process in the workspace died with Killed (exit 137) and no traceback, or the whole workspace restarted the workspace’s memory cgroup hit its limit: processes and /dev/shm share one ceiling coder stat mem, “Workspace memory” below

Workspace memory (Coder)

A Coder workspace is one container with one memory cgroup: the shell, the coding agent, Jupyter kernels, DataLoader workers and everything written to /dev/shm share the limit set by the workspace’s memory parameter (GiB, maximum 96; the default is lower, so check rather than assume). The request is far below the limit (12Gi at most), so the limit is a hard ceiling, not reserved memory. Crossing it makes the kernel OOM-kill the largest process in the cgroup. Two outcomes, both common:

  • A child dies (a python training script, pytest -n, a bun build): the container survives and the user sees Killed (exit 137, signal 9) with no traceback. kubectl get pods shows nothing.
  • The agent or init process dies: the whole workspace restarts and every running process is gone.

Check headroom before starting anything heavy in the workspace, and whenever a process died without a trace:

  • coder stat mem (coder stat cpu; coder stat for all of it): usage against the cgroup limit, the same numbers the workspace page shows.
  • cat /sys/fs/cgroup/memory.max /sys/fs/cgroup/memory.current: limit and usage in bytes. grep -E '^(anon|file|shmem) ' /sys/fs/cgroup/memory.stat splits usage into process memory, page cache and tmpfs/shm. grep oom_kill /sys/fs/cgroup/memory.events counts kills in this container so far.
  • env | grep WORKSPACE_: workspaces built from the current kubernetes-dev template export WORKSPACE_MEMORY_LIMIT_GIB and WORKSPACE_CPU_LIMIT.

Memory-backed filesystems are charged to the same limit and the kernel cannot reclaim them. /dev/shm is a 16Gi memory-backed emptyDir: a dataset unpacked there, or tensors that many DataLoader workers share through it, take that much away from the processes long before it reports No space left on device. df -h -t tmpfs lists what is memory-backed; du -sh /dev/shm shows what is in it. Keep /dev/shm for shared memory between the processes of one job and remove what is left when the job ends. Files go in the home directory (a PVC, persists across restarts) or /tmp (the container’s writable layer on node disk, lost on restart), never on a tmpfs; datasets are not copied into the workspace at all when a job can stage them from S3 into /cache.

When the workspace is tight:

  • Run the heavy step as a job (build_and_submit_vxrun, run-batch). A job has its own memory limit, chosen per run, and never competes with the workspace.
  • In the workspace, lower num_workers, batch size and in-memory caching before reaching for more memory.
  • If the work really belongs in the workspace, raise the memory parameter in the workspace settings (up to 96) and rebuild.

Storage

  • /output → /mnt/artifacts/<user>/<image>/<workflow> (NFS, run-image only): durable job output; fine for small or infrequent checkpoints. NFS is legacy; nothing new should depend on /mnt/*.
  • run-batch: no NFS mounts. /output is a temporary per-item claim on the node’s NVMe, published to s3://workflow-output by the pack step and read back with vxrun batch results. Faster than NFS and the path for new work.
  • s3://scratch/<username>/<experiment-or-workflow>/ (Ceph RGW, [scratch] profile in Coder and jobs): temporary development data, experiments and checkpoints, with a two-week (14-day) GC policy and shared size quota. Frequent large checkpoint rewrites fit here better than on NFS. Scratch is not for permanent storage or repeatable workflow inputs and outputs. Once happy with the data, publish it as a permanent vxdata resource and use that resource in future workflows.
  • /cache: node-local scratch (--local-cache 64Gi, TopoLVM volume on the node’s NVMe), lost with the Pod.
  • Object storage via vxdata-sdk / vxs3 (Ceph RGW): permanent vxdata resources for datasets, model weights and published results. Keep both the resource and its data in vxdata; a resource pointing at scratch still loses its data to GC. vxrun storage uploads and optimises datasets through the vxoptimise namespace. See read_doc(“vx/cluster/storage-practices”).

Builds

Tekton tag-image task / Argo buildah template produce the image; BuildKit cache notes live in infra_k8s/docs/buildkit-cache.md. The Dockerfile’s CMD/ENTRYPOINT is the default job command.

Repeatable runs

The Workflow carries kuberun.virdx.com/commit and the image digest; together they are the only record of what a job executed. A run is repeatable when that record is complete: every line of code in the image, every input immutable. Three things break it, and the tools treat them differently:

  • Code executed from /mnt/* or the home directory (python /mnt/storage/users/x/train.py, PYTHONPATH=/mnt/scratch/lib). Whoever edits the file changes the next run of the same commit. vxrun warns on such a command (build_and_submit_vxrun dry run and vxrun plan: warnings, prefix “Not repeatable”; DAG validation: a per-task warning). Nothing is blocked yet. Fix: move the code into the repository.
  • Code pulled in at run time (sys.path.insert into a mount, git clone, curl | sh, pip install in cmd). review_job reports these as high findings (code-from-shared-mount, code-fetch-at-runtime, runtime-install). Fix: vendor or install in the Dockerfile at a pinned revision.
  • An environment that drifts between builds: untagged or :latest base image, unpinned fetches or pip install in the Dockerfile. review_job reports these from the Dockerfile (unpinned-base-image, dockerfile-fetches-at-build, unpinned-dependency-install).

Data may live outside the image, but only as published, immutable inputs: vxdata resources staged into /cache, not scratch (GC) or a mutable NFS path.

Navigation

Type to search…

↑↓ navigate↵ selectEsc close