A k3s cluster in Frankfurt, managed as GitOps from infra_k8s via ArgoCD.
Users never touch manifests for ordinary work: vxrun (alias kuberun)
builds an image and submits an Argo Workflow for them.
The path of a job
vxrun runin a Git checkout commits and pushes the selected project directory (autocommit is on by default), then asks Tekton (default) or Argo to build the image and push it to the registry atzot.fra.virdx.dev:5000. The build result is an immutable digest.- vxrun submits an Argo
Workflowin namespaceargo-workflowsthat references the sharedrun-imageWorkflowTemplate with that digest, the CMD/ENTRYPOINT, resources and labels.vxrun batchreferencesrun-batchinstead: one DAG task per item of anitems.json, same image and resources, each attempt with its own/outputdirectory; by default apackstep publishes each item’s output as a vxs3 bundle and an exit handler completes the batch as one bundle in theworkflow-outputbucket (see read_doc(“vx/cluster/submitting-jobs”), read_doc(“vxrun/batches”) and read_doc(“vxrun/workflows”)). - Kyverno injects the submitter’s identity and queue; Kueue admits the Pod against the user’s quota; kube-scheduler places it on a GPU node.
- Inside a
run-imagePod:/output(envVXRUN_OUTPUT_DIR; legacy aliasKUBERUN_OUTPUT) is/mnt/artifacts/<user>/<image>/<workflow>(NFS, legacy) and is the durable result, shared by all retries of the same Workflow;/cache(envVXRUN_CACHE_DIR; legacy aliasKUBERUN_CACHE) is node-local scratch whenlocal_cachewas requested;/mnt/storageis mounted read-only; all of/mnt/artifactsis mounted read-write. Arun-batchPod has no NFS mounts:/outputis a temporary per-item TopoLVM claim that thepackstep publishes to S3, and inputs are staged from S3 into/cache(see read_doc(“vx/cluster/storage-practices”)). Container scratch is lost. Pods are removed about one hour after completion; logs stay in Loki for 90 days. Workflow objects expire 2 days after success, 7 days after failure.
Argo UI: https://argo-workflows.fra.virdx.dev/workflows/argo-workflows/<workflow>.
Kubernetes login for Coder workspaces: https://coder.fra.virdx.dev/external-auth/k3s.
Scheduling: one Kueue cohort, two kinds of queue
- Personal (
--no-preemptible, the default): every user has a ClusterQueue and LocalQueueuser-<first>-<last>rendered frominfra_iam. Nominal quota 36 CPU, 264Gi, 2 GPUs (doubled from 18 / 132Gi on 2026-10-05), no borrowing. Quota counts Pod requests including the Argo wait sidecar (100m CPU, 128Mi), so two 16 CPU / 128Gi containers fit and a third, or one 36 CPU, does not. vxrun’srequest_ratio(default 0.5) halves CPU and memory requests relative to limits. These caps are shared with the user’s full-GPU Coder workspace and are admission caps, not reserved cards. - Preemptible (
--preemptible): one shared ClusterQueuepreemptiblethat borrows whatever the cohort leaves idle, fed by per-user LocalQueuesuser-<first>-<last>-preemptible. Admission order between users is usage-based fair sharing with a 48 hour half-life: a user who ran a lot of preemptible GPU time recently waits behind one who did not. The old shared LocalQueuepreemptiblestill exists with pre-cutover workflows on it. - Eviction is not Kueue reclaim. Personal Pods carry priority class
guaranteed(1000); preemptible Pods carrypreemptible(-1). When a personal Pod needs a GPU that a preemptible Pod holds, kube-scheduler evicts the preemptible Pod. Each attempt also has a 12 houractiveDeadlineSeconds; Argo retries up to 10 times, from scratch. Resume logic is the job’s job. - Team queues
team-ve2e,team-histoandteam-diffsimhold the RTX 6000 PRO quota (rendered from infra_iam).--team <name>setsvirdx.dev/team: <name>; Kyverno routes the workflow toteam-<name>and rejects it unless the submitter is listed under that team in theteam-membershipConfigMap. A user may be in several teams and picks one per job. - Labels drive routing:
virdx.dev/preemptible: "true"andvirdx.dev/gpu-class. Kyverno turns them into hard node selectors (GPU product plus node class). Do not hand-write nodeSelectors.gpu_class: autoresolves to Ada for personal jobs, PRO for team jobs and either type for preemptible jobs. Since 2026-10-14 PRO is team-or-preemptible only: a personalrtx-6000-prois rejected by vxrun and by the cluster. - Pods wait on the scheduling gate
kueue.x-k8s.io/admissionuntil admitted; the Pod labelkueue.x-k8s.io/queue-namesays which queue they are in. - The only resource flavor in use is
shared(any GPU node);generalandproare legacy. Each identity may run 10 Workflows concurrently; Workflow parallelism defaults to 20 (max 25); the namespace caps at 200 Workflows.
Hardware
| Node | GPUs | CPU / memory (approx.) |
|---|---|---|
| compute-002 | 5 × RTX 6000 Ada (48 GB), class rtx-6000 (3 more Adas time-sliced for Coder) |
382 CPU / 2.2 TiB |
| compute-001, compute-003 | 8 × RTX PRO 6000 Blackwell (96 GB) each, class rtx-6000-pro |
~512 CPU / 1.6 TiB each |
| compute-004 to 006 | none (infra nodes) | jobs never land here: Kueue adds nvidia.com/gpu.present |
One Pod cannot span nodes or mix GPU types. Custom containers are capped at
192 CPU / 512Gi / 8 GPUs by policy; run-image offers a narrower CPU enum.
Nominal cohort GPU quota (68) exceeds the 21 physical GPUs, so admission does
not guarantee a card is free.
Where things are stuck
| Symptom | Meaning | Look at |
|---|---|---|
| Workflow Pending, no Pod | per-user semaphore (10 concurrent Workflows) | get_run(workflow) nodes |
Pod Pending with gate kueue.x-k8s.io/admission, personal queue |
your own quota is full, including a GPU Coder workspace | get_capacity personal_queue usage |
| Pod Pending with gate, preemptible queue | no idle cohort quota, or others rank ahead on fair sharing | get_capacity preemptible_queue (fair_sharing) |
| Pod Pending, no gate | admitted, but no node has the GPU free or the class selector excludes nodes; a personal Pod will evict a preemptible one | pod conditions, get_capacity gpu inventory |
| Preemptible job restarted | evicted by a guaranteed Pod, or hit the 12 h attempt deadline; after 10 retries it fails | expected; checkpoint/resume is the fix |
Not authenticated to Kubernetes |
Coder kube token expired | the Coder external-auth link above |
OOMKilled, ImagePullBackOff in get_run nodes |
memory limit too low; image or registry problem | get_run_logs, raise memory, rebuild |
A process in the workspace died with Killed (exit 137) and no traceback, or the whole workspace restarted |
the workspace’s memory cgroup hit its limit: processes and /dev/shm share one ceiling |
coder stat mem, “Workspace memory” below |
Workspace memory (Coder)
A Coder workspace is one container with one memory cgroup: the shell, the
coding agent, Jupyter kernels, DataLoader workers and everything written to
/dev/shm share the limit set by the workspace’s memory parameter (GiB,
maximum 96; the default is lower, so check rather than assume). The request is
far below the limit (12Gi at most), so the limit is a hard ceiling, not reserved
memory. Crossing it makes the kernel OOM-kill the largest process in the
cgroup. Two outcomes, both common:
- A child dies (a
pythontraining script,pytest -n, abunbuild): the container survives and the user seesKilled(exit 137, signal 9) with no traceback.kubectl get podsshows nothing. - The agent or init process dies: the whole workspace restarts and every running process is gone.
Check headroom before starting anything heavy in the workspace, and whenever a process died without a trace:
coder stat mem(coder stat cpu;coder statfor all of it): usage against the cgroup limit, the same numbers the workspace page shows.cat /sys/fs/cgroup/memory.max /sys/fs/cgroup/memory.current: limit and usage in bytes.grep -E '^(anon|file|shmem) ' /sys/fs/cgroup/memory.statsplits usage into process memory, page cache and tmpfs/shm.grep oom_kill /sys/fs/cgroup/memory.eventscounts kills in this container so far.env | grep WORKSPACE_: workspaces built from the currentkubernetes-devtemplate exportWORKSPACE_MEMORY_LIMIT_GIBandWORKSPACE_CPU_LIMIT.
Memory-backed filesystems are charged to the same limit and the kernel cannot
reclaim them. /dev/shm is a 16Gi memory-backed emptyDir: a dataset unpacked
there, or tensors that many DataLoader workers share through it, take that
much away from the processes long before it reports No space left on device. df -h -t tmpfs lists what is memory-backed; du -sh /dev/shm shows
what is in it. Keep /dev/shm for shared memory between the processes of one
job and remove what is left when the job ends. Files go in the home directory
(a PVC, persists across restarts) or /tmp (the container’s writable layer on
node disk, lost on restart), never on a tmpfs; datasets are not copied into
the workspace at all when a job can stage them from S3 into /cache.
When the workspace is tight:
- Run the heavy step as a job (
build_and_submit_vxrun,run-batch). A job has its own memory limit, chosen per run, and never competes with the workspace. - In the workspace, lower
num_workers, batch size and in-memory caching before reaching for more memory. - If the work really belongs in the workspace, raise the
memoryparameter in the workspace settings (up to 96) and rebuild.
Storage
/output→/mnt/artifacts/<user>/<image>/<workflow>(NFS,run-imageonly): durable job output; fine for small or infrequent checkpoints. NFS is legacy; nothing new should depend on/mnt/*.run-batch: no NFS mounts./outputis a temporary per-item claim on the node’s NVMe, published tos3://workflow-outputby thepackstep and read back withvxrun batch results. Faster than NFS and the path for new work.s3://scratch/<username>/<experiment-or-workflow>/(Ceph RGW,[scratch]profile in Coder and jobs): temporary development data, experiments and checkpoints, with a two-week (14-day) GC policy and shared size quota. Frequent large checkpoint rewrites fit here better than on NFS. Scratch is not for permanent storage or repeatable workflow inputs and outputs. Once happy with the data, publish it as a permanent vxdata resource and use that resource in future workflows./cache: node-local scratch (--local-cache 64Gi, TopoLVM volume on the node’s NVMe), lost with the Pod.- Object storage via
vxdata-sdk/vxs3(Ceph RGW): permanent vxdata resources for datasets, model weights and published results. Keep both the resource and its data in vxdata; a resource pointing at scratch still loses its data to GC.vxrun storageuploads and optimises datasets through thevxoptimisenamespace. See read_doc(“vx/cluster/storage-practices”).
Builds
Tekton tag-image task / Argo buildah template produce the image; BuildKit
cache notes live in infra_k8s/docs/buildkit-cache.md. The Dockerfile’s
CMD/ENTRYPOINT is the default job command.
Repeatable runs
The Workflow carries kuberun.virdx.com/commit and the image digest; together
they are the only record of what a job executed. A run is repeatable when that
record is complete: every line of code in the image, every input immutable.
Three things break it, and the tools treat them differently:
- Code executed from
/mnt/*or the home directory (python /mnt/storage/users/x/train.py,PYTHONPATH=/mnt/scratch/lib). Whoever edits the file changes the next run of the same commit.vxrunwarns on such a command (build_and_submit_vxrundry run andvxrun plan:warnings, prefix “Not repeatable”; DAG validation: a per-task warning). Nothing is blocked yet. Fix: move the code into the repository. - Code pulled in at run time (
sys.path.insertinto a mount,git clone,curl | sh,pip installincmd).review_jobreports these as high findings (code-from-shared-mount,code-fetch-at-runtime,runtime-install). Fix: vendor or install in the Dockerfile at a pinned revision. - An environment that drifts between builds: untagged or
:latestbase image, unpinned fetches orpip installin the Dockerfile.review_jobreports these from the Dockerfile (unpinned-base-image,dockerfile-fetches-at-build,unpinned-dependency-install).
Data may live outside the image, but only as published, immutable inputs:
vxdata resources staged into /cache, not scratch (GC) or a mutable NFS path.