Skip to content
VIRDX

Metrics

How to read what the cluster and a job are doing from Prometheus, as the user: the path and identity, what the tools return, the recording rules, the raw series and labels behind them, the dashboards, and example PromQL.

Path and identity

Every Prometheus on the platform (the VM pair and the in-cluster kube-prometheus-stack, plus the S3-archived history) is merged by Thanos Query and fronted by Grafana at https://grafana.fra.virdx.dev. Grafana accepts a Keycloak token through its JWT auth ([auth.jwt], header X-JWT-Assertion, verified against the virdx realm JWKS, only the issuer claim is checked, Grafana user = the token’s preferred_username), so query_metrics, get_capacity, get_capacity(window), get_run_logs, vxrun metrics, vxrun gpus, vxrun capacity --history and vxrun logs POST to Grafana’s /api/ds/query and run under the user’s own account with Viewer rights. In a Coder workspace the token comes from coder external-auth access-token k3s --extra id_token (the access token the kubeconfig helper uses is rejected by Grafana); on a laptop it is the kubelogin token from the kubeconfig; grafana_token_command overrides both. There is no separate credential and nothing to configure: grafana_url (default https://grafana.fra.virdx.dev), grafana_datasource (prometheus, the Thanos Query on the Grafana host) and grafana_logs_datasource (loki) in the user’s ~/.vxrun/config.yaml are the only knobs, and a project .vxrun.yaml cannot set them. Laptops need the VPN; Coder workspaces reach Grafana directly. vxrun ships the Virdx IPA CA that signs the certificate; grafana_ca_file adds another bundle.

An auth error from these tools means the token was rejected: ask the user to run kubectl auth whoami (refreshes the login). If that works and metrics still fail, Grafana’s JWT auth is off on the platform side: file a send_feedback (docs).

Retention and query limits

Store Keeps
In-cluster Prometheus (two replicas, Thanos deduplicates) 90 days, 500Gi NVMe
Thanos object store (Ceph RGW) raw samples 365 days, 5-minute downsamples 730 days
Loki (logs) 90 days

Range queries return at most 200 points: the step defaults to window / 200 (at least 15s), so a 30-day range comes back at about 3.6-hour resolution; pass step for finer data over a shorter window. Recording rules evaluate every 30s to 60s, so a job shorter than a minute may have no samples. One request times out after 30s; split very wide or high-cardinality queries.

What query_metrics returns

With workflow: per Pod of that Workflow over since (default 1h; widen it for a job that finished earlier), cpu_cores_avg and cpu_request, memory_bytes_max and memory_request, the derived cpu_utilisation and memory_utilisation (fractions of the request), gpu_request, and gpus: per physical GPU index gpu_util_avg, gpu_util_max (percent) and gpu_memory_bytes_max. notes explains empty results (Pod not yet scheduled, finished before the window, or no DCGM samples). The CPU average is a rate() over the window, so it is not diluted when the Pod ran for only part of it; the memory and GPU figures are peaks.

Reading it: memory utilisation under 0.3 with a large request means the request can shrink (the rulebook’s right-sizing fix); GPU utilisation averaging under 40 % while CPU sits at its request points at a data-loading bottleneck (see the review_job rulebook, NFS hot reads and local_cache); GPU memory far below the card’s size (Ada 48 GB, PRO 96 GB) means a smaller GPU class or a larger batch would do. Requests are what request_ratio left of the limits (default 0.5), so a utilisation of 1.5 against the request is still inside the limit.

With expr: raw PromQL. Instant at now unless since is set, then a range with step. Series come back as labels, points ([epoch_seconds, value]) and value (latest).

What get_capacity returns

Every physical GPU on the cluster (vxrun gpus is the CLI form), joined from DCGM to the Pod holding it, the Workflow and the submitter, with the figures averaged over since (default 30m). gpus has one entry per card: node, gpu (index on the node), uuid, model, gpu_class (rtx-6000-pro, rtx-6000), allocated, pod, namespace, workflow (empty for a Coder workspace), username, priority_class (guaranteed / preemptible), util_now, util_avg, util_max (percent), memory_used_bytes_max, memory_total_bytes, power_avg_watts, temp_max_celsius, shared (true for a card the device plugin time-slices, with slices the replica count; null when the plugin config could not be read) and idle (allocated and util_avg under idle_below, default 5 %). summary per GPU class: total, allocated, free (whole cards the scheduler can still hand out), idle, util_avg_allocated, shared_gpus. by_user: GPUs, idle GPUs, mean utilisation and workflows per submitter, most GPUs first. nodes: per node the DCGM card count, whole_allocatable, allocated, free, shared_slices_allocatable / shared_slices_used / workspaces_sharing, shared_gpu_indices and slices_per_gpu from the device plugin config, and shared_gpus. user filters gpus and by_user only; the summary stays cluster-wide.

The join is DCGM’s exported_pod / exported_namespace, set by the device plugin for a Pod holding a whole card, matched against kube_pod_info (node, creator, priority class) and kube_pod_labels (virdx.dev/username, virdx.dev/gpu-class) for Pods that request nvidia_com_gpu or nvidia_com_gpu_shared. The window aggregates are keyed by (UUID, exported_pod), so a card that moved from one Pod to the next inside the window reports the current Pod’s figures. Time-sliced cards (Coder’s nvidia.com/gpu.shared replicas on the Ada node) carry no exported_pod, so DCGM cannot say which workspace drives them. Which cards are sliced comes from the device plugin ConfigMap nvidia/mig-mixed (infra_k8s/current/argo-cd/nvidia.yaml): the node label nvidia.com/device-plugin.config picks the key (ada-shared on compute-002), whose sharing.timeSlicing.resources[].devices lists NVML indices, the same index DCGM reports as gpu. Those cards are shared with their slices, show the utilisation the slices generate, and nodes and notes give the pool per node. If the ConfigMap is not readable, the pool is counted (DCGM cards beyond the whole-GPU allocatable) but the cards are not named and shared is null. vx run metrics WORKFLOW --json returns the same node and uuid per GPU, so a run’s card can be looked up in this list.

Reading it: free above zero for a class means a personal-mode job of that class can start now (quota permitting); idle cards with a username are held but unused, the usual sign of a job stuck on data loading or waiting on a lock; util_avg_allocated well under 50 % across a class means the cluster is reserved rather than busy.

get_capacity(window="24h") adds, over the window, the recording rules virdxcluster:gpu:used_by_user (per user), virdxcluster:gpu:used, virdxcluster:gpu:available_for_workflows, both pending_workloads series and queue_wait:max_seconds, each as the raw series plus latest / avg / max.

Which question, which series

Question Use
Is the cluster busy right now? get_capacity (Kueue state) and sum(virdxcluster:gpu:used) / sum(virdxcluster:gpu:available_for_workflows)
How busy was it over the last day/week? get_capacity(window="24h"), or the same rules as a range query
Who holds the GPUs? get_capacity (per card, with utilisation), or virdxcluster:gpu:used_by_user{username, modelName}
Are there free GPUs of a class right now? get_capacity summary.<class>.free
How much of my quota am I using? virdx_usage:kueue_quota_ratio:by_queue{cluster_queue="user-<first>-<last>"}
Why is the preemptible queue not admitting? virdx_usage:kueue_usage:by_queue{cluster_queue="preemptible"} against the cohort’s idle nominal quota; get_capacity fair_sharing
How long do jobs wait for admission? virdxcluster:queue_wait:max_seconds, virdxcluster:workflow:pending_age_seconds{name}, histogram_quantile(0.9, sum by (le) (rate(kueue_admission_wait_time_seconds_bucket[12h])))
What did my job actually use? vx run metrics WORKFLOW --json
Are allocated GPUs idle? get_capacity (idle per card, summary.<class>.idle), or DCGM DCGM_FI_DEV_GPU_UTIL joined to the Pod (example below)
Which physical card did my run use? vx run metrics WORKFLOW --json gpus.<index>.node / uuid; the same card in get_capacity
GPU-hours per user or project over a period virdx_usage:gpu_request:by_user_project_model integrated with sum_over_time (example below)
Is /cache or a PVC running out? virdxcluster:vg:* (TopoLVM volume group per node), virdxcluster:pvc:*

Logs (Loki)

Vector tails every Pod in argo-workflows and vxoptimise and ships the lines to Loki, which keeps them 90 days. Argo deletes successful Pods about an hour after completion, so Loki is the only source for a finished job. get_run_logs reads the live Pod when there is one and falls back to Loki through the same Grafana path (datasource uid loki); since, grep and all_steps query Loki directly. Stream labels: namespace, workflow, username, container, stream (stdout/stderr); pod and node are structured metadata, filterable with | pod="...". Pod name is per retry, so query by workflow and filter by container or pod when needed. The Explore link returned with the lines is the same one the Argo UI shows as “Workflow logs (Loki)”.

{namespace="argo-workflows", workflow="run-1", container="main"} |= "Traceback"
{namespace="argo-workflows", username="jane.doe"} |= "CUDA out of memory"

Recording rules

Three PrometheusRule files in infra_k8s/current/monitoring/ define everything with a virdxcluster: or virdx_usage: prefix. Prefer them for cluster-wide questions: they already encode the right joins and are cheap over long windows. All of them were present on the live Prometheus on 2026-10-07.

virdx-cluster-recording-rules.yaml (prefix virdxcluster:)

Scope is the argo-workflows namespace unless noted. “Running” means kube_pod_status_phase{phase="Running"}. username comes from the Workflow’s virdx.dev/username label through kube_argoworkflow_created_at, so it is the submitter, not a Pod label.

Rule Labels Meaning
virdxcluster:cpu:used, virdxcluster:memory:used none Limits (not requests) of Running workflow Pods: what the dashboard calls “reserved”
virdxcluster:cpu:usage, virdxcluster:memory:usage none What those Pods actually consume: CPU rate(...[5m]) in cores, memory working set in bytes
virdxcluster:gpu:used none GPU requests of Running workflow Pods
virdxcluster:gpu:used_by_model, virdxcluster:gpu:total_by_model modelName GPUs in use (all namespaces) / allocatable, per DCGM model name (NVIDIA RTX 6000 Ada Generation, NVIDIA RTX PRO 6000 Blackwell Server Edition)
virdxcluster:gpu:used_by_user username, modelName GPUs requested by Running workflow Pods, per submitter and model
virdxcluster:cpu:used_by_user, virdxcluster:memory:used_by_user username CPU / memory limits of Running workflow Pods per submitter
virdxcluster:cpu:available_for_workflows, memory:..., gpu:... none Allocatable across all nodes minus requests of Running Pods outside argo-workflows (Coder, infra). This is the ceiling workflows can use, not what is free now: free now = available_for_workflows - used
virdxcluster:workloads:guaranteed, virdxcluster:workloads:preemptible none Kueue-admitted active workloads, personal (every ClusterQueue except preemptible) vs the preemptible ClusterQueue
virdxcluster:pending_workloads:guaranteed, ...:preemptible none Pending workloads per LocalQueue tier: preemptible matches `name=~“preemptible
virdxcluster:queue_wait:max_seconds none Age of the oldest Workflow still in phase Pending
virdxcluster:workflow_pod:running_age_seconds pod, username, priority_class, created_by_name How long each Running workflow Pod has run, with its tier (guaranteed / preemptible)
virdxcluster:workflow:pending_age_seconds name, username, queue_name How long each Pending Workflow has waited
virdxcluster:pvc:capacity_bytes, virdxcluster:pvc:available_bytes namespace, persistentvolumeclaim PVC size and headroom (kubelet stats, last seen within 30d)
virdxcluster:vg:size_bytes, virdxcluster:vg:available_bytes node TopoLVM volume group kubevg per node: the pool local_cache (/cache) volumes come from

kueue-capacity-rules.yaml (prefix virdx_usage:kueue_, 30s interval)

Per-queue quota accounting from Kueue’s own metrics (metrics.enableClusterQueueResources is on). Labels cluster_queue, flavor (shared for everything current), resource (cpu, memory in bytes, nvidia.com/gpu). Queue names: user-<first>-<last> personal, preemptible the shared borrowing queue (nominal 0, so it has no ratio), team-<name> the team queues (ve2e, histo, diffsim).

Rule Meaning
virdx_usage:kueue_usage:by_queue admitted usage, borrowing included
virdx_usage:kueue_quota:by_queue nominal quota (personal default 36 CPU / 264Gi / 2 GPUs)
virdx_usage:kueue_quota_ratio:by_queue usage / nominal; above 1 means borrowing
virdx_usage:team_gpu_usage, virdx_usage:team_gpu_quota team queues only (team-ve2e, team-histo, team-diffsim)

virdx-usage-accounting-rules.yaml (prefix virdx_usage:, 30s interval)

Allocated requests of Running workload containers (Argo wait sidecars and init containers excluded) in argo-workflows and coder-workspaces, by username (Pod label virdx.dev/username) and project (the image name under zot.fra.virdx.dev:5000/, so mono/apps/worker style names for monorepo projects; Coder dev images are coder; unattributable series are unknown).

Rule Labels
virdx_usage:cpu_request_cores:by_user_project username, project
virdx_usage:memory_request_bytes:by_user_project username, project
virdx_usage:gpu_request:by_user_project_model username, project, gpu_model (node label nvidia.com/gpu.product; time-sliced Coder slices appear as <product>-shared and are never whole GPUs)

These are gauges sampled every 30s, designed to be integrated: sum_over_time(rule[window]) * 30 gives resource-seconds. The usage_report tool (~/virdx/usage_report) does this for core-hours, GB-hours and GPU-hours.

Alerts that use them

PersistentVolumeUsageHigh (PVC under 20 % free), VolumeGroupUsageHigh (node kubevg under 15 % free, so new local_cache volumes will fail), WorkflowCountQuotaNearLimit (namespace Workflow count over 80 % of the 200 cap), and Pod restart alerts. They fire to the platform team; a job that cannot get its /cache volume or a Workflow refused by the count quota is the visible symptom.

Raw series and labels

  • Pods and Workflows (kube-state-metrics): Workflow Pods live in argo-workflows. kube_pod_info{created_by_kind="Workflow", created_by_name="<workflow>", node, priority_class} maps Pod to Workflow and node; use last_over_time(...[window]) so finished Pods still match. kube_pod_labels exports the Pod labels virdx.dev/username, virdx.dev/gpu-class and virdx.dev/preemptible as label_virdx_dev_username and so on (an allowlist; other labels are not exported), and kube_node_labels exports nvidia.com/gpu.product as label_nvidia_com_gpu_product. Requests and limits are in kube_pod_container_resource_requests / _limits with resource="cpu"|"memory"|"nvidia_com_gpu"|"nvidia_com_gpu_shared"; kube_pod_status_phase, kube_pod_start_time and kube_node_status_allocatable complete the set.
  • Workflow objects: kube_argoworkflow_created_at, kube_argoworkflow_started_at and kube_argoworkflow_finished_at (epoch seconds) carry name, namespace, username, queue_name (the Kueue LocalQueue) and phase. Submission-to-start latency is started_at - created_at; a Workflow with phase="Pending" and no started_at is still waiting.
  • Kueue: kueue_cluster_queue_resource_usage, kueue_cluster_queue_nominal_quota, kueue_cluster_queue_borrowing_limit (cluster_queue, flavor, resource), kueue_admitted_active_workloads{cluster_queue}, kueue_local_queue_pending_workloads{name, namespace}, kueue_admission_wait_time_seconds_bucket.
  • Container usage (cAdvisor): container_cpu_usage_seconds_total and container_memory_working_set_bytes with container="main" for the job itself (wait is the Argo sidecar).
  • GPU (DCGM exporter, every GPU node): DCGM_FI_DEV_GPU_UTIL (%), DCGM_FI_DEV_MEM_COPY_UTIL (%), DCGM_FI_DEV_FB_USED / _FB_FREE (MiB), DCGM_FI_DEV_POWER_USAGE (W), DCGM_FI_DEV_GPU_TEMP and _MEMORY_TEMP (C), DCGM_FI_DEV_SM_CLOCK / _MEM_CLOCK (MHz), DCGM_FI_DEV_XID_ERRORS, DCGM_FI_DEV_PCIE_REPLAY_COUNTER, DCGM_FI_DEV_TOTAL_ENERGY_CONSUMPTION (mJ). The profiling fields (DCGM_FI_PROF_*, SM occupancy, tensor activity) are deliberately not exported: the profiling module crashes on the Ada node. Labels: Hostname (node), gpu (index on the node), modelName, UUID, and the Pod using the card as exported_pod / exported_namespace / exported_container. Join to kube_pod_info with label_replace to rename pod to exported_pod. A GPU with no exported_pod is unallocated.

Dashboards

Provisioned by Ansible (infra_ansible/roles/grafana/files/dashboards/), all at https://grafana.fra.virdx.dev/d/<uid>; they show the same numbers these tools return, so quote them when a user wants a picture rather than a figure.

Dashboard uid Shows
Cluster Scheduler Overview algxggw the virdxcluster: rules: CPU / memory / GPU reserved and used against available_for_workflows, running and pending Pods per tier, p90 Kueue admission wait, GPU allocation per user
NVIDIA DCGM Dashboard pgv-h42QjU_J per-GPU utilisation, memory, power, temperature, clocks
Argo Workflows Logs loki-argo-workflows Loki streams by workflow / user / container
Kubernetes / Views / Pods, Nodes k8s_views_pods, k8s_views_nodes per-Pod and per-node CPU, memory, network

Example expressions

# GPUs this user holds right now
sum(virdxcluster:gpu:used_by_user{username="jane.doe"})

# Cluster GPU allocation fraction over the day (range, since=24h)
sum(virdxcluster:gpu:used) / sum(virdxcluster:gpu:available_for_workflows)

# GPUs free for a new workflow Pod right now (admission still needs quota)
sum(virdxcluster:gpu:available_for_workflows) - sum(virdxcluster:gpu:used)

# This user's personal quota in use (1 = full; GPU is the usual limiter)
virdx_usage:kueue_quota_ratio:by_queue{cluster_queue="user-jane-doe"}

# Who is on the preemptible queue, and with what
virdx_usage:kueue_usage:by_queue{cluster_queue="preemptible"}

# Allocated GPUs that sit idle (under 5 % utilisation averaged over 30 minutes)
count(avg_over_time(DCGM_FI_DEV_GPU_UTIL{exported_pod!=""}[30m]) < 5)

# GPU-hours per user over the last 7 days (30s samples)
sum by (username, gpu_model) (sum_over_time(virdx_usage:gpu_request:by_user_project_model[7d])) * 30 / 3600

# Submission-to-start latency of a workflow, in seconds
kube_argoworkflow_started_at{name="run-1"} - on (name) kube_argoworkflow_created_at{name="run-1"}

# Peak memory of a workflow's main containers over 6h
max by (pod) (max_over_time(container_memory_working_set_bytes{namespace="argo-workflows", container="main"}[6h])
  * on (namespace, pod) group_left() max by (namespace, pod) (last_over_time(kube_pod_info{created_by_kind="Workflow", created_by_name="run-1"}[6h])))

# GPU utilisation per GPU of one Pod
avg by (gpu) (avg_over_time(DCGM_FI_DEV_GPU_UTIL{exported_pod="run-1-main-123"}[1h]))

# Headroom for local_cache volumes per node
virdxcluster:vg:available_bytes / virdxcluster:vg:size_bytes

Sources and maintenance

Re-check this page when any of these change: the three rule files above, infra_k8s/current/argo-cd/kube-prometheus-stack.yaml (kube-state-metrics label allowlist and the kube_argoworkflow_* custom-resource metrics, retention), infra_k8s/current/argo-cd/nvidia.yaml (the DCGM field list and the device plugin time-slicing config), infra_k8s/current/monitoring/thanos-compactor.yaml (object-store retention), infra_ansible/inventories/group_vars/grafana/grafana.yaml (JWT auth, datasource URLs) and infra_ansible/roles/grafana/files/dashboards/. The client side is mono/packages/vxrun/src/vxrun/metrics.py (CAPACITY_SERIES, job_queries, gpu_queries).

Navigation

Type to search…

↑↓ navigate↵ selectEsc close