How to read what the cluster and a job are doing from Prometheus, as the user: the path and identity, what the tools return, the recording rules, the raw series and labels behind them, the dashboards, and example PromQL.
Path and identity
Every Prometheus on the platform (the VM pair and the in-cluster
kube-prometheus-stack, plus the S3-archived history) is merged by Thanos Query and
fronted by Grafana at https://grafana.fra.virdx.dev. Grafana accepts a Keycloak
token through its JWT auth ([auth.jwt], header X-JWT-Assertion, verified against
the virdx realm JWKS, only the issuer claim is checked, Grafana user = the token’s
preferred_username), so query_metrics, get_capacity, get_capacity(window),
get_run_logs, vxrun metrics, vxrun gpus, vxrun capacity --history and vxrun logs
POST to Grafana’s
/api/ds/query and run under the user’s own account with Viewer rights. In a Coder
workspace the token comes from coder external-auth access-token k3s --extra id_token
(the access token the kubeconfig helper uses is rejected by Grafana); on a laptop it
is the kubelogin token from the kubeconfig; grafana_token_command overrides both.
There is no separate credential and nothing to configure: grafana_url (default
https://grafana.fra.virdx.dev), grafana_datasource (prometheus, the Thanos
Query on the Grafana host) and grafana_logs_datasource (loki) in the user’s
~/.vxrun/config.yaml are the only knobs, and a project .vxrun.yaml cannot set
them. Laptops need the VPN; Coder workspaces reach Grafana directly. vxrun ships the
Virdx IPA CA that signs the certificate; grafana_ca_file adds another bundle.
An auth error from these tools means the token was rejected: ask the user to run
kubectl auth whoami (refreshes the login). If that works and metrics still fail,
Grafana’s JWT auth is off on the platform side: file a send_feedback (docs).
Retention and query limits
| Store | Keeps |
|---|---|
| In-cluster Prometheus (two replicas, Thanos deduplicates) | 90 days, 500Gi NVMe |
| Thanos object store (Ceph RGW) | raw samples 365 days, 5-minute downsamples 730 days |
| Loki (logs) | 90 days |
Range queries return at most 200 points: the step defaults to window / 200 (at
least 15s), so a 30-day range comes back at about 3.6-hour resolution; pass step
for finer data over a shorter window. Recording rules evaluate every 30s to 60s,
so a job shorter than a minute may have no samples. One request times out after 30s;
split very wide or high-cardinality queries.
What query_metrics returns
With workflow: per Pod of that Workflow over since (default 1h; widen it for a
job that finished earlier), cpu_cores_avg and cpu_request, memory_bytes_max
and memory_request, the derived cpu_utilisation and memory_utilisation
(fractions of the request), gpu_request, and gpus: per physical GPU index
gpu_util_avg, gpu_util_max (percent) and gpu_memory_bytes_max. notes
explains empty results (Pod not yet scheduled, finished before the window, or no
DCGM samples). The CPU average is a rate() over the window, so it is not diluted
when the Pod ran for only part of it; the memory and GPU figures are peaks.
Reading it: memory utilisation under 0.3 with a large request means the request can
shrink (the rulebook’s right-sizing fix); GPU utilisation averaging under 40 % while
CPU sits at its request points at a data-loading bottleneck (see
the review_job rulebook, NFS hot reads and local_cache); GPU memory far below
the card’s size (Ada 48 GB, PRO 96 GB) means a smaller GPU class or a larger batch
would do. Requests are what request_ratio left of the limits (default 0.5), so a
utilisation of 1.5 against the request is still inside the limit.
With expr: raw PromQL. Instant at now unless since is set, then a range with
step. Series come back as labels, points ([epoch_seconds, value]) and
value (latest).
What get_capacity returns
Every physical GPU on the cluster (vxrun gpus is the CLI form), joined from
DCGM to the Pod holding it, the Workflow and the submitter, with the figures
averaged over since (default 30m). gpus has one entry per card: node,
gpu (index on the node), uuid, model, gpu_class (rtx-6000-pro,
rtx-6000), allocated, pod, namespace, workflow (empty for a Coder
workspace), username, priority_class (guaranteed / preemptible),
util_now, util_avg, util_max (percent), memory_used_bytes_max,
memory_total_bytes, power_avg_watts, temp_max_celsius, shared (true for
a card the device plugin time-slices, with slices the replica count; null when
the plugin config could not be read) and idle (allocated and util_avg under
idle_below, default 5 %). summary per GPU
class: total, allocated, free (whole cards the scheduler can still hand
out), idle, util_avg_allocated, shared_gpus. by_user: GPUs, idle GPUs,
mean utilisation and workflows per submitter, most GPUs first. nodes: per node
the DCGM card count, whole_allocatable, allocated, free,
shared_slices_allocatable / shared_slices_used / workspaces_sharing,
shared_gpu_indices and slices_per_gpu from the device plugin config, and
shared_gpus. user filters gpus and by_user only; the summary stays
cluster-wide.
The join is DCGM’s exported_pod / exported_namespace, set by the device plugin
for a Pod holding a whole card, matched against kube_pod_info (node, creator,
priority class) and kube_pod_labels (virdx.dev/username, virdx.dev/gpu-class)
for Pods that request nvidia_com_gpu or nvidia_com_gpu_shared. The window
aggregates are keyed by (UUID, exported_pod), so a card that moved from one Pod
to the next inside the window reports the current Pod’s figures. Time-sliced
cards (Coder’s nvidia.com/gpu.shared replicas on the Ada node) carry no
exported_pod, so DCGM cannot say which workspace drives them. Which cards are
sliced comes from the device plugin ConfigMap nvidia/mig-mixed
(infra_k8s/current/argo-cd/nvidia.yaml): the node label
nvidia.com/device-plugin.config picks the key (ada-shared on compute-002),
whose sharing.timeSlicing.resources[].devices lists NVML indices, the same index
DCGM reports as gpu. Those cards are shared with their slices, show the
utilisation the slices generate, and nodes and notes give the pool per node.
If the ConfigMap is not readable, the pool is counted (DCGM cards beyond the
whole-GPU allocatable) but the cards are not named and shared is null. vx run metrics WORKFLOW --json returns the same
node and uuid per GPU, so a run’s card can be looked up in this list.
Reading it: free above zero for a class means a personal-mode job of that
class can start now (quota permitting); idle cards with a username are held
but unused, the usual sign of a job stuck on data loading or waiting on a lock;
util_avg_allocated well under 50 % across a class means the cluster is
reserved rather than busy.
get_capacity(window="24h") adds, over the window, the recording rules
virdxcluster:gpu:used_by_user (per user), virdxcluster:gpu:used,
virdxcluster:gpu:available_for_workflows, both pending_workloads series and
queue_wait:max_seconds, each as the raw series plus latest / avg / max.
Which question, which series
| Question | Use |
|---|---|
| Is the cluster busy right now? | get_capacity (Kueue state) and sum(virdxcluster:gpu:used) / sum(virdxcluster:gpu:available_for_workflows) |
| How busy was it over the last day/week? | get_capacity(window="24h"), or the same rules as a range query |
| Who holds the GPUs? | get_capacity (per card, with utilisation), or virdxcluster:gpu:used_by_user{username, modelName} |
| Are there free GPUs of a class right now? | get_capacity summary.<class>.free |
| How much of my quota am I using? | virdx_usage:kueue_quota_ratio:by_queue{cluster_queue="user-<first>-<last>"} |
| Why is the preemptible queue not admitting? | virdx_usage:kueue_usage:by_queue{cluster_queue="preemptible"} against the cohort’s idle nominal quota; get_capacity fair_sharing |
| How long do jobs wait for admission? | virdxcluster:queue_wait:max_seconds, virdxcluster:workflow:pending_age_seconds{name}, histogram_quantile(0.9, sum by (le) (rate(kueue_admission_wait_time_seconds_bucket[12h]))) |
| What did my job actually use? | vx run metrics WORKFLOW --json |
| Are allocated GPUs idle? | get_capacity (idle per card, summary.<class>.idle), or DCGM DCGM_FI_DEV_GPU_UTIL joined to the Pod (example below) |
| Which physical card did my run use? | vx run metrics WORKFLOW --json gpus.<index>.node / uuid; the same card in get_capacity |
| GPU-hours per user or project over a period | virdx_usage:gpu_request:by_user_project_model integrated with sum_over_time (example below) |
Is /cache or a PVC running out? |
virdxcluster:vg:* (TopoLVM volume group per node), virdxcluster:pvc:* |
Logs (Loki)
Vector tails every Pod in argo-workflows and vxoptimise and ships the lines to
Loki, which keeps them 90 days. Argo deletes successful Pods about an hour after
completion, so Loki is the only source for a finished job. get_run_logs reads the
live Pod when there is one and falls back to Loki through the same Grafana path
(datasource uid loki); since, grep and all_steps query Loki directly.
Stream labels: namespace, workflow, username, container, stream
(stdout/stderr); pod and node are structured metadata, filterable with
| pod="...". Pod name is per retry, so query by workflow and filter by
container or pod when needed. The Explore link returned with the lines is the
same one the Argo UI shows as “Workflow logs (Loki)”.
{namespace="argo-workflows", workflow="run-1", container="main"} |= "Traceback"
{namespace="argo-workflows", username="jane.doe"} |= "CUDA out of memory"Recording rules
Three PrometheusRule files in infra_k8s/current/monitoring/ define everything
with a virdxcluster: or virdx_usage: prefix. Prefer them for cluster-wide
questions: they already encode the right joins and are cheap over long windows.
All of them were present on the live Prometheus on 2026-10-07.
virdx-cluster-recording-rules.yaml (prefix virdxcluster:)
Scope is the argo-workflows namespace unless noted. “Running” means
kube_pod_status_phase{phase="Running"}. username comes from the Workflow’s
virdx.dev/username label through kube_argoworkflow_created_at, so it is the
submitter, not a Pod label.
| Rule | Labels | Meaning |
|---|---|---|
virdxcluster:cpu:used, virdxcluster:memory:used |
none | Limits (not requests) of Running workflow Pods: what the dashboard calls “reserved” |
virdxcluster:cpu:usage, virdxcluster:memory:usage |
none | What those Pods actually consume: CPU rate(...[5m]) in cores, memory working set in bytes |
virdxcluster:gpu:used |
none | GPU requests of Running workflow Pods |
virdxcluster:gpu:used_by_model, virdxcluster:gpu:total_by_model |
modelName |
GPUs in use (all namespaces) / allocatable, per DCGM model name (NVIDIA RTX 6000 Ada Generation, NVIDIA RTX PRO 6000 Blackwell Server Edition) |
virdxcluster:gpu:used_by_user |
username, modelName |
GPUs requested by Running workflow Pods, per submitter and model |
virdxcluster:cpu:used_by_user, virdxcluster:memory:used_by_user |
username |
CPU / memory limits of Running workflow Pods per submitter |
virdxcluster:cpu:available_for_workflows, memory:..., gpu:... |
none | Allocatable across all nodes minus requests of Running Pods outside argo-workflows (Coder, infra). This is the ceiling workflows can use, not what is free now: free now = available_for_workflows - used |
virdxcluster:workloads:guaranteed, virdxcluster:workloads:preemptible |
none | Kueue-admitted active workloads, personal (every ClusterQueue except preemptible) vs the preemptible ClusterQueue |
virdxcluster:pending_workloads:guaranteed, ...:preemptible |
none | Pending workloads per LocalQueue tier: preemptible matches `name=~“preemptible |
virdxcluster:queue_wait:max_seconds |
none | Age of the oldest Workflow still in phase Pending |
virdxcluster:workflow_pod:running_age_seconds |
pod, username, priority_class, created_by_name |
How long each Running workflow Pod has run, with its tier (guaranteed / preemptible) |
virdxcluster:workflow:pending_age_seconds |
name, username, queue_name |
How long each Pending Workflow has waited |
virdxcluster:pvc:capacity_bytes, virdxcluster:pvc:available_bytes |
namespace, persistentvolumeclaim |
PVC size and headroom (kubelet stats, last seen within 30d) |
virdxcluster:vg:size_bytes, virdxcluster:vg:available_bytes |
node |
TopoLVM volume group kubevg per node: the pool local_cache (/cache) volumes come from |
kueue-capacity-rules.yaml (prefix virdx_usage:kueue_, 30s interval)
Per-queue quota accounting from Kueue’s own metrics
(metrics.enableClusterQueueResources is on). Labels cluster_queue, flavor
(shared for everything current), resource (cpu, memory in bytes,
nvidia.com/gpu). Queue names: user-<first>-<last> personal, preemptible the
shared borrowing queue (nominal 0, so it has no ratio), team-<name> the team
queues (ve2e, histo, diffsim).
| Rule | Meaning |
|---|---|
virdx_usage:kueue_usage:by_queue |
admitted usage, borrowing included |
virdx_usage:kueue_quota:by_queue |
nominal quota (personal default 36 CPU / 264Gi / 2 GPUs) |
virdx_usage:kueue_quota_ratio:by_queue |
usage / nominal; above 1 means borrowing |
virdx_usage:team_gpu_usage, virdx_usage:team_gpu_quota |
team queues only (team-ve2e, team-histo, team-diffsim) |
virdx-usage-accounting-rules.yaml (prefix virdx_usage:, 30s interval)
Allocated requests of Running workload containers (Argo wait sidecars and init
containers excluded) in argo-workflows and coder-workspaces, by username
(Pod label virdx.dev/username) and project (the image name under
zot.fra.virdx.dev:5000/, so mono/apps/worker style names for monorepo
projects; Coder dev images are coder; unattributable series are unknown).
| Rule | Labels |
|---|---|
virdx_usage:cpu_request_cores:by_user_project |
username, project |
virdx_usage:memory_request_bytes:by_user_project |
username, project |
virdx_usage:gpu_request:by_user_project_model |
username, project, gpu_model (node label nvidia.com/gpu.product; time-sliced Coder slices appear as <product>-shared and are never whole GPUs) |
These are gauges sampled every 30s, designed to be integrated:
sum_over_time(rule[window]) * 30 gives resource-seconds. The usage_report tool
(~/virdx/usage_report) does this for core-hours, GB-hours and GPU-hours.
Alerts that use them
PersistentVolumeUsageHigh (PVC under 20 % free), VolumeGroupUsageHigh (node
kubevg under 15 % free, so new local_cache volumes will fail),
WorkflowCountQuotaNearLimit (namespace Workflow count over 80 % of the 200 cap),
and Pod restart alerts. They fire to the platform team; a job that cannot get its
/cache volume or a Workflow refused by the count quota is the visible symptom.
Raw series and labels
- Pods and Workflows (kube-state-metrics): Workflow Pods live in
argo-workflows.kube_pod_info{created_by_kind="Workflow", created_by_name="<workflow>", node, priority_class}maps Pod to Workflow and node; uselast_over_time(...[window])so finished Pods still match.kube_pod_labelsexports the Pod labelsvirdx.dev/username,virdx.dev/gpu-classandvirdx.dev/preemptibleaslabel_virdx_dev_usernameand so on (an allowlist; other labels are not exported), andkube_node_labelsexportsnvidia.com/gpu.productaslabel_nvidia_com_gpu_product. Requests and limits are inkube_pod_container_resource_requests/_limitswithresource="cpu"|"memory"|"nvidia_com_gpu"|"nvidia_com_gpu_shared";kube_pod_status_phase,kube_pod_start_timeandkube_node_status_allocatablecomplete the set. - Workflow objects:
kube_argoworkflow_created_at,kube_argoworkflow_started_atandkube_argoworkflow_finished_at(epoch seconds) carryname,namespace,username,queue_name(the Kueue LocalQueue) andphase. Submission-to-start latency isstarted_at - created_at; a Workflow withphase="Pending"and nostarted_atis still waiting. - Kueue:
kueue_cluster_queue_resource_usage,kueue_cluster_queue_nominal_quota,kueue_cluster_queue_borrowing_limit(cluster_queue,flavor,resource),kueue_admitted_active_workloads{cluster_queue},kueue_local_queue_pending_workloads{name, namespace},kueue_admission_wait_time_seconds_bucket. - Container usage (cAdvisor):
container_cpu_usage_seconds_totalandcontainer_memory_working_set_byteswithcontainer="main"for the job itself (waitis the Argo sidecar). - GPU (DCGM exporter, every GPU node):
DCGM_FI_DEV_GPU_UTIL(%),DCGM_FI_DEV_MEM_COPY_UTIL(%),DCGM_FI_DEV_FB_USED/_FB_FREE(MiB),DCGM_FI_DEV_POWER_USAGE(W),DCGM_FI_DEV_GPU_TEMPand_MEMORY_TEMP(C),DCGM_FI_DEV_SM_CLOCK/_MEM_CLOCK(MHz),DCGM_FI_DEV_XID_ERRORS,DCGM_FI_DEV_PCIE_REPLAY_COUNTER,DCGM_FI_DEV_TOTAL_ENERGY_CONSUMPTION(mJ). The profiling fields (DCGM_FI_PROF_*, SM occupancy, tensor activity) are deliberately not exported: the profiling module crashes on the Ada node. Labels:Hostname(node),gpu(index on the node),modelName,UUID, and the Pod using the card asexported_pod/exported_namespace/exported_container. Join tokube_pod_infowithlabel_replaceto renamepodtoexported_pod. A GPU with noexported_podis unallocated.
Dashboards
Provisioned by Ansible (infra_ansible/roles/grafana/files/dashboards/), all at
https://grafana.fra.virdx.dev/d/<uid>; they show the same numbers these tools
return, so quote them when a user wants a picture rather than a figure.
| Dashboard | uid | Shows |
|---|---|---|
| Cluster Scheduler Overview | algxggw |
the virdxcluster: rules: CPU / memory / GPU reserved and used against available_for_workflows, running and pending Pods per tier, p90 Kueue admission wait, GPU allocation per user |
| NVIDIA DCGM Dashboard | pgv-h42QjU_J |
per-GPU utilisation, memory, power, temperature, clocks |
| Argo Workflows Logs | loki-argo-workflows |
Loki streams by workflow / user / container |
| Kubernetes / Views / Pods, Nodes | k8s_views_pods, k8s_views_nodes |
per-Pod and per-node CPU, memory, network |
Example expressions
# GPUs this user holds right now
sum(virdxcluster:gpu:used_by_user{username="jane.doe"})
# Cluster GPU allocation fraction over the day (range, since=24h)
sum(virdxcluster:gpu:used) / sum(virdxcluster:gpu:available_for_workflows)
# GPUs free for a new workflow Pod right now (admission still needs quota)
sum(virdxcluster:gpu:available_for_workflows) - sum(virdxcluster:gpu:used)
# This user's personal quota in use (1 = full; GPU is the usual limiter)
virdx_usage:kueue_quota_ratio:by_queue{cluster_queue="user-jane-doe"}
# Who is on the preemptible queue, and with what
virdx_usage:kueue_usage:by_queue{cluster_queue="preemptible"}
# Allocated GPUs that sit idle (under 5 % utilisation averaged over 30 minutes)
count(avg_over_time(DCGM_FI_DEV_GPU_UTIL{exported_pod!=""}[30m]) < 5)
# GPU-hours per user over the last 7 days (30s samples)
sum by (username, gpu_model) (sum_over_time(virdx_usage:gpu_request:by_user_project_model[7d])) * 30 / 3600
# Submission-to-start latency of a workflow, in seconds
kube_argoworkflow_started_at{name="run-1"} - on (name) kube_argoworkflow_created_at{name="run-1"}
# Peak memory of a workflow's main containers over 6h
max by (pod) (max_over_time(container_memory_working_set_bytes{namespace="argo-workflows", container="main"}[6h])
* on (namespace, pod) group_left() max by (namespace, pod) (last_over_time(kube_pod_info{created_by_kind="Workflow", created_by_name="run-1"}[6h])))
# GPU utilisation per GPU of one Pod
avg by (gpu) (avg_over_time(DCGM_FI_DEV_GPU_UTIL{exported_pod="run-1-main-123"}[1h]))
# Headroom for local_cache volumes per node
virdxcluster:vg:available_bytes / virdxcluster:vg:size_bytesSources and maintenance
Re-check this page when any of these change: the three rule files above,
infra_k8s/current/argo-cd/kube-prometheus-stack.yaml (kube-state-metrics label
allowlist and the kube_argoworkflow_* custom-resource metrics, retention),
infra_k8s/current/argo-cd/nvidia.yaml (the DCGM field list and the device plugin
time-slicing config),
infra_k8s/current/monitoring/thanos-compactor.yaml (object-store retention),
infra_ansible/inventories/group_vars/grafana/grafana.yaml (JWT auth, datasource
URLs) and infra_ansible/roles/grafana/files/dashboards/. The client side is
mono/packages/vxrun/src/vxrun/metrics.py (CAPACITY_SERIES, job_queries,
gpu_queries).