---
title: "Metrics"
image: "https://docs.virdx.dev/img/virdx-social-card.png"
---

> Documentation Index
> Fetch the complete documentation index at: https://docs.virdx.dev/llms.txt
> Use this file to discover all available pages before exploring further.

# Metrics

How to read what the cluster and a job are doing from Prometheus, as the user:
the path and identity, what the tools return, the recording rules, the raw
series and labels behind them, the dashboards, and example PromQL.

## Path and identity

Every Prometheus on the platform (the VM pair and the in-cluster
kube-prometheus-stack, plus the S3-archived history) is merged by Thanos Query and
fronted by Grafana at `https://grafana.fra.virdx.dev`. Grafana accepts a Keycloak
token through its JWT auth (`[auth.jwt]`, header `X-JWT-Assertion`, verified against
the virdx realm JWKS, only the issuer claim is checked, Grafana user = the token's
`preferred_username`), so `query_metrics`, `get_capacity`, `get_capacity(window)`,
`get_run_logs`, `vxrun metrics`, `vxrun gpus`, `vxrun capacity --history` and `vxrun logs`
POST to Grafana's
`/api/ds/query` and run under the user's own account with Viewer rights. In a Coder
workspace the token comes from `coder external-auth access-token k3s --extra id_token`
(the access token the kubeconfig helper uses is rejected by Grafana); on a laptop it
is the kubelogin token from the kubeconfig; `grafana_token_command` overrides both.
There is no separate credential and nothing to configure: `grafana_url` (default
`https://grafana.fra.virdx.dev`), `grafana_datasource` (`prometheus`, the Thanos
Query on the Grafana host) and `grafana_logs_datasource` (`loki`) in the user's
`~/.vxrun/config.yaml` are the only knobs, and a project `.vxrun.yaml` cannot set
them. Laptops need the VPN; Coder workspaces reach Grafana directly. vxrun ships the
Virdx IPA CA that signs the certificate; `grafana_ca_file` adds another bundle.

An `auth` error from these tools means the token was rejected: ask the user to run
`kubectl auth whoami` (refreshes the login). If that works and metrics still fail,
Grafana's JWT auth is off on the platform side: file a `send_feedback` (docs).

### Retention and query limits

| Store | Keeps |
|---|---|
| In-cluster Prometheus (two replicas, Thanos deduplicates) | 90 days, 500Gi NVMe |
| Thanos object store (Ceph RGW) | raw samples 365 days, 5-minute downsamples 730 days |
| Loki (logs) | 90 days |

Range queries return at most 200 points: the step defaults to `window / 200` (at
least 15s), so a 30-day range comes back at about 3.6-hour resolution; pass `step`
for finer data over a shorter window. Recording rules evaluate every 30s to 60s,
so a job shorter than a minute may have no samples. One request times out after 30s;
split very wide or high-cardinality queries.

## What `query_metrics` returns

With `workflow`: per Pod of that Workflow over `since` (default 1h; widen it for a
job that finished earlier), `cpu_cores_avg` and `cpu_request`, `memory_bytes_max`
and `memory_request`, the derived `cpu_utilisation` and `memory_utilisation`
(fractions of the request), `gpu_request`, and `gpus`: per physical GPU index
`gpu_util_avg`, `gpu_util_max` (percent) and `gpu_memory_bytes_max`. `notes`
explains empty results (Pod not yet scheduled, finished before the window, or no
DCGM samples). The CPU average is a `rate()` over the window, so it is not diluted
when the Pod ran for only part of it; the memory and GPU figures are peaks.

Reading it: memory utilisation under 0.3 with a large request means the request can
shrink (the rulebook's right-sizing fix); GPU utilisation averaging under 40 % while
CPU sits at its request points at a data-loading bottleneck (see
the review_job rulebook, NFS hot reads and `local_cache`); GPU memory far below
the card's size (Ada 48 GB, PRO 96 GB) means a smaller GPU class or a larger batch
would do. Requests are what `request_ratio` left of the limits (default 0.5), so a
utilisation of 1.5 against the request is still inside the limit.

With `expr`: raw PromQL. Instant at now unless `since` is set, then a range with
`step`. Series come back as `labels`, `points` (`[epoch_seconds, value]`) and
`value` (latest).

## What `get_capacity` returns

Every physical GPU on the cluster (`vxrun gpus` is the CLI form), joined from
DCGM to the Pod holding it, the Workflow and the submitter, with the figures
averaged over `since` (default 30m). `gpus` has one entry per card: `node`,
`gpu` (index on the node), `uuid`, `model`, `gpu_class` (`rtx-6000-pro`,
`rtx-6000`), `allocated`, `pod`, `namespace`, `workflow` (empty for a Coder
workspace), `username`, `priority_class` (`guaranteed` / `preemptible`),
`util_now`, `util_avg`, `util_max` (percent), `memory_used_bytes_max`,
`memory_total_bytes`, `power_avg_watts`, `temp_max_celsius`, `shared` (true for
a card the device plugin time-slices, with `slices` the replica count; null when
the plugin config could not be read) and `idle` (allocated and `util_avg` under
`idle_below`, default 5 %). `summary` per GPU
class: `total`, `allocated`, `free` (whole cards the scheduler can still hand
out), `idle`, `util_avg_allocated`, `shared_gpus`. `by_user`: GPUs, idle GPUs,
mean utilisation and workflows per submitter, most GPUs first. `nodes`: per node
the DCGM card count, `whole_allocatable`, `allocated`, `free`,
`shared_slices_allocatable` / `shared_slices_used` / `workspaces_sharing`,
`shared_gpu_indices` and `slices_per_gpu` from the device plugin config, and
`shared_gpus`. `user` filters `gpus` and `by_user` only; the summary stays
cluster-wide.

The join is DCGM's `exported_pod` / `exported_namespace`, set by the device plugin
for a Pod holding a whole card, matched against `kube_pod_info` (node, creator,
priority class) and `kube_pod_labels` (`virdx.dev/username`, `virdx.dev/gpu-class`)
for Pods that request `nvidia_com_gpu` or `nvidia_com_gpu_shared`. The window
aggregates are keyed by `(UUID, exported_pod)`, so a card that moved from one Pod
to the next inside the window reports the current Pod's figures. Time-sliced
cards (Coder's `nvidia.com/gpu.shared` replicas on the Ada node) carry no
`exported_pod`, so DCGM cannot say which workspace drives them. Which cards are
sliced comes from the device plugin ConfigMap `nvidia/mig-mixed`
(`infra_k8s/current/argo-cd/nvidia.yaml`): the node label
`nvidia.com/device-plugin.config` picks the key (`ada-shared` on compute-002),
whose `sharing.timeSlicing.resources[].devices` lists NVML indices, the same index
DCGM reports as `gpu`. Those cards are `shared` with their `slices`, show the
utilisation the slices generate, and `nodes` and `notes` give the pool per node.
If the ConfigMap is not readable, the pool is counted (DCGM cards beyond the
whole-GPU allocatable) but the cards are not named and `shared` is null. `vx run metrics WORKFLOW --json` returns the same
`node` and `uuid` per GPU, so a run's card can be looked up in this list.

Reading it: `free` above zero for a class means a personal-mode job of that
class can start now (quota permitting); `idle` cards with a `username` are held
but unused, the usual sign of a job stuck on data loading or waiting on a lock;
`util_avg_allocated` well under 50 % across a class means the cluster is
reserved rather than busy.

`get_capacity(window="24h")` adds, over the window, the recording rules
`virdxcluster:gpu:used_by_user` (per user), `virdxcluster:gpu:used`,
`virdxcluster:gpu:available_for_workflows`, both `pending_workloads` series and
`queue_wait:max_seconds`, each as the raw series plus latest / avg / max.

### Which question, which series

| Question | Use |
|---|---|
| Is the cluster busy right now? | `get_capacity` (Kueue state) and `sum(virdxcluster:gpu:used) / sum(virdxcluster:gpu:available_for_workflows)` |
| How busy was it over the last day/week? | `get_capacity(window="24h")`, or the same rules as a range query |
| Who holds the GPUs? | `get_capacity` (per card, with utilisation), or `virdxcluster:gpu:used_by_user{username, modelName}` |
| Are there free GPUs of a class right now? | `get_capacity` `summary.<class>.free` |
| How much of my quota am I using? | `virdx_usage:kueue_quota_ratio:by_queue{cluster_queue="user-<first>-<last>"}` |
| Why is the preemptible queue not admitting? | `virdx_usage:kueue_usage:by_queue{cluster_queue="preemptible"}` against the cohort's idle nominal quota; `get_capacity` fair_sharing |
| How long do jobs wait for admission? | `virdxcluster:queue_wait:max_seconds`, `virdxcluster:workflow:pending_age_seconds{name}`, `histogram_quantile(0.9, sum by (le) (rate(kueue_admission_wait_time_seconds_bucket[12h])))` |
| What did my job actually use? | `vx run metrics WORKFLOW --json` |
| Are allocated GPUs idle? | `get_capacity` (`idle` per card, `summary.<class>.idle`), or DCGM `DCGM_FI_DEV_GPU_UTIL` joined to the Pod (example below) |
| Which physical card did my run use? | `vx run metrics WORKFLOW --json` `gpus.<index>.node` / `uuid`; the same card in `get_capacity` |
| GPU-hours per user or project over a period | `virdx_usage:gpu_request:by_user_project_model` integrated with `sum_over_time` (example below) |
| Is `/cache` or a PVC running out? | `virdxcluster:vg:*` (TopoLVM volume group per node), `virdxcluster:pvc:*` |

## Logs (Loki)

Vector tails every Pod in `argo-workflows` and `vxoptimise` and ships the lines to
Loki, which keeps them 90 days. Argo deletes successful Pods about an hour after
completion, so Loki is the only source for a finished job. `get_run_logs` reads the
live Pod when there is one and falls back to Loki through the same Grafana path
(datasource uid `loki`); `since`, `grep` and `all_steps` query Loki directly.
Stream labels: `namespace`, `workflow`, `username`, `container`, `stream`
(stdout/stderr); `pod` and `node` are structured metadata, filterable with
`| pod="..."`. Pod name is per retry, so query by `workflow` and filter by
`container` or `pod` when needed. The Explore link returned with the lines is the
same one the Argo UI shows as "Workflow logs (Loki)".

```logql
{namespace="argo-workflows", workflow="run-1", container="main"} |= "Traceback"
{namespace="argo-workflows", username="jane.doe"} |= "CUDA out of memory"
```

## Recording rules

Three PrometheusRule files in `infra_k8s/current/monitoring/` define everything
with a `virdxcluster:` or `virdx_usage:` prefix. Prefer them for cluster-wide
questions: they already encode the right joins and are cheap over long windows.
All of them were present on the live Prometheus on 2026-10-07.

### `virdx-cluster-recording-rules.yaml` (prefix `virdxcluster:`)

Scope is the `argo-workflows` namespace unless noted. "Running" means
`kube_pod_status_phase{phase="Running"}`. `username` comes from the Workflow's
`virdx.dev/username` label through `kube_argoworkflow_created_at`, so it is the
submitter, not a Pod label.

| Rule | Labels | Meaning |
|---|---|---|
| `virdxcluster:cpu:used`, `virdxcluster:memory:used` | none | **Limits** (not requests) of Running workflow Pods: what the dashboard calls "reserved" |
| `virdxcluster:cpu:usage`, `virdxcluster:memory:usage` | none | What those Pods actually consume: CPU `rate(...[5m])` in cores, memory working set in bytes |
| `virdxcluster:gpu:used` | none | GPU **requests** of Running workflow Pods |
| `virdxcluster:gpu:used_by_model`, `virdxcluster:gpu:total_by_model` | `modelName` | GPUs in use (all namespaces) / allocatable, per DCGM model name (`NVIDIA RTX 6000 Ada Generation`, `NVIDIA RTX PRO 6000 Blackwell Server Edition`) |
| `virdxcluster:gpu:used_by_user` | `username`, `modelName` | GPUs requested by Running workflow Pods, per submitter and model |
| `virdxcluster:cpu:used_by_user`, `virdxcluster:memory:used_by_user` | `username` | CPU / memory **limits** of Running workflow Pods per submitter |
| `virdxcluster:cpu:available_for_workflows`, `memory:...`, `gpu:...` | none | Allocatable across all nodes minus requests of Running Pods **outside** `argo-workflows` (Coder, infra). This is the ceiling workflows can use, not what is free now: free now = `available_for_workflows - used` |
| `virdxcluster:workloads:guaranteed`, `virdxcluster:workloads:preemptible` | none | Kueue-admitted active workloads, personal (every ClusterQueue except `preemptible`) vs the preemptible ClusterQueue |
| `virdxcluster:pending_workloads:guaranteed`, `...:preemptible` | none | Pending workloads per LocalQueue tier: preemptible matches `name=~"preemptible|user-.*-preemptible"` (the per-user preemptible LocalQueues and the legacy shared one), guaranteed is every other LocalQueue |
| `virdxcluster:queue_wait:max_seconds` | none | Age of the oldest Workflow still in phase Pending |
| `virdxcluster:workflow_pod:running_age_seconds` | `pod`, `username`, `priority_class`, `created_by_name` | How long each Running workflow Pod has run, with its tier (`guaranteed` / `preemptible`) |
| `virdxcluster:workflow:pending_age_seconds` | `name`, `username`, `queue_name` | How long each Pending Workflow has waited |
| `virdxcluster:pvc:capacity_bytes`, `virdxcluster:pvc:available_bytes` | `namespace`, `persistentvolumeclaim` | PVC size and headroom (kubelet stats, last seen within 30d) |
| `virdxcluster:vg:size_bytes`, `virdxcluster:vg:available_bytes` | `node` | TopoLVM volume group `kubevg` per node: the pool `local_cache` (`/cache`) volumes come from |

### `kueue-capacity-rules.yaml` (prefix `virdx_usage:kueue_`, 30s interval)

Per-queue quota accounting from Kueue's own metrics
(`metrics.enableClusterQueueResources` is on). Labels `cluster_queue`, `flavor`
(`shared` for everything current), `resource` (`cpu`, `memory` in bytes,
`nvidia.com/gpu`). Queue names: `user-<first>-<last>` personal, `preemptible` the
shared borrowing queue (nominal 0, so it has no ratio), `team-<name>` the team
queues (`ve2e`, `histo`, `diffsim`).

| Rule | Meaning |
|---|---|
| `virdx_usage:kueue_usage:by_queue` | admitted usage, borrowing included |
| `virdx_usage:kueue_quota:by_queue` | nominal quota (personal default 36 CPU / 264Gi / 2 GPUs) |
| `virdx_usage:kueue_quota_ratio:by_queue` | usage / nominal; above 1 means borrowing |
| `virdx_usage:team_gpu_usage`, `virdx_usage:team_gpu_quota` | team queues only (`team-ve2e`, `team-histo`, `team-diffsim`) |

### `virdx-usage-accounting-rules.yaml` (prefix `virdx_usage:`, 30s interval)

Allocated **requests** of Running workload containers (Argo wait sidecars and init
containers excluded) in `argo-workflows` and `coder-workspaces`, by `username`
(Pod label `virdx.dev/username`) and `project` (the image name under
`zot.fra.virdx.dev:5000/`, so `mono/apps/worker` style names for monorepo
projects; Coder dev images are `coder`; unattributable series are `unknown`).

| Rule | Labels |
|---|---|
| `virdx_usage:cpu_request_cores:by_user_project` | `username`, `project` |
| `virdx_usage:memory_request_bytes:by_user_project` | `username`, `project` |
| `virdx_usage:gpu_request:by_user_project_model` | `username`, `project`, `gpu_model` (node label `nvidia.com/gpu.product`; time-sliced Coder slices appear as `<product>-shared` and are never whole GPUs) |

These are gauges sampled every 30s, designed to be integrated:
`sum_over_time(rule[window]) * 30` gives resource-seconds. The `usage_report` tool
(`~/virdx/usage_report`) does this for core-hours, GB-hours and GPU-hours.

### Alerts that use them

`PersistentVolumeUsageHigh` (PVC under 20 % free), `VolumeGroupUsageHigh` (node
`kubevg` under 15 % free, so new `local_cache` volumes will fail),
`WorkflowCountQuotaNearLimit` (namespace Workflow count over 80 % of the 200 cap),
and Pod restart alerts. They fire to the platform team; a job that cannot get its
`/cache` volume or a Workflow refused by the count quota is the visible symptom.

## Raw series and labels

- **Pods and Workflows** (kube-state-metrics): Workflow Pods live in
  `argo-workflows`. `kube_pod_info{created_by_kind="Workflow",
  created_by_name="<workflow>", node, priority_class}` maps Pod to Workflow and
  node; use `last_over_time(...[window])` so finished Pods still match.
  `kube_pod_labels` exports the Pod labels `virdx.dev/username`,
  `virdx.dev/gpu-class` and `virdx.dev/preemptible` as `label_virdx_dev_username`
  and so on (an allowlist; other labels are not exported), and `kube_node_labels`
  exports `nvidia.com/gpu.product` as `label_nvidia_com_gpu_product`. Requests and
  limits are in `kube_pod_container_resource_requests` / `_limits` with
  `resource="cpu"|"memory"|"nvidia_com_gpu"|"nvidia_com_gpu_shared"`;
  `kube_pod_status_phase`, `kube_pod_start_time` and `kube_node_status_allocatable`
  complete the set.
- **Workflow objects**: `kube_argoworkflow_created_at`, `kube_argoworkflow_started_at`
  and `kube_argoworkflow_finished_at` (epoch seconds) carry `name`, `namespace`,
  `username`, `queue_name` (the Kueue LocalQueue) and `phase`. Submission-to-start
  latency is `started_at - created_at`; a Workflow with `phase="Pending"` and no
  `started_at` is still waiting.
- **Kueue**: `kueue_cluster_queue_resource_usage`, `kueue_cluster_queue_nominal_quota`,
  `kueue_cluster_queue_borrowing_limit` (`cluster_queue`, `flavor`, `resource`),
  `kueue_admitted_active_workloads{cluster_queue}`,
  `kueue_local_queue_pending_workloads{name, namespace}`,
  `kueue_admission_wait_time_seconds_bucket`.
- **Container usage** (cAdvisor): `container_cpu_usage_seconds_total` and
  `container_memory_working_set_bytes` with `container="main"` for the job itself
  (`wait` is the Argo sidecar).
- **GPU** (DCGM exporter, every GPU node): `DCGM_FI_DEV_GPU_UTIL` (%),
  `DCGM_FI_DEV_MEM_COPY_UTIL` (%), `DCGM_FI_DEV_FB_USED` / `_FB_FREE` (MiB),
  `DCGM_FI_DEV_POWER_USAGE` (W), `DCGM_FI_DEV_GPU_TEMP` and `_MEMORY_TEMP` (C),
  `DCGM_FI_DEV_SM_CLOCK` / `_MEM_CLOCK` (MHz), `DCGM_FI_DEV_XID_ERRORS`,
  `DCGM_FI_DEV_PCIE_REPLAY_COUNTER`, `DCGM_FI_DEV_TOTAL_ENERGY_CONSUMPTION` (mJ).
  The profiling fields (`DCGM_FI_PROF_*`, SM occupancy, tensor activity) are
  deliberately not exported: the profiling module crashes on the Ada node. Labels:
  `Hostname` (node), `gpu` (index on the node), `modelName`, `UUID`, and the Pod
  using the card as `exported_pod` / `exported_namespace` / `exported_container`.
  Join to `kube_pod_info` with `label_replace` to rename `pod` to `exported_pod`.
  A GPU with no `exported_pod` is unallocated.

## Dashboards

Provisioned by Ansible (`infra_ansible/roles/grafana/files/dashboards/`), all at
`https://grafana.fra.virdx.dev/d/<uid>`; they show the same numbers these tools
return, so quote them when a user wants a picture rather than a figure.

| Dashboard | uid | Shows |
|---|---|---|
| Cluster Scheduler Overview | `algxggw` | the `virdxcluster:` rules: CPU / memory / GPU reserved and used against `available_for_workflows`, running and pending Pods per tier, p90 Kueue admission wait, GPU allocation per user |
| NVIDIA DCGM Dashboard | `pgv-h42QjU_J` | per-GPU utilisation, memory, power, temperature, clocks |
| Argo Workflows Logs | `loki-argo-workflows` | Loki streams by workflow / user / container |
| Kubernetes / Views / Pods, Nodes | `k8s_views_pods`, `k8s_views_nodes` | per-Pod and per-node CPU, memory, network |

## Example expressions

```promql
# GPUs this user holds right now
sum(virdxcluster:gpu:used_by_user{username="jane.doe"})

# Cluster GPU allocation fraction over the day (range, since=24h)
sum(virdxcluster:gpu:used) / sum(virdxcluster:gpu:available_for_workflows)

# GPUs free for a new workflow Pod right now (admission still needs quota)
sum(virdxcluster:gpu:available_for_workflows) - sum(virdxcluster:gpu:used)

# This user's personal quota in use (1 = full; GPU is the usual limiter)
virdx_usage:kueue_quota_ratio:by_queue{cluster_queue="user-jane-doe"}

# Who is on the preemptible queue, and with what
virdx_usage:kueue_usage:by_queue{cluster_queue="preemptible"}

# Allocated GPUs that sit idle (under 5 % utilisation averaged over 30 minutes)
count(avg_over_time(DCGM_FI_DEV_GPU_UTIL{exported_pod!=""}[30m]) < 5)

# GPU-hours per user over the last 7 days (30s samples)
sum by (username, gpu_model) (sum_over_time(virdx_usage:gpu_request:by_user_project_model[7d])) * 30 / 3600

# Submission-to-start latency of a workflow, in seconds
kube_argoworkflow_started_at{name="run-1"} - on (name) kube_argoworkflow_created_at{name="run-1"}

# Peak memory of a workflow's main containers over 6h
max by (pod) (max_over_time(container_memory_working_set_bytes{namespace="argo-workflows", container="main"}[6h])
  * on (namespace, pod) group_left() max by (namespace, pod) (last_over_time(kube_pod_info{created_by_kind="Workflow", created_by_name="run-1"}[6h])))

# GPU utilisation per GPU of one Pod
avg by (gpu) (avg_over_time(DCGM_FI_DEV_GPU_UTIL{exported_pod="run-1-main-123"}[1h]))

# Headroom for local_cache volumes per node
virdxcluster:vg:available_bytes / virdxcluster:vg:size_bytes
```

## Sources and maintenance

Re-check this page when any of these change: the three rule files above,
`infra_k8s/current/argo-cd/kube-prometheus-stack.yaml` (kube-state-metrics label
allowlist and the `kube_argoworkflow_*` custom-resource metrics, retention),
`infra_k8s/current/argo-cd/nvidia.yaml` (the DCGM field list and the device plugin
time-slicing config),
`infra_k8s/current/monitoring/thanos-compactor.yaml` (object-store retention),
`infra_ansible/inventories/group_vars/grafana/grafana.yaml` (JWT auth, datasource
URLs) and `infra_ansible/roles/grafana/files/dashboards/`. The client side is
`mono/packages/vxrun/src/vxrun/metrics.py` (`CAPACITY_SERIES`, `job_queries`,
`gpu_queries`).

Source: https://docs.virdx.dev/vx/cluster/metrics/index.mdx
