---
title: "2026-09-02 Factory GPU controls audit"
description: "Audited live Argo and Kubernetes controls for limiting Factory GPU consumption without affecting other workloads."
image: "https://docs.virdx.dev/img/virdx-social-card.png"
---

> Documentation Index
> Fetch the complete documentation index at: https://docs.virdx.dev/llms.txt
> Use this file to discover all available pages before exploring further.

# 2026-09-02 Factory GPU controls audit

## Work Done

Audited the live `virdx` Kubernetes cluster with read-only `kubectl` and searched the `virdx/mono` and `virdx/infra_k8s` repositories. No audited repository or cluster resource was changed.

Confirmed that Factory runs from the `inference/inference-api` Deployment as ServiceAccount `inference-api-workflow-submitter` and submits labelled Workflows into `argo-workflows` through the `run-image` WorkflowTemplate. Factory GPU jobs currently request one full `nvidia.com/gpu` device. The cluster has 16 physical GPUs but only 14 schedulable full-GPU resources because two physical GPUs are partitioned into MIG resources.

Documented the live controls:

- `run-image` accepts 0, 1, or 2 GPUs and sets `runtimeClassName: nvidia`.
- Kyverno rejects more than 2 GPUs per workflow container.
- The Factory submitter's Kueue ClusterQueue has a 2-GPU nominal quota with no borrowing for normal jobs.
- `virdx.dev/preemptible: "true"` routes jobs to the shared zero-nominal, 14-GPU borrowing queue, assigns the `preemptible` PriorityClass, and injects a 12-hour deadline and ten retries.
- A live ConfigMap-backed Argo workflow semaphore limits the Factory submitter to 10 concurrent Workflows across normal and preemptible queues.
- `argo-workflows` has no GPU ResourceQuota. Shared controller and namespace parallelism controls are not suitable for a Factory-only limit.
- Nodes are untainted and there is no dedicated Factory node pool.

Recommended using the existing submitter semaphore as the immediate runtime dial. Its source is the self-healed `user-scheduling` ArgoCD application from `virdx/infra_iam`, so the durable adjustment belongs there. For stronger semantic isolation, add a second Factory-GPU-only semaphore to GPU workflows and preserve the existing per-user semaphore. A dedicated Factory Kueue queue is the exact resource-accounting alternative but requires explicit Kyverno pod queue routing and more careful quota ownership.

Recommended adding optional `virdx.dev/preemptible: "true"` metadata in `mono/apps/factory/src/factory/argo.py`. Existing Kueue reclamation should provide preemption instead of Kubernetes scheduler preemption, which is deliberately disabled for workflow PriorityClasses.

## Lessons Learned: Pitfalls

The local `infra_k8s` checkout was stale relative to the live ArgoCD revision. It still described the older inference image and older per-user quota defaults. Live resources and ArgoCD application status were required to avoid reporting obsolete values.

The phrase "16 GPUs" hides the distinction between physical cards and extended resources. Factory requests full `nvidia.com/gpu`, of which only 14 are schedulable; MIG resources are separate and unavailable through the current template request.

A namespace ResourceQuota cannot select Factory's workflow label. Adding a GPU quota to `argo-workflows` would affect every Argo user. A direct patch to the live semaphore ConfigMap would also be reverted by the `user-scheduling` application's self-heal.

The current semaphore counts Workflows, not GPUs. It is an exact GPU ceiling only while each selected Factory GPU Workflow requests one GPU. CPU-only Factory Workflows also consume the submitter-wide semaphore. Caller-supplied and Kyverno-injected semaphore lists must be tested through server-side admission before assuming both entries survive strategic merge.

Preemptible retries are mostly compatible with the services' stable identifiers and skip checks, but interruption after blob upload and before resource creation can leave orphan uploads and repeat transfer work.

## Lessons Learned: Improvements

The infrastructure SOP should state that user queue and workflow-semaphore manifests are owned by `virdx/infra_iam`, not `infra_k8s`, and that the live Factory identity is `inference-api-workflow-submitter`.

The cluster documentation should distinguish 16 physical GPUs from 14 full `nvidia.com/gpu` resources plus MIG resources. It should also record that the current preemptible path is Kueue eviction through quota reclamation, while both workflow PriorityClasses use `preemptionPolicy: Never`.

A future Factory control should use a Factory-specific label and semaphore for GPU jobs only. The admission test should verify that both the Factory semaphore and the injected per-user semaphore appear on the final Workflow.

Source: https://docs.virdx.dev/knowledge/inbox/2026-09-02-infrastructure-factory-gpu-controls-audit/index.mdx