2026-09-02 Factory GPU controls audit

Work Done

Audited the live virdx Kubernetes cluster with read-only kubectl and searched the virdx/mono and virdx/infra_k8s repositories. No audited repository or cluster resource was changed.

Confirmed that Factory runs from the inference/inference-api Deployment as ServiceAccount inference-api-workflow-submitter and submits labelled Workflows into argo-workflows through the run-image WorkflowTemplate. Factory GPU jobs currently request one full nvidia.com/gpu device. The cluster has 16 physical GPUs but only 14 schedulable full-GPU resources because two physical GPUs are partitioned into MIG resources.

Documented the live controls:

• run-image accepts 0, 1, or 2 GPUs and sets runtimeClassName: nvidia.
• Kyverno rejects more than 2 GPUs per workflow container.
• The Factory submitter’s Kueue ClusterQueue has a 2-GPU nominal quota with no borrowing for normal jobs.
• virdx.dev/preemptible: "true" routes jobs to the shared zero-nominal, 14-GPU borrowing queue, assigns the preemptible PriorityClass, and injects a 12-hour deadline and ten retries.
• A live ConfigMap-backed Argo workflow semaphore limits the Factory submitter to 10 concurrent Workflows across normal and preemptible queues.
• argo-workflows has no GPU ResourceQuota. Shared controller and namespace parallelism controls are not suitable for a Factory-only limit.
• Nodes are untainted and there is no dedicated Factory node pool.

Recommended using the existing submitter semaphore as the immediate runtime dial. Its source is the self-healed user-scheduling ArgoCD application from virdx/infra_iam, so the durable adjustment belongs there. For stronger semantic isolation, add a second Factory-GPU-only semaphore to GPU workflows and preserve the existing per-user semaphore. A dedicated Factory Kueue queue is the exact resource-accounting alternative but requires explicit Kyverno pod queue routing and more careful quota ownership.

Recommended adding optional virdx.dev/preemptible: "true" metadata in mono/apps/factory/src/factory/argo.py. Existing Kueue reclamation should provide preemption instead of Kubernetes scheduler preemption, which is deliberately disabled for workflow PriorityClasses.

Lessons Learned: Pitfalls

The local infra_k8s checkout was stale relative to the live ArgoCD revision. It still described the older inference image and older per-user quota defaults. Live resources and ArgoCD application status were required to avoid reporting obsolete values.

The phrase “16 GPUs” hides the distinction between physical cards and extended resources. Factory requests full nvidia.com/gpu, of which only 14 are schedulable; MIG resources are separate and unavailable through the current template request.

A namespace ResourceQuota cannot select Factory’s workflow label. Adding a GPU quota to argo-workflows would affect every Argo user. A direct patch to the live semaphore ConfigMap would also be reverted by the user-scheduling application’s self-heal.

The current semaphore counts Workflows, not GPUs. It is an exact GPU ceiling only while each selected Factory GPU Workflow requests one GPU. CPU-only Factory Workflows also consume the submitter-wide semaphore. Caller-supplied and Kyverno-injected semaphore lists must be tested through server-side admission before assuming both entries survive strategic merge.

Preemptible retries are mostly compatible with the services’ stable identifiers and skip checks, but interruption after blob upload and before resource creation can leave orphan uploads and repeat transfer work.

Lessons Learned: Improvements

The infrastructure SOP should state that user queue and workflow-semaphore manifests are owned by virdx/infra_iam, not infra_k8s, and that the live Factory identity is inference-api-workflow-submitter.

The cluster documentation should distinguish 16 physical GPUs from 14 full nvidia.com/gpu resources plus MIG resources. It should also record that the current preemptible path is Kueue eviction through quota reclamation, while both workflow PriorityClasses use preemptionPolicy: Never.

A future Factory control should use a Factory-specific label and semaphore for GPU jobs only. The admission test should verify that both the Factory semaphore and the injected per-user semaphore appear on the final Workflow.