---
title: "2026-09-02 Factory backlog capacity audit"
description: "Audited Factory scheduling and commit boundaries for long-running histo and viseg backlogs."
image: "https://docs.virdx.dev/img/virdx-social-card.png"
---

> Documentation Index
> Fetch the complete documentation index at: https://docs.virdx.dev/llms.txt
> Use this file to discover all available pages before exploring further.

# 2026-09-02 Factory backlog capacity audit

## Work Done

- Audited `virdx/mono` Factory at `893169b4`, including core reconciliation, Argo submission payloads, rule registry, inference runners, vxData SDK transfer and create semantics, and the Factory README.
- Established current batch units: 25 histo slides or 64 viseg volumes per Workflow, with all backlog chunks submitted at once and no Factory-side concurrency cap.
- Identified durability boundaries: histo registers only after every slide in a Workflow has been attempted and one bulk directory upload completes; viseg commits after each pass of eight volumes.
- Identified retry and idempotency gaps, including concurrent sync races, strict resource-create conflicts, terminal failures being resubmitted forever, and completion predicates that do not verify full output bundles or processing parameters.
- Recommended a staged surgical design: bounded per-rule admission, a shared GPU backstop, per-slide histo commits, smaller viseg commit batches, low-priority preemptible pods with limited transient retries, and later durable poison-case backoff.

## Lessons Learned: Pitfalls

- The live `run-image` WorkflowTemplate is outside the mono checkout, so its resource limit and retry behavior could not be verified from the requested commit. The Executor integration was unavailable in this session.
- Existing histo workstream guidance was useful for libvips resource context but does not document Factory backlog scheduling or checkpoint boundaries.

## Lessons Learned: Improvements

- Add a Factory operations SOP that records Workflow case counts, per-pod execution order, output commit boundaries, in-flight detection, and the ownership/location of the `run-image` WorkflowTemplate.
- Document which knobs affect output identity versus only resource sizing. Current code comments claim completion considers all run knobs, while the actual histo and viseg predicates do not.
- Record the recommended ordering for long-backlog changes: cap admission first, make case outputs incremental, then enable low-priority preemption and limited transient retry.

Source: https://docs.virdx.dev/knowledge/inbox/2026-09-02-histo-factory-backlog-capacity-audit/index.mdx