Audited virdx/mono Factory at 893169b4, including core reconciliation, Argo submission payloads, rule registry, inference runners, vxData SDK transfer and create semantics, and the Factory README.
Established current batch units: 25 histo slides or 64 viseg volumes per Workflow, with all backlog chunks submitted at once and no Factory-side concurrency cap.
Identified durability boundaries: histo registers only after every slide in a Workflow has been attempted and one bulk directory upload completes; viseg commits after each pass of eight volumes.
Identified retry and idempotency gaps, including concurrent sync races, strict resource-create conflicts, terminal failures being resubmitted forever, and completion predicates that do not verify full output bundles or processing parameters.
Recommended a staged surgical design: bounded per-rule admission, a shared GPU backstop, per-slide histo commits, smaller viseg commit batches, low-priority preemptible pods with limited transient retries, and later durable poison-case backoff.
Lessons Learned: Pitfalls
The live run-image WorkflowTemplate is outside the mono checkout, so its resource limit and retry behavior could not be verified from the requested commit. The Executor integration was unavailable in this session.
Existing histo workstream guidance was useful for libvips resource context but does not document Factory backlog scheduling or checkpoint boundaries.
Lessons Learned: Improvements
Add a Factory operations SOP that records Workflow case counts, per-pod execution order, output commit boundaries, in-flight detection, and the ownership/location of the run-image WorkflowTemplate.
Document which knobs affect output identity versus only resource sizing. Current code comments claim completion considers all run knobs, while the actual histo and viseg predicates do not.
Record the recommended ordering for long-backlog changes: cap admission first, make case outputs incremental, then enable low-priority preemption and limited transient retry.
