Reviewed draft PR virdx/mono#361 against origin/main without editing mono.
Traced the factory runner, service completion predicate, dispatch batching, ad-hoc and rule callers, vxData SDK directory upload mapping, strict create/read-after-write behavior, and the vxData API create transaction.
Verified that uploading one run directory returns a trailing-slash S3 root whose tree matches the runner’s relative URL mapping. The distinct preprocessing_tissue_mask identifier also avoids taking ownership of source-provided foreground masks.
Found that registration is split across strict create transactions and writes component_0 before its maps and later components. A crash can therefore leave a source silently classified complete with missing outputs, while a crash after the original preprocessing mask but before component_0 makes retries fail on the existing strict-create identifier.
Found that histo 0.0.107 can return an empty component list without raising. The runner accepts that result, uploads and possibly registers only the original mask, and returns success although both completion predicates require component_0; the next reconciliation repeats it.
Found that each 25-slide job now retains all native-resolution WSI output trees under the container writable layer until the final directory upload. This creates an unrequested ephemeral-storage requirement and loses the whole batch’s computed progress if staging fills or the pod is evicted before upload.
Re-reviewed the staged fixes. Each scan now submits its mixed resource graph through one client.resources.create call, empty component lists fail before upload, and histo dispatch requests the deployed run-image template’s 1024 GiB node-local /cache volume while setting FACTORY_TMP_DIR=/cache. Confirmed the live cluster template accepts local_cache=1024Gi, mounts its ephemeral PVC at /cache, and uses the TopoLVM-backed lvm storage class with XFS, WaitForFirstConsumer, and delete reclaim semantics. Factory Ruff, ty, format, compile, and pre-commit checks pass; no concrete review findings remain.
Committed the final three-file fix as mono commit e6570716 (fix(factory): use local cache for histo preprocessing) and pushed it to draft PR virdx/mono#361.
Lessons Learned: Pitfalls
A resource used as a completion marker is only safe if it commits atomically with every output whose completion it represents. Checking that marker before work does not make a sequence of strict creates atomic or retryable.
A strict create is useful as a concurrency authority only when losing the race has a complete winner or when retry logic verifies and repairs the state. It cannot heal a process that previously committed only the first create.
One shared S3 prefix reduces namespace fragmentation, but forcing the whole batch to exist locally before allocation shifts durability and capacity risk to pod ephemeral storage.
Leaving reviewed changes staged without immediately reporting their exact scope made the work hard to trace, even though the delay was intended to avoid committing before review completed.
Lessons Learned: Improvements
Factory documentation should define the atomic publication boundary for a completed case and state whether zero-tissue scans are failures or valid empty outputs.
The deployment SOP should document and size the runner’s temporary storage, including its 25-slide native-resolution peak, or require per-scan upload/publication instead.
Report every material working-tree change as soon as it is made. Include the affected files, behavioral impact, validation state, and whether the diff is uncommitted, staged, committed, or pushed.
