---
title: "2026-09-01 Factory histo preprocessing S3 PR review"
description: "Adversarial review of run-scoped S3 staging, vxData registration, and completion semantics."
image: "https://docs.virdx.dev/img/virdx-social-card.png"
---

> Documentation Index
> Fetch the complete documentation index at: https://docs.virdx.dev/llms.txt
> Use this file to discover all available pages before exploring further.

# 2026-09-01 Factory histo preprocessing S3 PR review

## Work Done

- Reviewed draft PR [virdx/mono#361](https://github.com/virdx/mono/pull/361) against `origin/main` without editing mono.
- Traced the factory runner, service completion predicate, dispatch batching, ad-hoc and rule callers, vxData SDK directory upload mapping, strict create/read-after-write behavior, and the vxData API create transaction.
- Verified that uploading one run directory returns a trailing-slash S3 root whose tree matches the runner's relative URL mapping. The distinct `preprocessing_tissue_mask` identifier also avoids taking ownership of source-provided foreground masks.
- Found that registration is split across strict create transactions and writes `component_0` before its maps and later components. A crash can therefore leave a source silently classified complete with missing outputs, while a crash after the original preprocessing mask but before `component_0` makes retries fail on the existing strict-create identifier.
- Found that `histo` 0.0.107 can return an empty component list without raising. The runner accepts that result, uploads and possibly registers only the original mask, and returns success although both completion predicates require `component_0`; the next reconciliation repeats it.
- Found that each 25-slide job now retains all native-resolution WSI output trees under the container writable layer until the final directory upload. This creates an unrequested ephemeral-storage requirement and loses the whole batch's computed progress if staging fills or the pod is evicted before upload.
- Re-reviewed the staged fixes. Each scan now submits its mixed resource graph through one `client.resources.create` call, empty component lists fail before upload, and histo dispatch requests the deployed run-image template's 1024 GiB node-local `/cache` volume while setting `FACTORY_TMP_DIR=/cache`. Confirmed the live cluster template accepts `local_cache=1024Gi`, mounts its ephemeral PVC at `/cache`, and uses the TopoLVM-backed `lvm` storage class with XFS, `WaitForFirstConsumer`, and delete reclaim semantics. Factory Ruff, ty, format, compile, and pre-commit checks pass; no concrete review findings remain.
- Committed the final three-file fix as mono commit `e6570716` (`fix(factory): use local cache for histo preprocessing`) and pushed it to draft PR [virdx/mono#361](https://github.com/virdx/mono/pull/361).

## Lessons Learned: Pitfalls

- A resource used as a completion marker is only safe if it commits atomically with every output whose completion it represents. Checking that marker before work does not make a sequence of strict creates atomic or retryable.
- A strict create is useful as a concurrency authority only when losing the race has a complete winner or when retry logic verifies and repairs the state. It cannot heal a process that previously committed only the first create.
- One shared S3 prefix reduces namespace fragmentation, but forcing the whole batch to exist locally before allocation shifts durability and capacity risk to pod ephemeral storage.
- Leaving reviewed changes staged without immediately reporting their exact scope made the work hard to trace, even though the delay was intended to avoid committing before review completed.

## Lessons Learned: Improvements

- Factory documentation should define the atomic publication boundary for a completed case and state whether zero-tissue scans are failures or valid empty outputs.
- The deployment SOP should document and size the runner's temporary storage, including its 25-slide native-resolution peak, or require per-scan upload/publication instead.
- Report every material working-tree change as soon as it is made. Include the affected files, behavioral impact, validation state, and whether the diff is uncommitted, staged, committed, or pushed.

Source: https://docs.virdx.dev/knowledge/inbox/2026-09-01-histo-factory-s3-pr-review/index.mdx