Traced factory.services.histo_preprocessing.run: each Argo job handles up to 25 scans, downloads S3 sources temporarily, and now registers uploaded S3 output paths.
Traced vxData SDK and API upload allocation. Individual file calls allocate distinct timestamp-token prefixes. A single directory upload already uses presign/upload-dir and places the entire directory tree under one shared immutable prefix.
Implemented the runner as compute, one run-directory upload, then registration with generated S3 URLs. Every generated output now uploads regardless of whether its input came from disk or S3.
The temporary root is FACTORY_TMP_DIR, falling back to ~/.cache/factory/. Each job creates a microsecond-resolution timestamped directory. It removes failed per-scan partial trees before upload, keeps the run directory on overall failure, and deletes it after full success.
Reused one vxData client for the complete job. The SDK remains unchanged: uploading one parent directory already provides one shared immutable prefix for the job.
Removed _create_ignoring_existing. Commit adeb32a4 introduced it after the fixed original foreground-map identifier already existed while component resources did not; the foreground create failed before component registration, and the old outer loop swallowed the exception and reported success. Paul’s PR attributed the existing mask to an earlier or overlapping run, but its reproduction did not establish that provenance. Treating all create conflicts as success hid an identifier-ownership collision instead of making the caller choose create or update explicitly.
Inlined the one-use local input materialization and output registration helpers.
Replaced the misleading scan.preprocessing_descriptor/AssertionError skip with one batched in-job vxData query for each input’s canonical component_0 identifier. Existing components are now detected before source retrieval, S3 download, or preprocessing. The explicit creates remain the concurrency authority if another job wins after the preflight query.
Kept the completion predicates duplicated rather than introducing a shared abstraction. Both service.py and run.py now require a canonical component_0 whose descriptor is histo.preprocessing; the service collects all sibling components only after that completion marker exists.
Validated the predicate with raw scans covering complete component sets, a lone component_1, and a component_0 carrying the wrong descriptor. Factory formatting, Ruff lint, ty type checking, affected Nx targets, builds, and module boundaries pass.
Benchmarked exact-ID existence checks from the deployed Factory pod against the production vxData service, one reused client and 200 randomized calls per case. Median latency in milliseconds: generic resources.retrieve hit 1.798, miss 1.367; typed exact-ID query hit 1.679, miss 1.533. A ResourceNotFound miss is an HTTP 200 empty read followed by a local SDK exception; Python raise/catch cost measured 0.331 microseconds, negligible beside the 1–2 ms request.
Committed as mono 50a7d87b (feat(factory): upload histo preprocessing outputs to S3), pushed branch factory/histo-preprocessing-ceph-integration, and opened draft PR virdx/mono#361.
Lessons Learned: Pitfalls
S3 prefixes are not physical directories and do not themselves consume meaningful object-storage space. The real concern is namespace fragmentation, listing usability, and retry-created orphan objects.
Passing several directories or files to the current SDK is not equivalent to uploading one parent directory: upload planning allocates a new prefix for each top-level leaf. The shared-prefix behavior is reached only by one directory input.
Uploading before vxData metadata creation can leave orphan objects if registration fails; retry semantics need consideration even with run grouping.
Paul’s historical failure came from the CHIMERA RPE source data, not partial preprocessing integration. Production vxData contains exactly 190 histomap/CH.../tissue_mask resources matching the 190-scan cohort; all have task=FOREGROUND_MASK and generation_method=None. The reproduced CH00013 mask was created on 2026-08-11, before Paul’s 2026-08-14 fix. Paul’s revision already excluded correctly typed foreground maps from transformed inputs, but the routine independently generated another original mask with the same generic identifier, hit a create conflict before component registration, and swallowed the failure. Completion preflight uses /preprocessed/component_0; original-mask registration now queries the exact HistoMap identifier and creates only when absent, leaving source masks untouched.
preprocessing_descriptor belongs to the retrieved resource itself; a raw source keeps a null descriptor after derived components are created. It cannot detect whether the source was already processed. Factory completion is represented by derived component identifiers.
Lessons Learned: Improvements
The vxData SDK storage SOP should state explicitly that one directory argument is the unit of shared-prefix allocation, while separate top-level arguments receive separate allocations.
Factory preprocessing documentation should describe output persistence as the staged compute/upload/register flow now implemented in run.py.
