hio S3-first storage redesign

Status: unmerged, not approved. histo branch feat/fk/s3-io (commits 39c3c1f2..f216f8da as of 2026-08-07) proposes a new packages/hio library — a physical-space raster abstraction backed by S3/Ceph — plus a histo.io seam that migrates existing readers/writers onto it. Four independent read-only adversarial reviews were run the same day (three sub-reviews plus one consolidated pass); this page merges their findings so a future agent doesn’t have to re-derive them from inbox/.

If you’re picking this branch up: read this whole page before touching the branch, then re-verify each item below still applies (or has been fixed) before treating it as current.

Why it isn’t ready to merge

All four reviews independently converged on the same class of problem: the implementation does not yet provide the safety properties its own README and proposal.md claim.

Transaction / concurrency defects (demonstrated, not just theorized)

• The library’s default overwrite=True write path mutates keys already referenced by published metadata — there is no snapshot isolation or atomic replacement, despite the README’s “commit is one small write, last” claim (true only for first publication under a single-writer/immutable-URI assumption).
• In-place overwrite remains publicly readable mid-write; a concurrent reader can observe a torn object (data from one writer, metadata from another).
• Existence-based resume is unsafe: it checks shape/dtype only, not a durably-bound geometry/layout/producer-identity fingerprint, and only holds under a single fenced writer (which nothing enforces). Demonstrated: a partial writer resumes across a changed physical geometry; a reopened writer with incompatible geometry reuses stale shards; reopening a writer against an all-fill shard cannot complete it.

Spatial / calibration defects (demonstrated)

• Image.read(region).shape can disagree with the declared output geometry for a requested level (e.g. non-integer Level.mpp(1.5) reports a shape/MPP inconsistent with the returned array).
• Non-square pyramid rounding stamps the x axis with the y axis’s downsample factor.
• OME-XML-only calibration is silently discarded by the TIFF adapter (only TIFF resolution tags are read).
• NGFF 0.5 dimension_names are absent from written metadata, despite the branch’s own references.md documenting the requirement.

Migration-specific regressions (main vs. feat/fk/s3-io)

• Panoptic input fingerprints are dropped.
• Gland-inference model windows are re-anchored at storage shard boundaries instead of the original coordinate frame (shard-dependent inference — same input, different result depending on how it’s chunked).
• The histo.io seam drops fill_value.

Evidence / benchmark validity

• The proposal’s own required production benchmark had not been run before several implementation decisions were already made.
• Benchmarks ran against a local loopback MinIO fixture only — useful as integration/API evidence, not as evidence for real S3/Ceph latency, throughput, retry, or concurrency tuning.
• At least one benchmarked “patch” path does not materialize the patch at all (non-materializing benchmark); batch fan-out is unbounded; some reported numbers are cache-confounded or not traceable to a reproducible artifact/counter in code. Labels like “GET”, “MB/s”, “patches/s” in the writeup don’t always mean what they say — trace every metric back to the code that counts it before trusting it.
• 35/35 hio unit tests and 12/12 seam tests pass, but the suites only cover happy-path stored levels/single-writer flows — they omit overwrite visibility, duplicate/concurrent writers, incompatible resume, all-fill resume, and read-shape/geometry agreement. Green CI here is not evidence of the properties above.

What “ready” looks like (recommendation from all four reviews)

Before merge or deployment:

1. Writes use immutable generations + conditional (CAS-style) dataset-scope publication — no post-commit mutation of a key another commit’s metadata already points at.
2. Writer fencing: only one writer can hold/complete a given generation.
3. Resume validates a complete creation/job manifest (geometry + layout + inputs + model/config fingerprint + code version), not shape/dtype alone.
4. Multi-array commits (e.g. panoptic semantic + instance maps) publish as one atomic dataset version.
5. Level resampling is truthful: read(region).shape matches declared geometry for every level policy, and each axis’s rounding is independent.
6. OME-NGFF 0.5 conformance (incl. dimension_names) checked against the official validator, not just visual inspection.
7. Benchmarks re-run against real S3/Ceph (not loopback MinIO only), with request/byte instrumentation, randomized order, explicit cache state, and a reproducible artifact per reported number.

Reference implementations to steal from rather than reinvent

Cited by the reviews as prior art hio’s abstraction should either adopt or explicitly justify diverging from: xarray, SpatialData (heterogeneous coordinate models), TensorStore / Icechunk (proven async data plane + immutable-snapshot transactions), and OME-NGFF (metadata/validator tooling).

Process note

Don’t repeat this: three sub-agents each filed a separate narrow inbox recap before the consolidated review landed, which briefly fragmented this finding across four inbox/ entries. When several subagents adversarially review the same branch in parallel, only the parent/consolidating agent should write the durable knowledge-base entry.

Source recaps

• inbox/2026-08-07-histo-hio-architecture-review.md
• inbox/2026-08-07-histo-hio-hard-design-review.md (the consolidated pass)
• inbox/2026-08-07-histo-s3-io-evidence-review.md
• inbox/2026-08-07-histo-s3-io-migration-semantics-review.md