---
title: "hio S3-first storage redesign"
description: "Consolidated findings from four independent adversarial reviews of the histo feat/fk/s3-io branch (hio library + histo.io seam) — do not merge until the listed invariants hold."
image: "https://docs.virdx.dev/img/virdx-social-card.png"
---

> Documentation Index
> Fetch the complete documentation index at: https://docs.virdx.dev/llms.txt
> Use this file to discover all available pages before exploring further.

# hio S3-first storage redesign

Status: **unmerged, not approved**. `histo` branch `feat/fk/s3-io`
(commits `39c3c1f2..f216f8da` as of 2026-08-07) proposes a new `packages/hio`
library — a physical-space raster abstraction backed by S3/Ceph — plus a
`histo.io` seam that migrates existing readers/writers onto it. Four
independent read-only adversarial reviews were run the same day (three
sub-reviews plus one consolidated pass); this page merges their findings so a
future agent doesn't have to re-derive them from `inbox/`.

If you're picking this branch up: **read this whole page before touching the
branch**, then re-verify each item below still applies (or has been fixed)
before treating it as current.

## Why it isn't ready to merge

All four reviews independently converged on the same class of problem:
**the implementation does not yet provide the safety properties its own
README and `proposal.md` claim.**

### Transaction / concurrency defects (demonstrated, not just theorized)

- The library's default `overwrite=True` write path **mutates keys already
  referenced by published metadata** — there is no snapshot isolation or
  atomic replacement, despite the README's "commit is one small write, last"
  claim (true only for first publication under a single-writer/immutable-URI
  assumption).
- In-place overwrite remains publicly readable mid-write; a concurrent reader
  can observe a torn object (data from one writer, metadata from another).
- Existence-based resume is unsafe: it checks shape/dtype only, not a
  durably-bound geometry/layout/producer-identity fingerprint, and only holds
  under a single fenced writer (which nothing enforces). Demonstrated: a
  partial writer resumes across a **changed physical geometry**; a
  reopened writer with incompatible geometry reuses stale shards; reopening
  a writer against an all-fill shard cannot complete it.

### Spatial / calibration defects (demonstrated)

- `Image.read(region).shape` can disagree with the declared output geometry
  for a requested level (e.g. non-integer `Level.mpp(1.5)` reports a shape/MPP
  inconsistent with the returned array).
- Non-square pyramid rounding stamps the **x axis with the y axis's**
  downsample factor.
- OME-XML-only calibration is silently discarded by the TIFF adapter (only
  TIFF resolution tags are read).
- NGFF 0.5 `dimension_names` are absent from written metadata, despite the
  branch's own `references.md` documenting the requirement.

### Migration-specific regressions (`main` vs. `feat/fk/s3-io`)

- Panoptic input fingerprints are dropped.
- Gland-inference model windows are re-anchored at **storage shard
  boundaries** instead of the original coordinate frame (shard-dependent
  inference — same input, different result depending on how it's chunked).
- The `histo.io` seam drops `fill_value`.

### Evidence / benchmark validity

- The proposal's own required production benchmark had not been run before
  several implementation decisions were already made.
- Benchmarks ran against a **local loopback MinIO fixture only** — useful as
  integration/API evidence, not as evidence for real S3/Ceph latency,
  throughput, retry, or concurrency tuning.
- At least one benchmarked "patch" path does not materialize the patch at
  all (non-materializing benchmark); batch fan-out is unbounded; some
  reported numbers are cache-confounded or not traceable to a reproducible
  artifact/counter in code. Labels like "GET", "MB/s", "patches/s" in the
  writeup don't always mean what they say — trace every metric back to the
  code that counts it before trusting it.
- 35/35 hio unit tests and 12/12 seam tests pass, but the suites only cover
  happy-path stored levels/single-writer flows — they omit overwrite
  visibility, duplicate/concurrent writers, incompatible resume, all-fill
  resume, and read-shape/geometry agreement. **Green CI here is not evidence
  of the properties above.**

## What "ready" looks like (recommendation from all four reviews)

Before merge or deployment:

1. Writes use **immutable generations** + conditional (CAS-style) dataset-scope
   publication — no post-commit mutation of a key another commit's metadata
   already points at.
2. Writer fencing: only one writer can hold/complete a given generation.
3. Resume validates a complete creation/job manifest (geometry + layout +
   inputs + model/config fingerprint + code version), not shape/dtype alone.
4. Multi-array commits (e.g. panoptic semantic + instance maps) publish as
   one atomic dataset version.
5. Level resampling is truthful: `read(region).shape` matches declared
   geometry for every level policy, and each axis's rounding is independent.
6. OME-NGFF 0.5 conformance (incl. `dimension_names`) checked against the
   official validator, not just visual inspection.
7. Benchmarks re-run against real S3/Ceph (not loopback MinIO only), with
   request/byte instrumentation, randomized order, explicit cache state, and
   a reproducible artifact per reported number.

## Reference implementations to steal from rather than reinvent

Cited by the reviews as prior art hio's abstraction should either adopt or
explicitly justify diverging from: **xarray**, **SpatialData** (heterogeneous
coordinate models), **TensorStore** / **Icechunk** (proven async data plane +
immutable-snapshot transactions), and **OME-NGFF** (metadata/validator
tooling).

## Process note

Don't repeat this: three sub-agents each filed a separate narrow inbox recap
before the consolidated review landed, which briefly fragmented this finding
across four `inbox/` entries. When several subagents adversarially review the
same branch in parallel, only the parent/consolidating agent should write the
durable knowledge-base entry.

## Source recaps

- `inbox/2026-08-07-histo-hio-architecture-review.md`
- `inbox/2026-08-07-histo-hio-hard-design-review.md` (the consolidated pass)
- `inbox/2026-08-07-histo-s3-io-evidence-review.md`
- `inbox/2026-08-07-histo-s3-io-migration-semantics-review.md`

Source: https://docs.virdx.dev/knowledge/wiki/workstreams/histo/projects/hio-s3-storage-redesign/index.mdx