---
title: "Storage, caching and transfers"
image: "https://docs.virdx.dev/img/virdx-social-card.png"
---

> Documentation Index
> Fetch the complete documentation index at: https://docs.virdx.dev/llms.txt
> Use this file to discover all available pages before exploring further.

# Storage, caching and transfers

{/* Vendored from mono/packages/vxs3/STORAGE.md @ 0de611c998fa7b7d09efee323500622878df929e by scripts/sync-docs.sh. Do not edit here. */}

Start with [README.md](/vx/cluster/vxs3/) for the public interfaces and common workflows.
This reference covers storage integration, publication guarantees, local disk use
and transport behaviour. Native array policies live in [ZARR_OPTIMISATION.md](https://github.com/virdx/mono/blob/0de611c998fa7b7d09efee323500622878df929e/packages/vxs3/ZARR_OPTIMISATION.md);
the bundle wire format lives in [FORMAT.md](https://github.com/virdx/mono/blob/0de611c998fa7b7d09efee323500622878df929e/packages/vxs3/FORMAT.md).

## Storage integrations

Vxs3 has no application SDK dependency. Applications own authentication, catalog
state and path allocation; vxs3 owns formats, validation and publication order.
The protocols in [storage.py](https://github.com/virdx/mono/blob/0de611c998fa7b7d09efee323500622878df929e/packages/vxs3/src/vxs3/core/storage.py) separate these responsibilities:

| Contract | Responsibility |
| --- | --- |
| `ObjectDownloader` | Download exact relative paths with known sizes and optional hashes into a new directory |
| `ObjectReader` | Add recursive object listing for tree operations |
| `BundleReader` | Add bounded reads of one exact manifest, without requiring a listing |
| `ObjectDestination` | Allocate a fresh output root and return an upload session |
| `UploadSession` | Upload explicit file mappings and provide read-back access for verification |

Readers expose a credential-free root URI ending in `/`. Paths are relative to
that root. Adapters own credentials, retries within their transfer contract and
presigned-URL refresh. The vxdata adapter is described in the
[SDK README](/vx/cluster/vxdata-sdk/#explicit-bundles).

### Direct S3 adapters

`S3Source` lists/downloads a prefix and implements both reader contracts.
`S3Destination` allocates a fresh UUID directory beneath an output parent.
`S3ExactDestination` uses a caller-chosen output parent and checks selected outputs
for absence; the caller must coordinate exclusive writers. Neither destination
replaces published or partial outputs. Source and destination stores may use
different endpoints, buckets and credentials.

All three adapters accept relative prefixes or `s3://bucket/prefix/` URIs.
Relative paths are within `store.prefix`; full URIs must include that prefix and
match the store's bucket. Paths are literal object keys without percent-decoding.
Query strings, fragments and `s3://host/bucket/path` forms are not supported.
These adapters select directory prefixes/output parents. A trailing slash is
optional; individual-object selection belongs to other interfaces, such as
`Source(store, key, size=...)` or the analysis CLI.

### Manifest reads

`BundleReader.read_metadata(path, max_bytes=..., etag=...)` reads one exact key,
enforces its byte limit while streaming and returns bytes plus validators.
For presigned adapters, `vxs3.transport.metadata.read_presigned_metadata` consumes the
complete signed GET URL without a HEAD request. On retry with a refreshed URL,
pass the original ETag to avoid mixing versions.

`open_bundle` accepts a manifest URI or `BundleRef`; the latter supplies the
expected SHA-256 automatically. The hash covers the stored manifest bytes.
Optional `etag` conditions the GET. The returned `Bundle` retains this manifest
snapshot for later operations; opening never infers completeness from a listing.

```python
from vxs3.bundle.io import open_bundle

bundle = open_bundle(ref.manifest_uri, store=store, sha256=ref.sha256)
```

An S3 URI without `store=` uses obstore's environment configuration. Custom
adapters use `open_bundle(ref, reader=reader)`; the URI must be within the reader's
root. Passing a reader as the source with a relative `manifest=` is also supported.
Persist storage URIs and hashes, never presigned URLs. Applications must check
their own catalog readiness or outer completion record before exposing an inner
manifest. A valid manifest alone does not verify its shards.

`Bundle.list_entries(prefix="images/", offset=0, limit=100)` selects a page sorted
by logical path. The prefix is a directory inside the bundle, independent of the
S3 prefix; it is not a glob. Pagination bounds returned entries, not the parsed
manifest's memory use. `get_member(path)` looks up a regular file without shard
I/O.

## Publication and scratch

`publish_bundle` accepts an already-packed local bundle, validates it, allocates
fresh keys, uploads and ETag-verifies shards, then publishes and verifies
the manifest last. Only then does it return `BundleRef`. Keep inputs immutable
throughout the call.

`scratch_dir` is optional and defaults to a fresh directory under
`VXS3_CACHE_DIR/publish`. If neither is configured, publication raises `ValueError`.
Scratch holds read-back downloads and the manifest copy/check, separately
from the packed input and persistent member cache. Temporary files are removed
on exit; an explicitly provided scratch parent may remain empty.

Publication uploads up to four shards at a time, each with its share of the
`concurrency` streams (optional, default `8`). A shard whose ETag matches the
local digest needs no scratch; one the store cannot vouch for (or every shard
with `read_back=True`) is downloaded into scratch and removed as soon as it is
checked, so allow up to four shards plus the manifest in addition to the
existing local bundle. `concurrency=1` limits this scratch to one shard (up to
4 GiB for bundles produced by `pack`). Native Zarr scratch requirements differ;
see [scratch and replacement](https://github.com/virdx/mono/blob/0de611c998fa7b7d09efee323500622878df929e/packages/vxs3/ZARR_OPTIMISATION.md#scratch-and-replacement).

Failed attempts can leave unreferenced shards; failure after the manifest PUT
can leave a complete bundle. Callers own cleanup/reconciliation and must not
overwrite a published allocation. Manifest-last publication is a completion
convention, not a multi-object S3 transaction.

## Composition

A batch whose items each publish a bundle beneath `items/<id>/` of one prefix
completes as a single bundle by publishing a merged manifest at that prefix:

```python
from vxs3.bundle.io import open_bundle, publish_manifest
from vxs3.bundle.manifest import merge_manifests

parts = {f"items/{item}": open_bundle(ref, store=store).inventory for item, ref in refs.items()}
root = publish_manifest(merge_manifests(parts), S3ExactDestination(store, "batches/run-1"))
```

`merge_manifests` is pure: it prefixes paths and shard keys and sums provenance
(see [composed bundles](https://github.com/virdx/mono/blob/0de611c998fa7b7d09efee323500622878df929e/packages/vxs3/FORMAT.md#composed-bundles)). `publish_manifest` allocates
only `manifest.parquet`, so the destination must tolerate the parts already
stored under other entries; `S3ExactDestination` checks only the selected
top-level entries. Before the upload it lists the destination once, recursively,
and requires every referenced shard at its key with its exact size; shard bytes
are not read again, since each part verified them when it was published. The
listing is paged by the store, so its cost grows with the bundle's object count,
not with the number of parts. The manifest is then
uploaded and read-back verified like any other publication. A missing or
differently sized shard fails before anything is written. Passing a local
manifest path publishes those exact bytes; a `Manifest` is serialised first.
Scratch holds only the manifest and follows the same `scratch_dir` /
`VXS3_CACHE_DIR` rules as `publish_bundle`.

Publishing the merged manifest last is the batch's completion convention, as
each part's manifest is its own. Readers of the merged bundle never consult the
parts' manifests; readers of a part never see the batch. Applications record
which parts a batch expected (an outer index or catalog row) so a merged
manifest written with some parts absent is a deliberate partial result, not an
accident.

## Streaming member and Zarr reads

See [D3N caching and range reads](https://github.com/virdx/mono/blob/0de611c998fa7b7d09efee323500622878df929e/packages/vxs3/D3N.md) for the server's overlap/length
rules and the physical request policy used below.

`Bundle.read_range(path, start=0, stop=None, max_bytes=64 * 2**20)` returns
member bytes in memory. Offsets are relative to that member, stop is exclusive,
and omitting stop selects the member's end. Bounds must lie within the member;
oversized requests fail before any transfer. Empty ranges need no GET. The
manifest's payload offset translates the interval into its containing tar,
including tar shards in composed bundles. No tar scan or local file is needed.

Streaming adapters implement the optional `core.storage.RangeReader` protocol.
Direct S3 and vxdata's presigned adapter support it. They check the exact
response range, total object size, body size and a strong ETag. The first
successful range pins each shard's ETag for that Bundle handle; concurrent
subsequent ranges, complete-range retries and URL renewal retain it. Changed
objects fail. Streaming honors the SDK's `download_routing`: `gateway` keeps
the original URL, `d3n` requires deployment inventory, and `auto` uses D3N when
inventory exists. Routed reads split at 256 MiB routing-cell boundaries and
use the bulk downloader's Ketama contract without staging a whole cell.
D3N mode expands reads to complete 4 MiB Ceph stripes, clamped at object EOF,
then returns only the requested bytes. It processes one stripe at a time and
pins the ETag across every stripe. Stripe requests never cross a routing cell.
Gateway mode retains exact ranges. For a caller-configured Ceph endpoint,
`S3Source(store, prefix, d3n=True)` enables the same alignment without selecting
endpoints. The object layout must use aligned 4 MiB stripes; both current
vxs3 and SDK upload part sizes are multiples of that size.

The streaming session owns its own circuit breakers and bounded retries;
failover retains ETags and never falls back to the gateway. Signed Host/path/
query bytes are preserved and TLS verifies the selected backend endpoint.

The SDK shares a `transport.streaming.PresignedRangeReader` across bundle
handles with matching transport settings. Its bounded cache of HTTP clients
reuses connections for the same signed URL. URL renewal creates a new client;
client shutdown waits for active reads and drops cached clients. Standalone
presigned readers can use this class as a context manager as well.

**Integrity:** a request covering a complete member verifies its SHA-256 from
the pinned manifest. A partial request cannot verify that whole-file hash.
Its ETag provides version consistency, not cryptographic verification against
the published content. The publisher's verification and immutable storage are
assumptions for partial reads; stronger partial-content verification would
require new block hashes or authenticated range metadata. Existing local
materialisation continues to verify every complete member.

`vxs3.zarr.store.BundleStore(bundle, prefix="images/example.zarr")` adapts this
interface to Zarr's read-only Store API. Whole values, explicit intervals,
offsets and suffix requests are translated within the member. Manifest-only
listing/existence/size queries need no shard reads. Missing manifest keys return
Zarr's missing-value result; failures reading a declared member propagate.
Zarr itself handles codecs, native shard indexes, groups and fill values.
Writes, deletes, append/resize and attribute updates are unsupported.

`max_read_bytes` defaults to 64 MiB per encoded read and may be raised explicitly;
requesting a full native shard may require a larger budget than a crop.
This bounds logical bytes; an active D3N read also holds a physical response
of at most 4 MiB plus transport/copy buffers. Expanded bytes are discarded as
each stripe completes. Before retries, overfetch per logical request is less than 8 MiB.
`concurrency=8` bounds active reads per store across event loops. Transport work
runs off the event loop, and cancelled reads wait for their worker to finish.
`cache_bytes=16 * 2**20` bounds cached payload bytes; the LRU also caps entries
at 4096 and caches only nonempty reads up to 256 KiB. Set it to zero to disable
caching. `close()` clears this memory cache. Decoded arrays, returned buffers
and the eagerly loaded manifest are separate memory costs. Neither streaming
interface requires `VXS3_CACHE_DIR`. Keep SDK clients open until reads finish.

Bundling stores preserves encoded bytes, metadata and chunking, including v2.
It performs no Zarr validation or automatic optimisation; compatibility is
determined by the consumer's installed Zarr and codecs. Native optimisation is
still available as an explicit workflow. Manifest entry/byte limits and whole
file packing limits remain in force. Local HTTP/array tests establish semantics,
not live Ceph throughput or compatibility with every external viewer.

## Cached member access

```python
paths = bundle.materialize(["images/a.bin", "images/b.bin"], cache_dir="./cache")
with bundle.open("labels/example.json", "r", cache_dir="./cache") as stream:
text = stream.read()
```

The optional `cache_dir` overrides `VXS3_CACHE_DIR`; omitting both raises
`ValueError`. `concurrency` is optional and defaults to `8`.

A miss reads only the missing files' byte ranges: batches of at most 64 MiB
per shard in shard order, with nearby ranges merged by the store, so files
packed side by side cost one read. Each file is checked against its SHA-256,
then atomically published as a cache entry; later requests need no GET. Keys
include the manifest URI/hash and file path/hash, never a presigned URL. POSIX
file locks coordinate threads/processes. Failed attempts publish no cache entry
and can be retried. At most one batch per shard is held in memory.

Cache roots belong to one trusted caller; do not share them across untrusted
users. Returned files have write bits removed and must remain immutable: copy
them before editing. Cache hits trust the completed entry, checking path
containment and member size without rehashing all contents. There is no automatic
eviction; remove entries only when no readers use their paths.

## Extract into a directory

```python
files = bundle.extract("./out", ["images/a.bin"])  # {path: ./out/images/a.bin}
bundle.extract("./out")  # every file and directory
```

`extract` writes files under their bundle paths into a directory that may
already exist. It reads only their byte ranges, in batches of at most 64 MiB,
and checks each file's SHA-256 before an atomic rename. Existing files with the
same size and SHA-256 are kept without any read; a different existing file
raises `FileExistsError`. It does not use the member cache.

`Bundle.download(destination)` restores the whole bundle separately into a new
directory. It downloads all shards into temporary local staging before unpacking;
allow space for both the downloaded bundle and restored output. It does not
populate or reuse the member cache. Local callers can use
`verify_bundle(manifest)` to validate all local shards without extraction.

## Default local storage

Explicit destination, scratch and cache arguments take precedence over
`VXS3_CACHE_DIR`. The environment is read at call time; relative cache roots
resolve against the working directory and `~` is expanded. A missing/blank root
raises `ValueError` when a default location is needed. An unusable root raises a
filesystem error without falling back to another disk.

With `VXS3_CACHE_DIR=/cache/.vxs3`, omitted output destinations use:

| Operation | Default destination |
| --- | --- |
| `download()`, `download_async()`, `download_zarr()` | `/cache/.vxs3/downloads/<unique-id>` |
| `unpack()`, `Bundle.download()` | `/cache/.vxs3/unpacked/<unique-id>` |
| `Bundle.materialize()`, `Bundle.open()` | Reusable entries beneath `/cache/.vxs3/members-v1` |

Fresh-output operations return the new absolute path and preserve other root
contents. Staging stays beside the destination on the selected volume. Existing
destinations are rejected. Outputs persist after the call; callers own cleanup.
`pack()` requires an explicit destination. These output locations provide no
automatic reuse; only member access uses the shared verified cache.

## Direct download request policy

`download()` and `download_async()` use a fixed 256 MiB range grid with no
per-call grid-size override. `vxs3.transport._ranges.download_ranges(size)` exposes the
planner without performing I/O. Python offsets are half-open; `range_header`
uses inclusive HTTP endpoints. Every nonempty object uses explicit ranges; the
last range ends at the exact object size. Empty objects need no data GET and rely
on trusted size/existence metadata from the caller.

Mapping keys are local relative filenames. Sources supply exact sizes and,
when available, SHA-256 hashes. One worker budget covers every object/range in
the call; `concurrency` is optional and defaults to `8`. A large object can occupy
all workers. Each response streams into its offset in one staged file; the
planner does not create a task or temporary file per range.

The first data response pins a strong ETag. Remaining requests and retries use
`If-Match`; callers may supply a known ETag for the first request too. Weak/missing
ETags are rejected. Sizes, ranges, byte counts and supplied SHA-256 digests are
checked before publishing the destination. ETags guard versions, not content
integrity. A version may also be pinned in a signed URL.

`attempts=3` bounds complete-range attempts with backoff. Disable obstore retries
with `retry_config={"max_retries": 0}` so interrupted streams cannot resume at a
different offset. Retries overwrite the entire original range. Expired/denied
URLs raise `AuthorizationError` (also `DownloadError` for downloads), carrying
an `etags` mapping of learned validators keyed by output name. The caller may
supply fresh URLs and retry the batch, preserving every reported ETag. Partial downloads
are discarded; persistent resume and automatic URL refresh are not implemented
in this downloader. All workers finish/cancel before staging cleanup.

Presigned transport defaults are centralized in `vxs3.transport._options.client_options`:
5-second connect timeout, 300-second read-stall timeout and 300-second total
request timeout including the response body. Explicit options override defaults.
`SSL_CERT_FILE` and hashed `SSL_CERT_DIR` certificates supplement backend trust
roots unless an explicit `root_certificate` PEM value is supplied. Proxy and
other HTTP settings are also passed as obstore client options. A Python HTTP
client or SSLContext cannot configure this transport. Caller-created S3 stores
continue to use their own configuration.

Metadata reads retry transient transport failures up to three times while
retaining the first response's strong ETag. Authorization failures expose the
same typed signal, with the metadata key (empty for a presigned URL) in `etags`.

Presigned GET URLs are passed intact; do not rebuild them or probe them with
HEAD. `Source.presigned` accepts obstore client options for custom CA/timeouts.
For credentials/STS, use a configured `S3Store`; credential refresh belongs to
its provider. Local writes and final checksum reads are synchronous. See the
[transport contract](https://github.com/virdx/mono/blob/0de611c998fa7b7d09efee323500622878df929e/packages/vxs3/FORMAT.md#transport-choice-obstore), the
[presigned download snippet](/vx/cluster/vxs3/#5-download-individual-objects) and the
[direct file download example](https://github.com/virdx/mono/blob/0de611c998fa7b7d09efee323500622878df929e/packages/vxs3/examples/download_file.py).

The range grid is independent of the bundle shard-size policy and can transfer
native objects above 4 GiB. Range distribution across RGW daemons depends on the
routing layer; it does not change D3N cache-file size. Live RGW throughput and
credential refresh require separate validation. The vxdata SDK uses this same downloader; it supplies sizes and handles URL renewal.

## D3N object worker pool

Install `vxs3[workers]` to use `DownloadPool`, including `pybreaker`
(conda/Pixi users add `pybreaker>=1.4,<2` from conda-forge alongside `vxs3`).
The conda package constrains this optional dependency but does not install it
automatically; importing `vxs3.transport.workers` without it reports how to install it.
The development environment includes it; Ketama routing needs no extra dependency.

```python
from vxs3.transport.instances import load_instances
from vxs3.transport.workers import DownloadObject, DownloadPool

def main():
objects = {
    "first.bin": DownloadObject("images", "first.bin", size=1048576),
    "second.bin": DownloadObject("images", "second.bin", size=2097152),
}
with DownloadPool(
    workers=4,
    instances=load_instances("/etc/virdx/d3n/d3n_instances.json"),
    s3_config={"region": "us-east-1"},
) as pool:
    local = pool.download(objects, "./downloaded")
    print(local, pool.circuit_states)

if __name__ == "__main__":
main()
```

Use an importable Python script with the main guard. The pool starts up to
`workers` processes using multiprocessing's `spawn` method. Each child creates
and reuses its own obstore clients by endpoint and bucket. Object bytes stream
directly to their offsets in preallocated staged files and never cross IPC.

Nonempty objects use `download_ranges(size)`: the shared 256 MiB grid, including
an explicit range for small objects and a final short cell. Each cell is routed
with its own start offset, object key and version. The first cell completes
before the rest are dispatched, pinning one strong ETag across every node and
retry. A single large object can then use the whole process budget. Plans are
lazy; the scheduler does not allocate one future or file per cell. Empty objects
use one whole-object GET to verify existence, size and ETag.

Supply exact sizes from trusted metadata, with `etag`, `version_id` and `sha256`
when available. Empty/dot key segments are rejected to preserve wire identity.
Every response validates the full object size, returned range, byte count and
strong ETag. Retries overwrite the complete original cell using `If-Match`.
SHA-256 is verified after all cells complete. The batch is published as a new
directory only after every object succeeds. Failed batches discard staging
after outstanding workers finish. Omit the destination to use
`VXS3_CACHE_DIR/downloads/<uuid>`.

One parent-owned breaker covers each **manifest instance name**, shared across
workers and successive batches. Each name maps to one unique HTTPS origin in
the pool's fixed inventory. Defaults:

- `fail_max=3`: three consecutive transient failures remove the instance from
  the active ring. Retries route the same cell through the remaining instances.
- `reset_timeout=30`: after 30 seconds, a request for the original owner can
  probe it. One probe runs per instance; success restores membership and an
  unsuccessful in-flight probe starts another cooldown. A probe cancelled before
  execution remains immediately eligible. If all circuits are open, any due
  instance can probe.
- `attempts=4`: at most four attempts per cell, including probes. Backoff is
  0.25, 0.5, 1, 2, then at most 4 seconds, shared with the direct downloader.
  Waiting retries occupy no process and do not block other ready cells.
  An attempt limit below the failure threshold can stop before eviction.
  If no instance or probe is available, the batch fails promptly; retry later.

Connection/stream failures, timeouts, HTTP 408/429 and server errors count toward
eviction. Authentication, missing objects, other request errors, local disk
errors and integrity failures fail the batch without evicting a node. Obstore's
typed exceptions take precedence; its unmapped HTTP status text is covered by
real-server compatibility tests. Unrecognized native errors fail without eviction
or exposing native messages. In-flight results from before an eviction cannot
undo a later trip or probe. `active_instances` and `circuit_states` expose
snapshots. Separate pools can have different health views and temporarily route
the same cell differently. Once D3N routing is selected, exhausted circuits do
not switch the batch to the gateway.

401/403 raise `DownloadAuthorizationError` with an `etags` map keyed by local
output name, including validators learned while draining outstanding work.
Callers can refresh credentials and retry with those ETags. A 412, changed or
unusable ETag, invalid size/range, or SHA-256 mismatch raises
`DownloadIntegrityError`. Other transfer failures raise `DownloadError`.
A crashed child fails the current batch and discards the broken executor;
the next `download()` on the same pool starts fresh processes.

For objects without `presigned_url`, credentials come from obstore's environment
lookup or `s3_config`, including session tokens. Child-created S3 stores override
configured/environment endpoints and buckets with the selected manifest instance
and use path-style addressing.
Caller-created stores and credential callbacks are not accepted. The parent
resolves `transport.client_options()` once and passes the resulting options and
CA bytes to every child: `SSL_CERT_FILE` and hashed `SSL_CERT_DIR` certificates
supplement backend roots unless `root_certificate` is explicit. Defaults are a
5-second connect timeout, 300-second read-stall timeout and 300-second total
per-range timeout. Explicit options override these limits. Slow connections can
need a larger per-range budget; total object size does not enlarge individual
requests. Obstore retries are disabled so the parent controls every attempt.
Close the pool or use its context manager to join children. Concurrent calls on
one pool are rejected.

### Presigned requests and discovery

Supply a presigned GET as a credential on the object:

```python
objects = {
"shard.tar": DownloadObject(
    "example",
    "run/shard.tar",
    size_bytes,
    presigned_url=signed_url,
),
}
with DownloadPool(workers=4, routing="auto") as pool:
local = pool.download(objects, "./downloaded")
```

Run this inside the guarded `main()` from the example above. The caller obtains
the URL and exact size; vxs3 owns inventory discovery, cell mapping and failover.
One URL per object suffices across ranges and nodes. The original signed Host
(including its original port), encoded path and query are preserved. Only the
connection origin changes to the selected manifest endpoint. The child uses
obstore `HTTPStore` with HTTP/1, verifies TLS against the selected endpoint's
hostname using the shared CA roots, and does not add AWS Authorization headers.
Caller-supplied Host/Authorization defaults are replaced/removed for these routed
requests. No S3 credentials are required in the parent or child.

Routed URLs must be HTTPS and path-style, matching the supplied logical bucket
and key. If `X-Amz-SignedHeaders` is present it must be `host`: range and ETag
conditions vary between requests. `versionId` is decoded for routing and checked
against an explicit `version_id`, while its original encoding remains in the
query. Native exceptions and object representations never expose the URL.
401/403 still return the learned ETags; the caller can renew URLs and retry on
the same pool without losing its inventory or breaker state.

An explicit `instances` sequence is authoritative. Otherwise the pool calls
`vxs3.transport.instances.discover_instances()`: first `VXS3_D3N_INSTANCES`, then
`/etc/virdx/d3n/d3n_instances.json`. A missing environment-selected file, empty
environment value or malformed inventory raises an error. Discovery returns
`None` only when neither an environment path nor the mounted file is present.
Each pool loads a fixed snapshot once, including across URL renewal.

- `routing="d3n"` (the default): require discovered or explicit instances. Missing
  deployment inventory raises `DownloadError` before any transfer.
- `routing="auto"`: use discovered instances; without an inventory, use the
  original presigned URLs through the existing concurrent range downloader.
  This gateway path requires presigned sources, starts no child processes and
  has no per-D3N breakers (`active_instances == ()`, `circuit_states == {}`).
- `routing="gateway"`: use the original URLs and skip discovery. Cannot be
  combined with an explicit D3N instance list.

The gateway path uses an async bridge so the blocking pool method works
inside an active notebook event loop. Interruption cancels the batch and waits
for its tasks to finish cleanup before returning. Outside a notebook, it uses
the calling thread's event loop runner. `allow_http=True` remains required for
local HTTP gateway URLs;
D3N endpoints always require HTTPS. Never use the bundled snapshot as an
implicit production membership source; mount or explicitly configure inventory.

Local TLS tests verify SigV4, range placement, failover and renewal invariants.
Live D3N throughput and API-issued credential renewal have not been tested.
The vxdata SDK and Zarr integrations remain separate adapters using the same
range grid; SDK adoption of this pool is a separate change.

## Errors, progress and async use

Bundle operations expose `ManifestError`, `IntegrityError` and `StorageError`
under `Vxs3Error`. Invalid arguments can raise `ValueError`; local filesystem
errors can propagate. Zarr transfers expose `ZarrTransferError`.

Operations accepting `progress=callback` emit `Progress` records with phase,
completed/total counts and an optional relative path. `extract` counts files
per completed batch and `materialize` per resolved file, each starting at zero
of the total requested. Callbacks run synchronously and can interrupt work by
raising. The last bundle publication
event is emitted before the manifest PUT; no callback runs after successful
publication. Use the returned reference as the completion signal.

High-level bundle and Zarr operations are blocking; async applications can run
them in a worker thread. For direct downloads in Jupyter or an async application,
use top-level `await download_async(...)`. It has the same destination,
concurrency, retry and integrity behaviour as `download()`. Do not use
`asyncio.run()` or the blocking wrapper inside an existing event loop.

Source: https://docs.virdx.dev/vx/cluster/vxs3-storage/index.mdx
