Skip to content
VIRDX

Storage, caching and transfers

Start with README.md for the public interfaces and common workflows. This reference covers storage integration, publication guarantees, local disk use and transport behaviour. Native array policies live in ZARR_OPTIMISATION.md; the bundle wire format lives in FORMAT.md.

Storage integrations

Vxs3 has no application SDK dependency. Applications own authentication, catalog state and path allocation; vxs3 owns formats, validation and publication order. The protocols in storage.py separate these responsibilities:

Contract Responsibility
ObjectDownloader Download exact relative paths with known sizes and optional hashes into a new directory
ObjectReader Add recursive object listing for tree operations
BundleReader Add bounded reads of one exact manifest, without requiring a listing
ObjectDestination Allocate a fresh output root and return an upload session
UploadSession Upload explicit file mappings and provide read-back access for verification

Readers expose a credential-free root URI ending in /. Paths are relative to that root. Adapters own credentials, retries within their transfer contract and presigned-URL refresh. The vxdata adapter is described in the SDK README.

Direct S3 adapters

S3Source lists/downloads a prefix and implements both reader contracts. S3Destination allocates a fresh UUID directory beneath an output parent. S3ExactDestination uses a caller-chosen output parent and checks selected outputs for absence; the caller must coordinate exclusive writers. Neither destination replaces published or partial outputs. Source and destination stores may use different endpoints, buckets and credentials.

All three adapters accept relative prefixes or s3://bucket/prefix/ URIs. Relative paths are within store.prefix; full URIs must include that prefix and match the store’s bucket. Paths are literal object keys without percent-decoding. Query strings, fragments and s3://host/bucket/path forms are not supported. These adapters select directory prefixes/output parents. A trailing slash is optional; individual-object selection belongs to other interfaces, such as Source(store, key, size=...) or the analysis CLI.

Manifest reads

BundleReader.read_metadata(path, max_bytes=..., etag=...) reads one exact key, enforces its byte limit while streaming and returns bytes plus validators. For presigned adapters, vxs3.transport.metadata.read_presigned_metadata consumes the complete signed GET URL without a HEAD request. On retry with a refreshed URL, pass the original ETag to avoid mixing versions.

open_bundle accepts a manifest URI or BundleRef; the latter supplies the expected SHA-256 automatically. The hash covers the stored manifest bytes. Optional etag conditions the GET. The returned Bundle retains this manifest snapshot for later operations; opening never infers completeness from a listing.

from vxs3.bundle.io import open_bundle

bundle = open_bundle(ref.manifest_uri, store=store, sha256=ref.sha256)

An S3 URI without store= uses obstore’s environment configuration. Custom adapters use open_bundle(ref, reader=reader); the URI must be within the reader’s root. Passing a reader as the source with a relative manifest= is also supported. Persist storage URIs and hashes, never presigned URLs. Applications must check their own catalog readiness or outer completion record before exposing an inner manifest. A valid manifest alone does not verify its shards.

Bundle.list_entries(prefix="images/", offset=0, limit=100) selects a page sorted by logical path. The prefix is a directory inside the bundle, independent of the S3 prefix; it is not a glob. Pagination bounds returned entries, not the parsed manifest’s memory use. get_member(path) looks up a regular file without shard I/O.

Publication and scratch

publish_bundle accepts an already-packed local bundle, validates it, allocates fresh keys, uploads and ETag-verifies shards, then publishes and verifies the manifest last. Only then does it return BundleRef. Keep inputs immutable throughout the call.

scratch_dir is optional and defaults to a fresh directory under VXS3_CACHE_DIR/publish. If neither is configured, publication raises ValueError. Scratch holds read-back downloads and the manifest copy/check, separately from the packed input and persistent member cache. Temporary files are removed on exit; an explicitly provided scratch parent may remain empty.

Publication uploads up to four shards at a time, each with its share of the concurrency streams (optional, default 8). A shard whose ETag matches the local digest needs no scratch; one the store cannot vouch for (or every shard with read_back=True) is downloaded into scratch and removed as soon as it is checked, so allow up to four shards plus the manifest in addition to the existing local bundle. concurrency=1 limits this scratch to one shard (up to 4 GiB for bundles produced by pack). Native Zarr scratch requirements differ; see scratch and replacement.

Failed attempts can leave unreferenced shards; failure after the manifest PUT can leave a complete bundle. Callers own cleanup/reconciliation and must not overwrite a published allocation. Manifest-last publication is a completion convention, not a multi-object S3 transaction.

Composition

A batch whose items each publish a bundle beneath items/<id>/ of one prefix completes as a single bundle by publishing a merged manifest at that prefix:

from vxs3.bundle.io import open_bundle, publish_manifest
from vxs3.bundle.manifest import merge_manifests

parts = {f"items/{item}": open_bundle(ref, store=store).inventory for item, ref in refs.items()}
root = publish_manifest(merge_manifests(parts), S3ExactDestination(store, "batches/run-1"))

merge_manifests is pure: it prefixes paths and shard keys and sums provenance (see composed bundles). publish_manifest allocates only manifest.parquet, so the destination must tolerate the parts already stored under other entries; S3ExactDestination checks only the selected top-level entries. Before the upload it lists the destination once, recursively, and requires every referenced shard at its key with its exact size; shard bytes are not read again, since each part verified them when it was published. The listing is paged by the store, so its cost grows with the bundle’s object count, not with the number of parts. The manifest is then uploaded and read-back verified like any other publication. A missing or differently sized shard fails before anything is written. Passing a local manifest path publishes those exact bytes; a Manifest is serialised first. Scratch holds only the manifest and follows the same scratch_dir / VXS3_CACHE_DIR rules as publish_bundle.

Publishing the merged manifest last is the batch’s completion convention, as each part’s manifest is its own. Readers of the merged bundle never consult the parts’ manifests; readers of a part never see the batch. Applications record which parts a batch expected (an outer index or catalog row) so a merged manifest written with some parts absent is a deliberate partial result, not an accident.

Streaming member and Zarr reads

See D3N caching and range reads for the server’s overlap/length rules and the physical request policy used below.

Bundle.read_range(path, start=0, stop=None, max_bytes=64 * 2**20) returns member bytes in memory. Offsets are relative to that member, stop is exclusive, and omitting stop selects the member’s end. Bounds must lie within the member; oversized requests fail before any transfer. Empty ranges need no GET. The manifest’s payload offset translates the interval into its containing tar, including tar shards in composed bundles. No tar scan or local file is needed.

Streaming adapters implement the optional core.storage.RangeReader protocol. Direct S3 and vxdata’s presigned adapter support it. They check the exact response range, total object size, body size and a strong ETag. The first successful range pins each shard’s ETag for that Bundle handle; concurrent subsequent ranges, complete-range retries and URL renewal retain it. Changed objects fail. Streaming honors the SDK’s download_routing: gateway keeps the original URL, d3n requires deployment inventory, and auto uses D3N when inventory exists. Routed reads split at 256 MiB routing-cell boundaries and use the bulk downloader’s Ketama contract without staging a whole cell. D3N mode expands reads to complete 4 MiB Ceph stripes, clamped at object EOF, then returns only the requested bytes. It processes one stripe at a time and pins the ETag across every stripe. Stripe requests never cross a routing cell. Gateway mode retains exact ranges. For a caller-configured Ceph endpoint, S3Source(store, prefix, d3n=True) enables the same alignment without selecting endpoints. The object layout must use aligned 4 MiB stripes; both current vxs3 and SDK upload part sizes are multiples of that size.

The streaming session owns its own circuit breakers and bounded retries; failover retains ETags and never falls back to the gateway. Signed Host/path/ query bytes are preserved and TLS verifies the selected backend endpoint.

The SDK shares a transport.streaming.PresignedRangeReader across bundle handles with matching transport settings. Its bounded cache of HTTP clients reuses connections for the same signed URL. URL renewal creates a new client; client shutdown waits for active reads and drops cached clients. Standalone presigned readers can use this class as a context manager as well.

Integrity: a request covering a complete member verifies its SHA-256 from the pinned manifest. A partial request cannot verify that whole-file hash. Its ETag provides version consistency, not cryptographic verification against the published content. The publisher’s verification and immutable storage are assumptions for partial reads; stronger partial-content verification would require new block hashes or authenticated range metadata. Existing local materialisation continues to verify every complete member.

vxs3.zarr.store.BundleStore(bundle, prefix="images/example.zarr") adapts this interface to Zarr’s read-only Store API. Whole values, explicit intervals, offsets and suffix requests are translated within the member. Manifest-only listing/existence/size queries need no shard reads. Missing manifest keys return Zarr’s missing-value result; failures reading a declared member propagate. Zarr itself handles codecs, native shard indexes, groups and fill values. Writes, deletes, append/resize and attribute updates are unsupported.

max_read_bytes defaults to 64 MiB per encoded read and may be raised explicitly; requesting a full native shard may require a larger budget than a crop. This bounds logical bytes; an active D3N read also holds a physical response of at most 4 MiB plus transport/copy buffers. Expanded bytes are discarded as each stripe completes. Before retries, overfetch per logical request is less than 8 MiB. concurrency=8 bounds active reads per store across event loops. Transport work runs off the event loop, and cancelled reads wait for their worker to finish. cache_bytes=16 * 2**20 bounds cached payload bytes; the LRU also caps entries at 4096 and caches only nonempty reads up to 256 KiB. Set it to zero to disable caching. close() clears this memory cache. Decoded arrays, returned buffers and the eagerly loaded manifest are separate memory costs. Neither streaming interface requires VXS3_CACHE_DIR. Keep SDK clients open until reads finish.

Bundling stores preserves encoded bytes, metadata and chunking, including v2. It performs no Zarr validation or automatic optimisation; compatibility is determined by the consumer’s installed Zarr and codecs. Native optimisation is still available as an explicit workflow. Manifest entry/byte limits and whole file packing limits remain in force. Local HTTP/array tests establish semantics, not live Ceph throughput or compatibility with every external viewer.

Cached member access

paths = bundle.materialize(["images/a.bin", "images/b.bin"], cache_dir="./cache")
with bundle.open("labels/example.json", "r", cache_dir="./cache") as stream:
    text = stream.read()

The optional cache_dir overrides VXS3_CACHE_DIR; omitting both raises ValueError. concurrency is optional and defaults to 8.

A miss reads only the missing files’ byte ranges: batches of at most 64 MiB per shard in shard order, with nearby ranges merged by the store, so files packed side by side cost one read. Each file is checked against its SHA-256, then atomically published as a cache entry; later requests need no GET. Keys include the manifest URI/hash and file path/hash, never a presigned URL. POSIX file locks coordinate threads/processes. Failed attempts publish no cache entry and can be retried. At most one batch per shard is held in memory.

Cache roots belong to one trusted caller; do not share them across untrusted users. Returned files have write bits removed and must remain immutable: copy them before editing. Cache hits trust the completed entry, checking path containment and member size without rehashing all contents. There is no automatic eviction; remove entries only when no readers use their paths.

Extract into a directory

files = bundle.extract("./out", ["images/a.bin"])  # {path: ./out/images/a.bin}
bundle.extract("./out")  # every file and directory

extract writes files under their bundle paths into a directory that may already exist. It reads only their byte ranges, in batches of at most 64 MiB, and checks each file’s SHA-256 before an atomic rename. Existing files with the same size and SHA-256 are kept without any read; a different existing file raises FileExistsError. It does not use the member cache.

Bundle.download(destination) restores the whole bundle separately into a new directory. It downloads all shards into temporary local staging before unpacking; allow space for both the downloaded bundle and restored output. It does not populate or reuse the member cache. Local callers can use verify_bundle(manifest) to validate all local shards without extraction.

Default local storage

Explicit destination, scratch and cache arguments take precedence over VXS3_CACHE_DIR. The environment is read at call time; relative cache roots resolve against the working directory and ~ is expanded. A missing/blank root raises ValueError when a default location is needed. An unusable root raises a filesystem error without falling back to another disk.

With VXS3_CACHE_DIR=/cache/.vxs3, omitted output destinations use:

Operation Default destination
download(), download_async(), download_zarr() /cache/.vxs3/downloads/<unique-id>
unpack(), Bundle.download() /cache/.vxs3/unpacked/<unique-id>
Bundle.materialize(), Bundle.open() Reusable entries beneath /cache/.vxs3/members-v1

Fresh-output operations return the new absolute path and preserve other root contents. Staging stays beside the destination on the selected volume. Existing destinations are rejected. Outputs persist after the call; callers own cleanup. pack() requires an explicit destination. These output locations provide no automatic reuse; only member access uses the shared verified cache.

Direct download request policy

download() and download_async() use a fixed 256 MiB range grid with no per-call grid-size override. vxs3.transport._ranges.download_ranges(size) exposes the planner without performing I/O. Python offsets are half-open; range_header uses inclusive HTTP endpoints. Every nonempty object uses explicit ranges; the last range ends at the exact object size. Empty objects need no data GET and rely on trusted size/existence metadata from the caller.

Mapping keys are local relative filenames. Sources supply exact sizes and, when available, SHA-256 hashes. One worker budget covers every object/range in the call; concurrency is optional and defaults to 8. A large object can occupy all workers. Each response streams into its offset in one staged file; the planner does not create a task or temporary file per range.

The first data response pins a strong ETag. Remaining requests and retries use If-Match; callers may supply a known ETag for the first request too. Weak/missing ETags are rejected. Sizes, ranges, byte counts and supplied SHA-256 digests are checked before publishing the destination. ETags guard versions, not content integrity. A version may also be pinned in a signed URL.

attempts=3 bounds complete-range attempts with backoff. Disable obstore retries with retry_config={"max_retries": 0} so interrupted streams cannot resume at a different offset. Retries overwrite the entire original range. Expired/denied URLs raise AuthorizationError (also DownloadError for downloads), carrying an etags mapping of learned validators keyed by output name. The caller may supply fresh URLs and retry the batch, preserving every reported ETag. Partial downloads are discarded; persistent resume and automatic URL refresh are not implemented in this downloader. All workers finish/cancel before staging cleanup.

Presigned transport defaults are centralized in vxs3.transport._options.client_options: 5-second connect timeout, 300-second read-stall timeout and 300-second total request timeout including the response body. Explicit options override defaults. SSL_CERT_FILE and hashed SSL_CERT_DIR certificates supplement backend trust roots unless an explicit root_certificate PEM value is supplied. Proxy and other HTTP settings are also passed as obstore client options. A Python HTTP client or SSLContext cannot configure this transport. Caller-created S3 stores continue to use their own configuration.

Metadata reads retry transient transport failures up to three times while retaining the first response’s strong ETag. Authorization failures expose the same typed signal, with the metadata key (empty for a presigned URL) in etags.

Presigned GET URLs are passed intact; do not rebuild them or probe them with HEAD. Source.presigned accepts obstore client options for custom CA/timeouts. For credentials/STS, use a configured S3Store; credential refresh belongs to its provider. Local writes and final checksum reads are synchronous. See the transport contract, the presigned download snippet and the direct file download example.

The range grid is independent of the bundle shard-size policy and can transfer native objects above 4 GiB. Range distribution across RGW daemons depends on the routing layer; it does not change D3N cache-file size. Live RGW throughput and credential refresh require separate validation. The vxdata SDK uses this same downloader; it supplies sizes and handles URL renewal.

D3N object worker pool

Install vxs3[workers] to use DownloadPool, including pybreaker (conda/Pixi users add pybreaker>=1.4,<2 from conda-forge alongside vxs3). The conda package constrains this optional dependency but does not install it automatically; importing vxs3.transport.workers without it reports how to install it. The development environment includes it; Ketama routing needs no extra dependency.

from vxs3.transport.instances import load_instances
from vxs3.transport.workers import DownloadObject, DownloadPool


def main():
    objects = {
        "first.bin": DownloadObject("images", "first.bin", size=1048576),
        "second.bin": DownloadObject("images", "second.bin", size=2097152),
    }
    with DownloadPool(
        workers=4,
        instances=load_instances("/etc/virdx/d3n/d3n_instances.json"),
        s3_config={"region": "us-east-1"},
    ) as pool:
        local = pool.download(objects, "./downloaded")
        print(local, pool.circuit_states)


if __name__ == "__main__":
    main()

Use an importable Python script with the main guard. The pool starts up to workers processes using multiprocessing’s spawn method. Each child creates and reuses its own obstore clients by endpoint and bucket. Object bytes stream directly to their offsets in preallocated staged files and never cross IPC.

Nonempty objects use download_ranges(size): the shared 256 MiB grid, including an explicit range for small objects and a final short cell. Each cell is routed with its own start offset, object key and version. The first cell completes before the rest are dispatched, pinning one strong ETag across every node and retry. A single large object can then use the whole process budget. Plans are lazy; the scheduler does not allocate one future or file per cell. Empty objects use one whole-object GET to verify existence, size and ETag.

Supply exact sizes from trusted metadata, with etag, version_id and sha256 when available. Empty/dot key segments are rejected to preserve wire identity. Every response validates the full object size, returned range, byte count and strong ETag. Retries overwrite the complete original cell using If-Match. SHA-256 is verified after all cells complete. The batch is published as a new directory only after every object succeeds. Failed batches discard staging after outstanding workers finish. Omit the destination to use VXS3_CACHE_DIR/downloads/<uuid>.

One parent-owned breaker covers each manifest instance name, shared across workers and successive batches. Each name maps to one unique HTTPS origin in the pool’s fixed inventory. Defaults:

  • fail_max=3: three consecutive transient failures remove the instance from the active ring. Retries route the same cell through the remaining instances.
  • reset_timeout=30: after 30 seconds, a request for the original owner can probe it. One probe runs per instance; success restores membership and an unsuccessful in-flight probe starts another cooldown. A probe cancelled before execution remains immediately eligible. If all circuits are open, any due instance can probe.
  • attempts=4: at most four attempts per cell, including probes. Backoff is 0.25, 0.5, 1, 2, then at most 4 seconds, shared with the direct downloader. Waiting retries occupy no process and do not block other ready cells. An attempt limit below the failure threshold can stop before eviction. If no instance or probe is available, the batch fails promptly; retry later.

Connection/stream failures, timeouts, HTTP 408/429 and server errors count toward eviction. Authentication, missing objects, other request errors, local disk errors and integrity failures fail the batch without evicting a node. Obstore’s typed exceptions take precedence; its unmapped HTTP status text is covered by real-server compatibility tests. Unrecognized native errors fail without eviction or exposing native messages. In-flight results from before an eviction cannot undo a later trip or probe. active_instances and circuit_states expose snapshots. Separate pools can have different health views and temporarily route the same cell differently. Once D3N routing is selected, exhausted circuits do not switch the batch to the gateway.

401/403 raise DownloadAuthorizationError with an etags map keyed by local output name, including validators learned while draining outstanding work. Callers can refresh credentials and retry with those ETags. A 412, changed or unusable ETag, invalid size/range, or SHA-256 mismatch raises DownloadIntegrityError. Other transfer failures raise DownloadError. A crashed child fails the current batch and discards the broken executor; the next download() on the same pool starts fresh processes.

For objects without presigned_url, credentials come from obstore’s environment lookup or s3_config, including session tokens. Child-created S3 stores override configured/environment endpoints and buckets with the selected manifest instance and use path-style addressing. Caller-created stores and credential callbacks are not accepted. The parent resolves transport.client_options() once and passes the resulting options and CA bytes to every child: SSL_CERT_FILE and hashed SSL_CERT_DIR certificates supplement backend roots unless root_certificate is explicit. Defaults are a 5-second connect timeout, 300-second read-stall timeout and 300-second total per-range timeout. Explicit options override these limits. Slow connections can need a larger per-range budget; total object size does not enlarge individual requests. Obstore retries are disabled so the parent controls every attempt. Close the pool or use its context manager to join children. Concurrent calls on one pool are rejected.

Presigned requests and discovery

Supply a presigned GET as a credential on the object:

objects = {
    "shard.tar": DownloadObject(
        "example",
        "run/shard.tar",
        size_bytes,
        presigned_url=signed_url,
    ),
}
with DownloadPool(workers=4, routing="auto") as pool:
    local = pool.download(objects, "./downloaded")

Run this inside the guarded main() from the example above. The caller obtains the URL and exact size; vxs3 owns inventory discovery, cell mapping and failover. One URL per object suffices across ranges and nodes. The original signed Host (including its original port), encoded path and query are preserved. Only the connection origin changes to the selected manifest endpoint. The child uses obstore HTTPStore with HTTP/1, verifies TLS against the selected endpoint’s hostname using the shared CA roots, and does not add AWS Authorization headers. Caller-supplied Host/Authorization defaults are replaced/removed for these routed requests. No S3 credentials are required in the parent or child.

Routed URLs must be HTTPS and path-style, matching the supplied logical bucket and key. If X-Amz-SignedHeaders is present it must be host: range and ETag conditions vary between requests. versionId is decoded for routing and checked against an explicit version_id, while its original encoding remains in the query. Native exceptions and object representations never expose the URL. 401/403 still return the learned ETags; the caller can renew URLs and retry on the same pool without losing its inventory or breaker state.

An explicit instances sequence is authoritative. Otherwise the pool calls vxs3.transport.instances.discover_instances(): first VXS3_D3N_INSTANCES, then /etc/virdx/d3n/d3n_instances.json. A missing environment-selected file, empty environment value or malformed inventory raises an error. Discovery returns None only when neither an environment path nor the mounted file is present. Each pool loads a fixed snapshot once, including across URL renewal.

  • routing="d3n" (the default): require discovered or explicit instances. Missing deployment inventory raises DownloadError before any transfer.
  • routing="auto": use discovered instances; without an inventory, use the original presigned URLs through the existing concurrent range downloader. This gateway path requires presigned sources, starts no child processes and has no per-D3N breakers (active_instances == (), circuit_states == {}).
  • routing="gateway": use the original URLs and skip discovery. Cannot be combined with an explicit D3N instance list.

The gateway path uses an async bridge so the blocking pool method works inside an active notebook event loop. Interruption cancels the batch and waits for its tasks to finish cleanup before returning. Outside a notebook, it uses the calling thread’s event loop runner. allow_http=True remains required for local HTTP gateway URLs; D3N endpoints always require HTTPS. Never use the bundled snapshot as an implicit production membership source; mount or explicitly configure inventory.

Local TLS tests verify SigV4, range placement, failover and renewal invariants. Live D3N throughput and API-issued credential renewal have not been tested. The vxdata SDK and Zarr integrations remain separate adapters using the same range grid; SDK adoption of this pool is a separate change.

Errors, progress and async use

Bundle operations expose ManifestError, IntegrityError and StorageError under Vxs3Error. Invalid arguments can raise ValueError; local filesystem errors can propagate. Zarr transfers expose ZarrTransferError.

Operations accepting progress=callback emit Progress records with phase, completed/total counts and an optional relative path. extract counts files per completed batch and materialize per resolved file, each starting at zero of the total requested. Callbacks run synchronously and can interrupt work by raising. The last bundle publication event is emitted before the manifest PUT; no callback runs after successful publication. Use the returned reference as the completion signal.

High-level bundle and Zarr operations are blocking; async applications can run them in a worker thread. For direct downloads in Jupyter or an async application, use top-level await download_async(...). It has the same destination, concurrency, retry and integrity behaviour as download(). Do not use asyncio.run() or the blocking wrapper inside an existing event loop.

Navigation

Type to search…

↑↓ navigate↵ selectEsc close