---
title: "vxs3"
image: "https://docs.virdx.dev/img/virdx-social-card.png"
---

> Documentation Index
> Fetch the complete documentation index at: https://docs.virdx.dev/llms.txt
> Use this file to discover all available pages before exploring further.

# vxs3

{/* Vendored from mono/packages/vxs3/README.md @ 86565ca8aaf6ae190ae41de8acd8e0742f31872f by scripts/sync-docs.sh. Do not edit here. */}

## Overview

`vxs3` is Virdx's Python library for preparing, inspecting and transferring data on S3-compatible storage, including our Ceph setup.

Proprietary to Virdx; internal distribution only.

JavaScript/TypeScript consumers can use the read-only
[`@virdx/vxs3-js` package](https://github.com/virdx/mono/blob/86565ca8aaf6ae190ae41de8acd8e0742f31872f/packages/vxs3-js/README.md), published privately to GitHub Packages.
Both implementations consume the same Ketama, routing and grid/stripe fixtures
under `tests/fixtures`; their package versions are independent.

## Goals

Storing files in Ceph/S3/Swift/MinIO is easy, storing them optimally is more challenging. Many of our files are heavily fragmented which makes retrieval inefficient and content harder to cache.

Therefore for each type of file (or collection of files) we should have a preferred storage mechanism along with any additional optimisation if required.

For `vxs3` there are three main workflows that should encompass nearly all of our needs:

1. **Individual objects**: Single files, typically larger than 4 MiB
2. **Bundles**: Collection of arbitrary files or directories
3. **Zarr v3**: Zarr native sharding

File formats like NIfTI are typically large enough to just be single files, so nothing special is required here. Bundles and native Zarr sharding are intended to cover most remaining cases.

## Setup

Install `vxs3` from the internal Artifactory conda channel. It includes obstore for S3 transfers and analysis; there is no separate core package.

The examples below use the vxs3 2.x module layout.

Zarr optimisation and streaming additionally need `numpy >=2,<3` and `zarr >=3.1.3,<3.3` in the conda environment. Internal Python wheel installations can request `vxs3[zarr]` (or `vxs3[all]`). Generic bundle and ordinary-object operations do not need these array dependencies.

Choose a local directory with enough free disk space for downloads and temporary data:

```bash
export VXS3_CACHE_DIR="$HOME/.cache/vxs3"
```

- Jobs submitted through `run-image` or `run-batch` with `local_cache` enabled automatically get `VXS3_CACHE_DIR=/cache/.vxs3`, alongside `VXRUN_CACHE_DIR=/cache` (legacy alias `KUBERUN_CACHE`). With a custom template, set it explicitly.
- Explicit destination/cache/scratch arguments override this setting.
- Operations that need a default location raise `ValueError` if it is unset or blank.
- Download outputs and member caches persist; temporary scratch is cleaned up after each operation.

For the S3 examples below, configure an `obstore` store with your endpoint and credentials. Credentials can come from the environment or a credential provider:

```python
from obstore.store import S3Store
from vxs3.transport.s3 import S3Destination, S3Source

store = S3Store(
"example-bucket",
endpoint="https://s3.fra.virdx.dev",
region="eu-central-1",
virtual_hosted_style_request=False,  # Use path-style bucket addressing.
retry_config={"max_retries": 0},
)
```

`S3Source`, `S3Destination` and `S3ExactDestination` accept either a full URI such as `s3://example-bucket/original/image.zarr/` or a relative prefix such as `original/image.zarr`. These select directory prefixes or output parents. The URI bucket must match `store`; if the store has its own prefix, full URIs must include it and relative paths are resolved within it. Keep the endpoint and credentials configured on `S3Store`. URI paths are literal object keys, without percent-decoding.

## Bundles

A bundle contains a manifest and uncompressed PAX-format tar shards, which store file data with headers and padding between files. The more files we have to keep track of (in vxdata or Ceph) and the more requests we make (to vxdata, Ceph or AWS S3) the harder it is to optimise. Bundles allow us to aggregate files together simplifying management and reducing the burden on our storage systems.

The format of a bundle is quite simple: we pack files together targeting 1GiB shards, if a file is larger than 1GiB it gets a shard by itself, provided it fits the 4GiB limit including tar overhead. Each shard is assembled and uploaded along with a manifest describing the layout/contents of the shards. The collection of shards is called a bundle.

## Zarr

Zarr v3 supports sharding natively, therefore the optimisation process is both a storage optimisation and a potential upgrade to Zarr v3.

The optimisation logic is quite simple: as with bundles we are targeting 1GiB objects where possible, grouping until we reach the desired size per object.

## Interfaces

| Task | Interface | Result |
| --- | --- | --- |
| Inspect data | `vxs3.analyze(source)` | `AnalysisReport` with JSON and Markdown renderers |
| Pack / restore local files | `vxs3.bundle.local.pack()` / `unpack()` | Manifest path / restored directory path |
| Verify a local bundle | `vxs3.bundle.local.verify_bundle()` | Validated `Manifest`; no extraction output |
| Read a manifest | `vxs3.bundle.manifest.read_manifest()` / `parse_manifest()` | File, directory and shard inventory |
| Publish / open a remote bundle | `vxs3.bundle.io.publish_bundle()` / `open_bundle()` | `BundleRef` / `Bundle` |
| Complete a batch of bundles | `vxs3.bundle.manifest.merge_manifests()` / `vxs3.bundle.io.publish_manifest()` | Merged `Manifest` / `BundleRef` of the root manifest |
| Browse a bundle | `Bundle.list_entries()` / `get_member()` | Inventory entries / one file's metadata |
| Read bundle contents | `Bundle.materialize()` / `open()` / `download()` | Cached file path / read-only stream / complete restored directory |
| Stream bundle contents | `Bundle.read_range()` / `vxs3.zarr.store.BundleStore` | Bounded in-memory bytes / read-only Zarr Store |
| Inspect / optimise local Zarr | `vxs3.zarr.workflows.plan_zarr()` / `optimize_local_zarr()` | `ZarrPlan` / output path |
| Transfer native Zarr | `upload_zarr()` / `download_zarr()` / `optimize_zarr()` in `vxs3.zarr.workflows` | `ZarrResult` for writes; local path for downloads |
| Download ordinary objects | `vxs3.transport.download.download()` / `download_async()` | Local directory containing the requested files |

Most interfaces are Python APIs. The current command-line interface provides `vxs3 analyze`; pack, upload and download workflows use Python.

## Workflow

### 1. Inspect before choosing a layout

```python
from vxs3 import AnalysisLimits, analyze

report = analyze("./input", limits=AnalysisLimits(max_objects=100_000))
print(report.to_markdown())
print(report.coverage["status"])  # complete, partial or failed

# with the S3 store configured above:
remote_report = analyze(S3Source(store, "datasets/example"))
```

Analysis reads inventories and bounded metadata, reporting file types, object sizes and suggested actions. It does not convert or upload data. Reading a bundle manifest reveals its logical files without downloading shards; that alone does not verify their contents.

The equivalent CLI is:

```bash
vxs3 analyze ./input
vxs3 analyze ./input --json --output report.json
vxs3 analyze s3://example-bucket/datasets/example/
```

For S3 CLI inputs, a trailing slash selects a prefix; otherwise the URI selects one object. Report files must be new. Exit codes are `0` for complete, `2` for partial and `1` for failed. Check coverage before relying on totals or recommendations.

### 2. Bundle ordinary files

```python
from vxs3.bundle.local import pack, unpack, verify_bundle
from vxs3.bundle.manifest import read_manifest

manifest = pack("./input", "./packed")
inventory = read_manifest(manifest)  # metadata only; no tar reads
for entry in inventory.files:
print(entry.path, entry.size)

verify_bundle(manifest)  # check local shards without extracting a copy
restored = unpack(manifest, "./restored")
```

`pack(..., workers=8)` writes shards in up to 8 worker processes (default: `VXS3_WORKERS`, else 0 for in-process). Worker processes import the calling module, so call it from a script under `if __name__ == "__main__":`.

`./packed` and `./restored` must not already exist. A directory restores its contents; a single-file input restores its basename inside the destination.

A bundle contains `manifest.parquet` (one row per file or directory, bundle facts in the footer) and uncompressed tar shards. Defaults are a **1 GiB shard target**, **4 GiB maximum including tar overhead**, and **100,000 entries per shard**. Files stay whole: a file that cannot fit fails. File bytes, relative paths and empty directories survive; filesystem permissions, timestamps and ownership do not. Symlinks and special files are rejected.

Bundles can include TIFF, NIfTI and Zarr stores, including mixed directory trees.
Packing preserves file bytes; it does not validate Zarr metadata, convert formats
or change chunking. Use the native workflow below when optimisation is needed.

### 3. Publish a bundle and read it remotely

Continue with `manifest` from the previous example and `store` from Setup:

```python
from vxs3.bundle.io import open_bundle, publish_bundle

ref = publish_bundle(
manifest,
S3Destination(store, "bundles"),
scratch_dir="./scratch",  # Optional; defaults to VXS3_CACHE_DIR.
concurrency=8,  # Optional; defaults to 8.
)
print(ref.manifest_uri)
print(ref.sha256)

# the reference supplies both the manifest URI and its expected SHA-256
bundle = open_bundle(ref, store=store)

for entry in bundle.list_entries(offset=0, limit=100):
print(entry.path)

# replace these member paths with files present in your input
info = bundle.get_member("images/example.tiff")  # metadata lookup only
paths = bundle.materialize(["images/example.tiff"])  # {path: local cached file}
with bundle.open("labels/example.json", "r") as stream:
text = stream.read()
files = bundle.extract("./selected", ["images/example.tiff"])  # into a directory, no cache

all_files = bundle.download("./remote-restored")
```

`scratch_dir` and `concurrency` are optional. Omit `scratch_dir` to use `VXS3_CACHE_DIR`; publication raises `ValueError` if neither is configured. Omit `concurrency` to use `8`; bundle downloads and member reads also default to `8`.

The manifest URI and SHA-256 can be used later to reopen the same bundle. Pass the reference as above, or use `open_bundle(ref.manifest_uri, store=store, sha256=ref.sha256)` to check the exact stored manifest bytes.

Publication uploads four shards at a time, sharing the `concurrency` streams between them (the default `8` gives each shard two part streams), and publishes the manifest last. Each shard is verified against the store's ETag as it uploads; one the store cannot attest (for example under SSE-KMS) is read back at once, and `read_back=True` downloads everything again. Publication scratch holds only those read-back copies, at most four shards, in addition to the existing local bundle. `upload_bundle(source, destination, workers=8)` packs and publishes in one pass with the same fan-out: each shard uploads as soon as it is written and is deleted locally once verified, so scratch holds only the shards in flight rather than the whole bundle.

`bundle.download("./remote-restored")` restores the complete input into that new directory. Member reads (`materialize`, `open`) instead fetch only the files' byte ranges, verify each file and keep it in a persistent cache under `VXS3_CACHE_DIR` (or an explicit `cache_dir`); later reads reuse it. Cached paths are read-only; copy files before editing. Cache eviction is manual. `extract` writes chosen files (or everything) into a directory without the cache. `extract(..., workers=8)` downloads each shard whole into a staging directory and cuts it in up to 8 worker processes that map the staged file; the default is `VXS3_WORKERS`, else 0 for ranged reads on threads. Call it from a script under `if __name__ == "__main__":`. A selection sends a shard to a worker only when at least a quarter of its file bytes are wanted. The staging directory is, in order: `scratch_dir` when passed; `/dev/shm` when `VXS3_SHM` is unset or on and it has room for `workers + 1` shards plus a margin (memory, nothing written to disk); `staging/` under `VXS3_CACHE_DIR` (node-local NVMe in jobs); else the system temporary directory. Set `VXS3_SHM=0`, or pass `shm=False`, on a memory-bound host; `shm=True` keeps the room check. Bundles published with the earlier `manifest.json.gz` (v3) still open. See [storage and caching details](/vx/cluster/vxs3-storage/).

### Stream Zarr from a bundle

```python
import zarr
from vxs3.zarr.store import BundleStore

# bundle is the pinned Bundle returned by open_bundle above.
store = BundleStore(bundle, prefix="images/example.zarr")
group = zarr.open_group(store=store, mode="r")
tile = group["0"][0:512, 0:512]
store.close()
```

`BundleStore` supports reads of Zarr v2/v3, including native sharding, through
the installed Zarr library and codecs. It fetches metadata/chunk ranges into
memory without extraction, disk caching or `VXS3_CACHE_DIR`. Published bundles
and the store are read-only. Other Zarr consumers need a compatible adapter.

Defaults are `max_read_bytes=64 * 2**20` per encoded value/range,
`concurrency=8` active requests and `cache_bytes=16 * 2**20` for a bounded
in-memory LRU of small reads. Increase the explicit read budget if an operation
requires a larger encoded chunk or a complete native shard. Decoded arrays use
additional memory. Full member reads verify SHA-256; partial reads validate
exact ranges and pin a strong shard ETag, but cannot verify the full-member
hash. See [streaming semantics](/vx/cluster/vxs3-storage/#streaming-member-and-zarr-reads).

D3N streaming reads expand to complete 4 MiB storage stripes and return only
the requested bytes. The SDK selects this with `download_routing="d3n"`;
for a caller-configured Ceph endpoint, use `S3Source(store, prefix, d3n=True)`.
See [D3N caching](https://github.com/virdx/mono/blob/86565ca8aaf6ae190ae41de8acd8e0742f31872f/packages/vxs3/D3N.md) for alignment, routing, memory and transfer costs.

For other formats, `bundle.read_range("image.tiff", 0, 4096)` returns bounded
bytes directly. Existing `open`, `materialize` and `download` retain their
verified local-file workflows.

### 3b. Complete a batch of bundles as one

When independent workers each publish a bundle beneath `items/<id>/` of a shared
prefix, a finaliser describes them all with one manifest at that prefix. No shard
is read or moved; the merged manifest only prefixes the parts' paths and keys.

```python
from vxs3.bundle.io import open_bundle, publish_manifest
from vxs3.bundle.manifest import merge_manifests
from vxs3.transport.s3 import S3ExactDestination

parts = {
f"items/{item}": open_bundle(ref, store=store).inventory  # each part's BundleRef
for item, ref in item_refs.items()
}
root = publish_manifest(
merge_manifests(parts),
S3ExactDestination(store, "batches/run-1"),  # the parts' parent prefix
scratch_dir="./scratch",  # Optional; defaults to VXS3_CACHE_DIR.
)
batch = open_bundle(root, store=store)
batch.list_entries("items/a/")  # one item's files
```

Every referenced shard must already be stored at its key with its exact size;
the manifest is written last and read back. Readers of a composed bundle need
vxs3 3.1 or later. See [composition](/vx/cluster/vxs3-storage/#composition), and the SDK's
[writing one bundle from many workers](/vx/cluster/vxdata-sdk/#writing-one-bundle-from-many-workers)
for the full worker and finaliser flow including catalog registration.

### 4. Work with native Zarr

For local planning and optimisation:

```python
from vxs3.zarr.workflows import optimize_local_zarr, plan_zarr

plan = plan_zarr("./image.zarr")
print(plan.to_json())
output = optimize_local_zarr("./image.zarr", "./optimised.zarr")
```

For S3/Ceph, use the store from Setup. These examples omit the optional `scratch_dir` and `concurrency`: upload/optimisation uses `VXS3_CACHE_DIR` for scratch, and all three transfer functions default to `concurrency=8`. Pass `scratch_dir` to override the location; upload/optimisation raises `ValueError` if neither it nor `VXS3_CACHE_DIR` is configured.

```python
from vxs3.zarr.workflows import download_zarr, optimize_zarr, upload_zarr

# optimise and verify a local store, then publish a new remote copy
result = upload_zarr("./image.zarr", S3Destination(store, "uploads"))
print(result.destination)  # native Zarr store URI
print(result.completion)  # e.g. image.zarr.vxs3.json beside image.zarr/

# download the same Zarr unchanged into a new local directory
local = download_zarr(S3Source(store, result.destination), "./downloaded.zarr")

# optimise an existing remote store into another fresh location
optimized = optimize_zarr(
S3Source(store, "original/image.zarr"),
S3Destination(store, "optimised"),
)
```

Upload and optimisation preserve decoded values and supported lossless codec parameters, and verify uploaded bytes. Resharding can change the encoded bytes. Supported Zarr v2 stores migrate to v3; unsupported codecs or layouts fail explicitly. Download preserves the existing representation. `ZarrPolicy` configures layout and memory budgets.

Remote optimisation stages the full source, rewritten output and a verification batch locally. Allow space for all three; there is no fixed size multiplier because the output layout and compression affect its size.

Outputs must use fresh locations; `overwrite=True` is unsupported. `result.completion` points to a sibling JSON record describing the conversion, uploaded after verification. Keep the source unchanged during the operation. See [Zarr support and limitations](https://github.com/virdx/mono/blob/86565ca8aaf6ae190ae41de8acd8e0742f31872f/packages/vxs3/ZARR_OPTIMISATION.md).

### 5. Download individual objects

```python
from vxs3.transport.download import Source, download

# obtain the signed GET URL and exact byte size from vxdata
sources = {
"image.tiff": Source.presigned(presigned_url, size=size_bytes),
}
local = download(sources, "./files", concurrency=8)  # optional; defaults to 8
image_path = local / "image.tiff"
```

For direct S3 credentials, use `Source(store, "images/image.tiff", size=size_bytes)` instead.

Mapping keys are local relative filenames. Supply `sha256` when available for content verification; bundle helpers supply it automatically from the manifest.

The direct vxs3 downloader uses a fixed **256 MiB range grid** and a shared concurrency budget. Size, range responses and ETags are checked, with optional SHA-256 verification, before publishing the destination directory. There is no persistent resume or automatic presigned-URL refresh; callers provide fresh URLs after expiry.

In Jupyter or another running event loop, replace the blocking download call with:

```python
from vxs3.transport.download import download_async

local = await download_async(sources, "./notebook-files", concurrency=8)  # optional; defaults to 8
```

## Ketama routing

The routing functions select a direct RGW cache instance for an object or range.
They use a [shared JSON list](https://github.com/virdx/mono/blob/86565ca8aaf6ae190ae41de8acd8e0742f31872f/packages/vxs3/src/vxs3/d3n_instances.json) of the 12
HAProxy D3N instances and a portable Ketama ring, with request-key construction
separate from hashing and selection.
Routing is included in the base Python and conda packages. Ring construction
and lookup use only the Python standard library, with no extra dependencies.
The bundled manifest is loaded and cached on the first selection that omits
`instances`. Importing the module or passing explicit instances does not read it.

```python
from vxs3.transport._ranges import download_ranges
from vxs3.transport.routing import select_instance

for part in download_ranges(size=1024**3):
instance = select_instance("example", "shard.tar", range_start=part.start)
print(instance.name, instance.endpoint, part.range_header)
```

These selection functions perform no I/O. The separate process pool below uses
them for downloads and failover. Ketama assignments differ from HAProxy's ring.
Mixed access paths can duplicate cached data; see [gateway alignment](https://github.com/virdx/mono/blob/86565ca8aaf6ae190ae41de8acd8e0742f31872f/packages/vxs3/ROUTING.md#gateway-alignment).
See [the routing contract](https://github.com/virdx/mono/blob/86565ca8aaf6ae190ae41de8acd8e0742f31872f/packages/vxs3/ROUTING.md) for the exact bytes/settings and shared
Python/JavaScript test vectors. `vxs3.transport._ketama.KetamaRing` also works independently
of the S3 request policy with a supplied list of equal-weight node names.
The verified JavaScript library is [hashring 3.2.0](https://github.com/virdx/mono/blob/86565ca8aaf6ae190ae41de8acd8e0742f31872f/packages/vxs3/ROUTING.md#javascript-library);
the contract documents its exact settings and a runnable compatibility check.
Kubernetes clients can mount the Ansible-managed `d3n-instances` ConfigMap.
Pass `load_instances("/etc/virdx/d3n/d3n_instances.json")` from
`vxs3.transport.instances` as `select_instance(..., instances=...)`; see
[manifest loading and distribution](https://github.com/virdx/mono/blob/86565ca8aaf6ae190ae41de8acd8e0742f31872f/packages/vxs3/ROUTING.md#shared-instance-manifest).

## D3N object worker pool

`vxs3.transport.workers.DownloadPool` downloads known-size objects through direct D3N
instances, routing each 256 MiB cell through Ketama. A shared process budget,
bounded retries and per-instance circuit breakers cover the batch; completed
files are published together after verification.

Objects can carry `presigned_url` instead of using S3 credentials. The pool
preserves the signed Host/path/query while connecting to the selected D3N node.
It discovers `VXS3_D3N_INSTANCES` or the mounted D3N inventory. Use
`routing="auto"` to retain the original gateway URLs when no inventory exists;
default `routing="d3n"` requires a deployment inventory or explicit `instances`.

Install `vxs3[workers]`, or add `pybreaker>=1.4,<2` from conda-forge alongside
`vxs3`. Run the pool from a guarded Python script. See the
[worker example and transport details](/vx/cluster/vxs3-storage/#d3n-object-worker-pool) for
configuration, errors and recovery.

## Further detail

For authenticated uploads and catalog registration through vxdata, use the
[vxdata SDK bundle guide](/vx/cluster/vxdata-sdk/#explicit-bundles).

- [Storage, caching and transfer details](/vx/cluster/vxs3-storage/): adapters, publication, local storage, retries and async use.
- [D3N caching](https://github.com/virdx/mono/blob/86565ca8aaf6ae190ae41de8acd8e0742f31872f/packages/vxs3/D3N.md): Ceph cache behavior, canonical stripe reads and routing.
- [Bundle format](https://github.com/virdx/mono/blob/86565ca8aaf6ae190ae41de8acd8e0742f31872f/packages/vxs3/FORMAT.md): shard layout, manifests, validation and compatibility.
- [Native Zarr](https://github.com/virdx/mono/blob/86565ca8aaf6ae190ae41de8acd8e0742f31872f/packages/vxs3/ZARR_OPTIMISATION.md): policies, codecs, migration and completion records.
- [Analysis](https://github.com/virdx/mono/blob/86565ca8aaf6ae190ae41de8acd8e0742f31872f/packages/vxs3/ANALYSIS.md) and [supported formats](/vx/cluster/vxs3-formats/).
- [Six basic examples](https://github.com/virdx/mono/blob/86565ca8aaf6ae190ae41de8acd8e0742f31872f/packages/vxs3/examples/README.md): upload and download files, bundles and Zarr.
- [Maintainer guide](https://github.com/virdx/mono/blob/86565ca8aaf6ae190ae41de8acd8e0742f31872f/packages/vxs3/AGENTS.md): implementation boundaries, invariants and development commands.

[NEXT_VERSION.md](https://github.com/virdx/mono/blob/86565ca8aaf6ae190ae41de8acd8e0742f31872f/packages/vxs3/NEXT_VERSION.md) describes proposed work; its planned interfaces
are not part of the current API.

Source: https://docs.virdx.dev/vx/cluster/vxs3/index.mdx
