Skip to content
VIRDX

vxs3

Overview

vxs3 is Virdx’s Python library for preparing, inspecting and transferring data on S3-compatible storage, including our Ceph setup.

Proprietary to Virdx; internal distribution only.

JavaScript/TypeScript consumers can use the read-only @virdx/vxs3-js package, published privately to GitHub Packages. Both implementations consume the same Ketama, routing and grid/stripe fixtures under tests/fixtures; their package versions are independent.

Goals

Storing files in Ceph/S3/Swift/MinIO is easy, storing them optimally is more challenging. Many of our files are heavily fragmented which makes retrieval inefficient and content harder to cache.

Therefore for each type of file (or collection of files) we should have a preferred storage mechanism along with any additional optimisation if required.

For vxs3 there are three main workflows that should encompass nearly all of our needs:

  1. Individual objects: Single files, typically larger than 4 MiB
  2. Bundles: Collection of arbitrary files or directories
  3. Zarr v3: Zarr native sharding

File formats like NIfTI are typically large enough to just be single files, so nothing special is required here. Bundles and native Zarr sharding are intended to cover most remaining cases.

Setup

Install vxs3 from the internal Artifactory conda channel. It includes obstore for S3 transfers and analysis; there is no separate core package.

The examples below use the vxs3 2.x module layout.

Zarr optimisation and streaming additionally need numpy >=2,<3 and zarr >=3.1.3,<3.3 in the conda environment. Internal Python wheel installations can request vxs3[zarr] (or vxs3[all]). Generic bundle and ordinary-object operations do not need these array dependencies.

Choose a local directory with enough free disk space for downloads and temporary data:

export VXS3_CACHE_DIR="$HOME/.cache/vxs3"
  • Jobs submitted through run-image or run-batch with local_cache enabled automatically get VXS3_CACHE_DIR=/cache/.vxs3, alongside VXRUN_CACHE_DIR=/cache (legacy alias KUBERUN_CACHE). With a custom template, set it explicitly.
  • Explicit destination/cache/scratch arguments override this setting.
  • Operations that need a default location raise ValueError if it is unset or blank.
  • Download outputs and member caches persist; temporary scratch is cleaned up after each operation.

For the S3 examples below, configure an obstore store with your endpoint and credentials. Credentials can come from the environment or a credential provider:

from obstore.store import S3Store
from vxs3.transport.s3 import S3Destination, S3Source

store = S3Store(
    "example-bucket",
    endpoint="https://s3.fra.virdx.dev",
    region="eu-central-1",
    virtual_hosted_style_request=False,  # Use path-style bucket addressing.
    retry_config={"max_retries": 0},
)

S3Source, S3Destination and S3ExactDestination accept either a full URI such as s3://example-bucket/original/image.zarr/ or a relative prefix such as original/image.zarr. These select directory prefixes or output parents. The URI bucket must match store; if the store has its own prefix, full URIs must include it and relative paths are resolved within it. Keep the endpoint and credentials configured on S3Store. URI paths are literal object keys, without percent-decoding.

Bundles

A bundle contains a manifest and uncompressed PAX-format tar shards, which store file data with headers and padding between files. The more files we have to keep track of (in vxdata or Ceph) and the more requests we make (to vxdata, Ceph or AWS S3) the harder it is to optimise. Bundles allow us to aggregate files together simplifying management and reducing the burden on our storage systems.

The format of a bundle is quite simple: we pack files together targeting 1GiB shards, if a file is larger than 1GiB it gets a shard by itself, provided it fits the 4GiB limit including tar overhead. Each shard is assembled and uploaded along with a manifest describing the layout/contents of the shards. The collection of shards is called a bundle.

Zarr

Zarr v3 supports sharding natively, therefore the optimisation process is both a storage optimisation and a potential upgrade to Zarr v3.

The optimisation logic is quite simple: as with bundles we are targeting 1GiB objects where possible, grouping until we reach the desired size per object.

Interfaces

Task Interface Result
Inspect data vxs3.analyze(source) AnalysisReport with JSON and Markdown renderers
Pack / restore local files vxs3.bundle.local.pack() / unpack() Manifest path / restored directory path
Verify a local bundle vxs3.bundle.local.verify_bundle() Validated Manifest; no extraction output
Read a manifest vxs3.bundle.manifest.read_manifest() / parse_manifest() File, directory and shard inventory
Publish / open a remote bundle vxs3.bundle.io.publish_bundle() / open_bundle() BundleRef / Bundle
Complete a batch of bundles vxs3.bundle.manifest.merge_manifests() / vxs3.bundle.io.publish_manifest() Merged Manifest / BundleRef of the root manifest
Browse a bundle Bundle.list_entries() / get_member() Inventory entries / one file’s metadata
Read bundle contents Bundle.materialize() / open() / download() Cached file path / read-only stream / complete restored directory
Stream bundle contents Bundle.read_range() / vxs3.zarr.store.BundleStore Bounded in-memory bytes / read-only Zarr Store
Inspect / optimise local Zarr vxs3.zarr.workflows.plan_zarr() / optimize_local_zarr() ZarrPlan / output path
Transfer native Zarr upload_zarr() / download_zarr() / optimize_zarr() in vxs3.zarr.workflows ZarrResult for writes; local path for downloads
Download ordinary objects vxs3.transport.download.download() / download_async() Local directory containing the requested files

Most interfaces are Python APIs. The current command-line interface provides vxs3 analyze; pack, upload and download workflows use Python.

Workflow

1. Inspect before choosing a layout

from vxs3 import AnalysisLimits, analyze

report = analyze("./input", limits=AnalysisLimits(max_objects=100_000))
print(report.to_markdown())
print(report.coverage["status"])  # complete, partial or failed

# with the S3 store configured above:
remote_report = analyze(S3Source(store, "datasets/example"))

Analysis reads inventories and bounded metadata, reporting file types, object sizes and suggested actions. It does not convert or upload data. Reading a bundle manifest reveals its logical files without downloading shards; that alone does not verify their contents.

The equivalent CLI is:

vxs3 analyze ./input
vxs3 analyze ./input --json --output report.json
vxs3 analyze s3://example-bucket/datasets/example/

For S3 CLI inputs, a trailing slash selects a prefix; otherwise the URI selects one object. Report files must be new. Exit codes are 0 for complete, 2 for partial and 1 for failed. Check coverage before relying on totals or recommendations.

2. Bundle ordinary files

from vxs3.bundle.local import pack, unpack, verify_bundle
from vxs3.bundle.manifest import read_manifest

manifest = pack("./input", "./packed")
inventory = read_manifest(manifest)  # metadata only; no tar reads
for entry in inventory.files:
    print(entry.path, entry.size)

verify_bundle(manifest)  # check local shards without extracting a copy
restored = unpack(manifest, "./restored")

pack(..., workers=8) writes shards in up to 8 worker processes (default: VXS3_WORKERS, else 0 for in-process). Worker processes import the calling module, so call it from a script under if __name__ == "__main__":.

./packed and ./restored must not already exist. A directory restores its contents; a single-file input restores its basename inside the destination.

A bundle contains manifest.parquet (one row per file or directory, bundle facts in the footer) and uncompressed tar shards. Defaults are a 1 GiB shard target, 4 GiB maximum including tar overhead, and 100,000 entries per shard. Files stay whole: a file that cannot fit fails. File bytes, relative paths and empty directories survive; filesystem permissions, timestamps and ownership do not. Symlinks and special files are rejected.

Bundles can include TIFF, NIfTI and Zarr stores, including mixed directory trees. Packing preserves file bytes; it does not validate Zarr metadata, convert formats or change chunking. Use the native workflow below when optimisation is needed.

3. Publish a bundle and read it remotely

Continue with manifest from the previous example and store from Setup:

from vxs3.bundle.io import open_bundle, publish_bundle

ref = publish_bundle(
    manifest,
    S3Destination(store, "bundles"),
    scratch_dir="./scratch",  # Optional; defaults to VXS3_CACHE_DIR.
    concurrency=8,  # Optional; defaults to 8.
)
print(ref.manifest_uri)
print(ref.sha256)

# the reference supplies both the manifest URI and its expected SHA-256
bundle = open_bundle(ref, store=store)

for entry in bundle.list_entries(offset=0, limit=100):
    print(entry.path)

# replace these member paths with files present in your input
info = bundle.get_member("images/example.tiff")  # metadata lookup only
paths = bundle.materialize(["images/example.tiff"])  # {path: local cached file}
with bundle.open("labels/example.json", "r") as stream:
    text = stream.read()
files = bundle.extract("./selected", ["images/example.tiff"])  # into a directory, no cache

all_files = bundle.download("./remote-restored")

scratch_dir and concurrency are optional. Omit scratch_dir to use VXS3_CACHE_DIR; publication raises ValueError if neither is configured. Omit concurrency to use 8; bundle downloads and member reads also default to 8.

The manifest URI and SHA-256 can be used later to reopen the same bundle. Pass the reference as above, or use open_bundle(ref.manifest_uri, store=store, sha256=ref.sha256) to check the exact stored manifest bytes.

Publication uploads four shards at a time, sharing the concurrency streams between them (the default 8 gives each shard two part streams), and publishes the manifest last. Each shard is verified against the store’s ETag as it uploads; one the store cannot attest (for example under SSE-KMS) is read back at once, and read_back=True downloads everything again. Publication scratch holds only those read-back copies, at most four shards, in addition to the existing local bundle. upload_bundle(source, destination, workers=8) packs and publishes in one pass with the same fan-out: each shard uploads as soon as it is written and is deleted locally once verified, so scratch holds only the shards in flight rather than the whole bundle.

bundle.download("./remote-restored") restores the complete input into that new directory. Member reads (materialize, open) instead fetch only the files’ byte ranges, verify each file and keep it in a persistent cache under VXS3_CACHE_DIR (or an explicit cache_dir); later reads reuse it. Cached paths are read-only; copy files before editing. Cache eviction is manual. extract writes chosen files (or everything) into a directory without the cache. extract(..., workers=8) downloads each shard whole into a staging directory and cuts it in up to 8 worker processes that map the staged file; the default is VXS3_WORKERS, else 0 for ranged reads on threads. Call it from a script under if __name__ == "__main__":. A selection sends a shard to a worker only when at least a quarter of its file bytes are wanted. The staging directory is, in order: scratch_dir when passed; /dev/shm when VXS3_SHM is unset or on and it has room for workers + 1 shards plus a margin (memory, nothing written to disk); staging/ under VXS3_CACHE_DIR (node-local NVMe in jobs); else the system temporary directory. Set VXS3_SHM=0, or pass shm=False, on a memory-bound host; shm=True keeps the room check. Bundles published with the earlier manifest.json.gz (v3) still open. See storage and caching details.

Stream Zarr from a bundle

import zarr
from vxs3.zarr.store import BundleStore

# bundle is the pinned Bundle returned by open_bundle above.
store = BundleStore(bundle, prefix="images/example.zarr")
group = zarr.open_group(store=store, mode="r")
tile = group["0"][0:512, 0:512]
store.close()

BundleStore supports reads of Zarr v2/v3, including native sharding, through the installed Zarr library and codecs. It fetches metadata/chunk ranges into memory without extraction, disk caching or VXS3_CACHE_DIR. Published bundles and the store are read-only. Other Zarr consumers need a compatible adapter.

Defaults are max_read_bytes=64 * 2**20 per encoded value/range, concurrency=8 active requests and cache_bytes=16 * 2**20 for a bounded in-memory LRU of small reads. Increase the explicit read budget if an operation requires a larger encoded chunk or a complete native shard. Decoded arrays use additional memory. Full member reads verify SHA-256; partial reads validate exact ranges and pin a strong shard ETag, but cannot verify the full-member hash. See streaming semantics.

D3N streaming reads expand to complete 4 MiB storage stripes and return only the requested bytes. The SDK selects this with download_routing="d3n"; for a caller-configured Ceph endpoint, use S3Source(store, prefix, d3n=True). See D3N caching for alignment, routing, memory and transfer costs.

For other formats, bundle.read_range("image.tiff", 0, 4096) returns bounded bytes directly. Existing open, materialize and download retain their verified local-file workflows.

3b. Complete a batch of bundles as one

When independent workers each publish a bundle beneath items/<id>/ of a shared prefix, a finaliser describes them all with one manifest at that prefix. No shard is read or moved; the merged manifest only prefixes the parts’ paths and keys.

from vxs3.bundle.io import open_bundle, publish_manifest
from vxs3.bundle.manifest import merge_manifests
from vxs3.transport.s3 import S3ExactDestination

parts = {
    f"items/{item}": open_bundle(ref, store=store).inventory  # each part's BundleRef
    for item, ref in item_refs.items()
}
root = publish_manifest(
    merge_manifests(parts),
    S3ExactDestination(store, "batches/run-1"),  # the parts' parent prefix
    scratch_dir="./scratch",  # Optional; defaults to VXS3_CACHE_DIR.
)
batch = open_bundle(root, store=store)
batch.list_entries("items/a/")  # one item's files

Every referenced shard must already be stored at its key with its exact size; the manifest is written last and read back. Readers of a composed bundle need vxs3 3.1 or later. See composition, and the SDK’s writing one bundle from many workers for the full worker and finaliser flow including catalog registration.

4. Work with native Zarr

For local planning and optimisation:

from vxs3.zarr.workflows import optimize_local_zarr, plan_zarr

plan = plan_zarr("./image.zarr")
print(plan.to_json())
output = optimize_local_zarr("./image.zarr", "./optimised.zarr")

For S3/Ceph, use the store from Setup. These examples omit the optional scratch_dir and concurrency: upload/optimisation uses VXS3_CACHE_DIR for scratch, and all three transfer functions default to concurrency=8. Pass scratch_dir to override the location; upload/optimisation raises ValueError if neither it nor VXS3_CACHE_DIR is configured.

from vxs3.zarr.workflows import download_zarr, optimize_zarr, upload_zarr

# optimise and verify a local store, then publish a new remote copy
result = upload_zarr("./image.zarr", S3Destination(store, "uploads"))
print(result.destination)  # native Zarr store URI
print(result.completion)  # e.g. image.zarr.vxs3.json beside image.zarr/

# download the same Zarr unchanged into a new local directory
local = download_zarr(S3Source(store, result.destination), "./downloaded.zarr")

# optimise an existing remote store into another fresh location
optimized = optimize_zarr(
    S3Source(store, "original/image.zarr"),
    S3Destination(store, "optimised"),
)

Upload and optimisation preserve decoded values and supported lossless codec parameters, and verify uploaded bytes. Resharding can change the encoded bytes. Supported Zarr v2 stores migrate to v3; unsupported codecs or layouts fail explicitly. Download preserves the existing representation. ZarrPolicy configures layout and memory budgets.

Remote optimisation stages the full source, rewritten output and a verification batch locally. Allow space for all three; there is no fixed size multiplier because the output layout and compression affect its size.

Outputs must use fresh locations; overwrite=True is unsupported. result.completion points to a sibling JSON record describing the conversion, uploaded after verification. Keep the source unchanged during the operation. See Zarr support and limitations.

5. Download individual objects

from vxs3.transport.download import Source, download

# obtain the signed GET URL and exact byte size from vxdata
sources = {
    "image.tiff": Source.presigned(presigned_url, size=size_bytes),
}
local = download(sources, "./files", concurrency=8)  # optional; defaults to 8
image_path = local / "image.tiff"

For direct S3 credentials, use Source(store, "images/image.tiff", size=size_bytes) instead.

Mapping keys are local relative filenames. Supply sha256 when available for content verification; bundle helpers supply it automatically from the manifest.

The direct vxs3 downloader uses a fixed 256 MiB range grid and a shared concurrency budget. Size, range responses and ETags are checked, with optional SHA-256 verification, before publishing the destination directory. There is no persistent resume or automatic presigned-URL refresh; callers provide fresh URLs after expiry.

In Jupyter or another running event loop, replace the blocking download call with:

from vxs3.transport.download import download_async

local = await download_async(sources, "./notebook-files", concurrency=8)  # optional; defaults to 8

Ketama routing

The routing functions select a direct RGW cache instance for an object or range. They use a shared JSON list of the 12 HAProxy D3N instances and a portable Ketama ring, with request-key construction separate from hashing and selection. Routing is included in the base Python and conda packages. Ring construction and lookup use only the Python standard library, with no extra dependencies. The bundled manifest is loaded and cached on the first selection that omits instances. Importing the module or passing explicit instances does not read it.

from vxs3.transport._ranges import download_ranges
from vxs3.transport.routing import select_instance

for part in download_ranges(size=1024**3):
    instance = select_instance("example", "shard.tar", range_start=part.start)
    print(instance.name, instance.endpoint, part.range_header)

These selection functions perform no I/O. The separate process pool below uses them for downloads and failover. Ketama assignments differ from HAProxy’s ring. Mixed access paths can duplicate cached data; see gateway alignment. See the routing contract for the exact bytes/settings and shared Python/JavaScript test vectors. vxs3.transport._ketama.KetamaRing also works independently of the S3 request policy with a supplied list of equal-weight node names. The verified JavaScript library is hashring 3.2.0; the contract documents its exact settings and a runnable compatibility check. Kubernetes clients can mount the Ansible-managed d3n-instances ConfigMap. Pass load_instances("/etc/virdx/d3n/d3n_instances.json") from vxs3.transport.instances as select_instance(..., instances=...); see manifest loading and distribution.

D3N object worker pool

vxs3.transport.workers.DownloadPool downloads known-size objects through direct D3N instances, routing each 256 MiB cell through Ketama. A shared process budget, bounded retries and per-instance circuit breakers cover the batch; completed files are published together after verification.

Objects can carry presigned_url instead of using S3 credentials. The pool preserves the signed Host/path/query while connecting to the selected D3N node. It discovers VXS3_D3N_INSTANCES or the mounted D3N inventory. Use routing="auto" to retain the original gateway URLs when no inventory exists; default routing="d3n" requires a deployment inventory or explicit instances.

Install vxs3[workers], or add pybreaker>=1.4,<2 from conda-forge alongside vxs3. Run the pool from a guarded Python script. See the worker example and transport details for configuration, errors and recovery.

Further detail

For authenticated uploads and catalog registration through vxdata, use the vxdata SDK bundle guide.

NEXT_VERSION.md describes proposed work; its planned interfaces are not part of the current API.

Navigation

Type to search…

↑↓ navigate↵ selectEsc close