Skip to content
VIRDX

vxdata-sdk — VxData Python client

Typed Python client for the VxData API (apps/vxdata-api), stainless-style: one Client exposing resource namespaces, with request/response types shared with the server via vxdata-schemas.

Quick start

from vxdata.sdk import Client

vxd = Client()  # API_URL and VXDATA_TOKEN from the environment or a .env file
vxd = Client(base_url="http://localhost:8000", token="...")  # explicit

vxd.ping()  # connectivity check (raises)
me = vxd.whoami()  # authenticated principal + grants

patient = vxd.resources.retrieve("patient/bb01")  # generic, any payload type
volumes = vxd.volumes.retrieve(["volume/a", "volume/b"])  # typed responses

Configuration resolves explicit argument > environment (.env included) > default. The constructor does no I/O. A client-level timestamp= is injected as as_of into every read/query unless overridden per call. Use the client as a context manager (or call close()) to release pooled connections.

For file collections, see explicit bundles: client.bundles packs, publishes and registers a Bundle with an optional records table, and reads files back selectively by byte range. The six basic examples cover uploading and downloading files, bundles and directories.

Namespaces

  • vxd.resources — generic CRUD over any payload type: retrieve, create, update, delete (soft, cascading), restore, query().
  • Typed namespaces — one per payload type (vxd.volumes, vxd.patients, vxd.imaging_studies, vxd.artefacts, …; the full name map lives in namespaces/registry.py). Same verbs with typed create/update/response models. Extras where the platform has them:
    • vxd.patients.register/resolve(external_uid, datasource_id) and vxd.imaging_studies.register/resolve(external_uid, patient_identifier) — server-side identifier allocation.
    • vxd.artefacts.attach_files(...) / vxd.artefacts.download(...).
    • vxd.bundles — catalog CRUD/query for published bundles plus upload, read_manifest, read_records, open and materialize; see explicit bundles.
  • vxd.lineage.parents(...) / vxd.lineage.children(...) — ancestry walks.
  • vxd.storage — blob transfer (below).

All verbs mirror the shape of their input: scalar in yields scalar out, list yields list, dict yields dict.

Queries

query() returns a lazy builder; nothing hits the server until collect:

from vxdata.sdk import ANY, F, has_ancestor

df = (
    vxd.pirads_lesion_assessments.query()
    .filter(F.score >= 4, has_ancestor("patient/bb01"))
    .sort("created_at", descending=True)
    .limit(100)
    .collect()  # polars DataFrame (server-side parquet)
)

models = vxd.volumes.query().filter(...).collect_as_pydantic()  # typed models

# Subqueries: pass a query builder to is_in, or compare via ANY/ALL
top = vxd.pirads_assessments.query().filter(F.score == 5).select("score")
worse = vxd.pirads_assessments.query().filter(F.score >= ANY(top)).collect()

F.<field> builds filters with comparison operators plus contains, startswith, endswith, between, before/after, is_null, is_in (lists, Series, or subqueries), and more; has_ancestor filters by lineage. head(n) is shorthand for limit(n).collect().

On JSON (dict/list) fields, JSON null counts as unset and ==/!=/is_in compare JSON values. contains tests list membership, a dict subset, or (with a string on a dict) text search; overlaps matches any list element. Ordering and startswith/endswith on JSON fields return a 400.

Blob storage (client.storage)

S3 transfer is presign-only: the SDK asks the API for presigned URLs and moves bytes directly against the object store; it never holds S3 credentials.

Listing objects and folders

listing = client.storage.list_objects("s3://vxdata/source-data/", recursive=False)
print(listing.prefixes)  # immediate folders, e.g. ["vxdata/source-data/leopard/"]
for obj in listing.objects:
    print(obj.key, obj.size)  # immediate objects; sizes in bytes

Recursive listing is the default; pass recursive=False to avoid traversing subfolders. Folder prefixes and object keys include the bucket but omit s3://. Use a trailing / for folder contents.

Transfers

url = client.storage.upload("weights.pt", group="my-project")  # -> s3:// url
local = client.storage.download(url, "./cache")  # Path
paths = client.storage.download([url_a, url_b], "./cache")  # list[Path]
root = client.storage.download("s3://bucket/g/run/", "./cache")  # whole prefix

Zarr stores can use these ordinary directory transfers to preserve their existing layout; keep the trailing / on the download URI. This does not optimise or validate Zarr or create a completion record. See the directory examples and vxs3 native Zarr workflow.

Downloads request a batch of presigned URLs and exact sizes from the API, then delegate transfers to vxs3’s DownloadPool. By default, requests go through the signed gateway URL without D3N discovery or worker processes. To enable direct D3N workers, set TransferConfig(download_routing="d3n"); see download routing below.

Vxs3 owns request ranges, concurrency, ETag/hash verification, staging and transient retries. The SDK renews rejected URLs once, retaining learned ETags and the pool’s node health state. The SDK preserves scalar/list/dict results and downloads into existing directories; files are replaced only after the transfer batch succeeds. There is no transaction across the final individual file replacements. Synchronous downloads work inside notebook event loops. Ctrl-C cancels gateway downloads and waits for cancellation before staging cleanup. When using D3N in scripts, put downloads and bundle operations inside if __name__ == "__main__": so spawned workers can import the script safely; the examples show this structure.

Uploads use the existing HTTP uploader: small files stream as one PUT, with multipart parts above 1 GiB. Both directions accept:

from vxdata.sdk import TransferConfig

client.storage.download(urls, dest, transfer_config=TransferConfig(max_streams=32))

max_streams (default 16) caps concurrent transfer workers across a call. timeout, max_attempts, and backoff now configure uploads only. Downloads use vxs3’s current retry policy; set download HTTP options through client_options (or client-wide storage_options, described under TLS below).

Downloads and bundles require vxs3[workers] >=1.2.0, including obstore and pybreaker. The conda SDK installs these dependencies; Python metadata exposes storage and bundles extras for internal wheel installations. NumPy and Zarr are optional vxs3 dependencies. Ordinary catalog imports remain lazy and do not load the transport.

Download routing

Choose routing per call with TransferConfig.download_routing. All modes use vxs3/obstore and retain range requests, retries and integrity checks.

download_routing Behavior
"gateway" (default) Use the signed gateway URL, skipping discovery and worker processes.
"d3n" Require a configured or mounted D3N inventory; fail if none is available.
"auto" Use discovered D3N instances; use the gateway when no inventory is configured.

To opt into parallel D3N downloads from a script:

from vxdata.sdk import Client, TransferConfig


def main():
    with Client() as client:
        client.storage.download(
            ["s3://bucket/path/a.tif", "s3://bucket/path/b.tif"],
            "./downloads",
            transfer_config=TransferConfig(download_routing="d3n", max_streams=8),
        )


if __name__ == "__main__":
    main()

Vxs3 discovers instances from VXS3_D3N_INSTANCES, then the mounted /etc/virdx/d3n/d3n_instances.json. In d3n mode, missing inventory is an error; the bundled development map is never implicit production membership. D3N endpoints must be reachable from the client, and routed presigned URLs must use HTTPS. For local MinIO with HTTP URLs, select gateway (or auto without an inventory) and set client_options={"allow_http": True}. An explicitly configured missing or invalid inventory raises an error in d3n and auto modes; gateway ignores inventory configuration.

The API keeps signing the gateway hostname. Vxs3 preserves the signed Host while connecting to the selected D3N node; no S3 credentials are needed. See vxs3 worker routing for deployment details.

The setting also applies to open, materialize, bundles.upload readback and shard downloads from bundles.open. Pass transfer_config when opening the bundle; subsequent streaming reads and downloads reuse it. Bundle.read_range and open_zarr_store honor routing, using aligned stripes in D3N mode. Bundle metadata and the separate materialize/extract read_ranges path stay on the gateway. See D3N cache behavior and request policy.

Each SDK client creates its download pool lazily and reuses its processes and node health across downloads, bundle readback and bulk shard reads. Batches sharing the pool run serially. Changing concurrency, routing or transport options replaces the pool. Use with Client() as client: or call client.close() to join workers; perform bundle reads while their client remains open. Streaming has a separate shared transport session with bounded reusable HTTP clients and its own node health; it does not use the bulk worker processes.

Explicit bundles

The vxs3 README documents the underlying format and direct S3 interfaces; this section covers the authenticated vxdata workflow. A Bundle is one vxs3 bundle (tar shards plus a parquet manifest of files) registered as bundle/<uuid4>, optionally with a records table: your own rows, of any grain, whose path_* columns name bundle files. Query the records, then fetch only the files they name.

import polars as pl
from vxdata.sdk import Client

records = pl.DataFrame(
    {"subject": [0, 1], "path_image": ["sub-0/image.nii.gz", "sub-1/image.nii.gz"]}
)
with Client() as client:
    resource = client.bundles.upload(
        "./input",
        group="example",
        title="example",
        records=records,
        records_meta={"generator": "example"},
        pack_by=["subject"],  # Optional: files read together sit together.
        scratch_dir="./scratch",  # Optional; defaults to VXS3_CACHE_DIR.
        concurrency=8,  # Optional; defaults to 8.
    )
    print(resource.identifier)  # bundle/<uuid4>

upload packs ordinary local files or a directory, uploads and reads back the shards for verification, publishes and verifies the manifest last, and only then registers the resource. Every path_* value must name a source file. pack_by sorts a one-row-per-file view (path, size, path_column and the naming records row) to choose the packing order; without it, files named by records are packed first in records order. Optional parent_identifier, license, access_level and derived_from use the normal resource metadata rules, and registration requires permission to write Bundle resources.

Reading starts from the registered resource, whose manifest_url and manifest_sha256 (and records_url/records_sha256) pin exact stored bytes:

with Client() as client:
    bundle = client.bundles.retrieve("bundle/<uuid4>")
    records = client.bundles.read_records(bundle)  # your table, verified
    manifest = client.bundles.read_manifest(bundle)  # one row per file/directory
    picked = records.filter(pl.col("subject") == 0)
    local = client.bundles.materialize(bundle, "./out", paths=picked)  # path_* -> ./out/...
    files = client.bundles.materialize(bundle, "./out", paths=["sub-1/image.nii.gz"])
    client.bundles.materialize(bundle, "./out")  # everything
    opened = client.bundles.open(bundle)  # vxs3 Bundle: list_entries, open, download, ...

materialize writes files under their bundle paths into dest, reading only their byte ranges; identical existing files are kept and differing ones raise FileExistsError. open returns the vxs3 Bundle for discovery (list_entries, get_member), cached member reads (materialize, open) and whole restores (download). Keep the client open while reading. Bundles registered before parquet manifests, whose manifest_url ends in manifest.json.gz, read the same way.

Streaming Zarr inside a bundle

Zarr stores can be uploaded inside ordinary bundles, including alongside other files. Their bytes, metadata and chunking are preserved without optimisation. Install NumPy/Zarr alongside vxs3 as described in the vxs3 setup guide, then:

import zarr
from vxdata.sdk import Client

with Client() as client:
    bundle = client.bundles.retrieve("bundle/<uuid4>")
    with client.bundles.open_zarr_store(bundle, "images/example.zarr") as store:
        group = zarr.open_group(store=store, mode="r")
        tile = group["0"][0:512, 0:512]

Reads fetch requested encoded values/ranges into memory. They require no local extraction or VXS3_CACHE_DIR; the Client must remain open while consuming the store. The adapter supports Zarr v2/v3 and native sharding through the installed Zarr codecs, and rejects writes. Existing records path_* columns still refer to individual files; pass the store’s bundle-relative directory as prefix.

Optional max_read_bytes (64 MiB), concurrency (8) and cache_bytes (16 MiB) bound each encoded read, active requests and the small-read memory cache. Larger chunks/full-shard reads need a larger explicit read budget; decoded arrays need additional memory. Complete members verify SHA-256. Partial reads validate ranges and pin shard ETags, but cannot verify the whole-member hash. See streaming semantics.

Pass transfer_config=TransferConfig(download_routing="d3n") to open_zarr_store for direct D3N caching, or "auto" to use discovered D3N instances when present. Import TransferConfig from vxdata.sdk. In D3N mode, wire reads cover complete 4 MiB stripes and return only the selected bytes; the default "gateway" retains exact ranges. The SDK reuses HTTP clients and preserves ETags across stripes and URL renewal. Physical overfetch and decoded arrays are additional to max_read_bytes; see the D3N guide.

Writing one bundle from many workers

A data-parallel job, such as an Argo Workflows batch, can have every worker publish its own bundle and still end up with one registered Bundle. Each worker publishes beneath items/<id>/ of a prefix chosen for the run; a finaliser then describes all of them with a single manifest at that prefix and registers it. No shard is read, moved or rewritten in the process, and nothing published is ever replaced: the parts stay valid bundles of their own, and the merged manifest is only metadata pointing at them where they are.

Publication uses vxs3 over direct S3 credentials, since the API’s upload allocation always returns a fresh random directory (see the note at the end of this section). In a worker, after writing results to ./output:

from obstore.store import S3Store
from vxs3.bundle.io import publish_bundle
from vxs3.bundle.local import pack
from vxs3.transport.s3 import S3ExactDestination

store = S3Store(  # credentials from the environment
    "workflow-output",
    endpoint="https://s3.fra.virdx.dev",
    region="eu-central-1",
    virtual_hosted_style_request=False,
    retry_config={"max_retries": 0},
)
run, item = "alex/run-1", "0042"  # the run's prefix and this worker's id

manifest = pack("./output", "/cache/pack")  # local shards + manifest
ref = publish_bundle(manifest, S3ExactDestination(store, f"{run}/items/{item}"))
print(ref.manifest_uri, ref.sha256)  # s3://workflow-output/alex/run-1/items/0042/manifest.parquet

publish_bundle uploads and ETag-verifies the shards, then writes the item’s manifest.parquet last, so its presence marks the item complete. A retry after a partial upload must use a fresh prefix or clean up first: S3ExactDestination refuses a prefix that already holds objects.

When the workers have finished, the finaliser merges the item manifests, writes the run’s root manifest last, and registers it with a records table of its own (here one row per item):

import polars as pl
from vxdata.sdk import Client
from vxs3.bundle.io import open_bundle, publish_manifest
from vxs3.bundle.manifest import merge_manifests
from vxs3.errors import StorageError

items = ["0001", "0002", "0042"]  # the ids the run expected
parts, rows = {}, []
for item in items:
    uri = f"s3://workflow-output/{run}/items/{item}/manifest.parquet"
    try:
        bundle = open_bundle(uri, store=store)  # pin with sha256=... if the worker reported it
    except StorageError:  # no complete item manifest: the worker failed
        rows.append({"item": item, "status": "failed"})
        continue
    parts[f"items/{item}"] = bundle.inventory
    rows.append({"item": item, "status": "succeeded"})

root = publish_manifest(merge_manifests(parts), S3ExactDestination(store, run))

with Client() as client:
    resource = client.bundles.register(
        root.manifest_uri,
        root.sha256,
        title="run-1",
        records=pl.DataFrame(rows),
        group="example",  # where the records table is stored
    )
print(resource.identifier)  # bundle/<uuid4>

merge_manifests prefixes each part’s paths and shard keys with its items/<id> and sums provenance; offsets and hashes are untouched because the shards are. publish_manifest lists the run’s prefix once, requires every referenced shard at its exact size, and then writes manifest.parquet at the root and reads it back. Readers of the result see one tree: items/0042/... lists that worker’s files, and client.bundles.materialize reads only the ranges it needs across items. Items the finaliser leaves out are simply absent from the tree, so record which items were expected, as the records table above does; a root manifest written with some parts missing is a deliberate partial result. The parts’ manifests are not members of the merged bundle. Reading a composed bundle needs vxs3 3.1 or later.

register reads the manifest and checks it against the hash before anything is written, stores records like upload does, and registers with the manifest’s footer (without shards) as manifest_meta. It uploads nothing else.

The API’s upload allocation always returns a fresh random directory, so the SDK cannot yet place parts as siblings or a manifest at their parent; an exact-prefix allocation endpoint is the follow-up that would let client.bundles publish a composed bundle without raw S3 credentials.

Publication and catalog registration are separate operations. If registration or its read-after-write fails, BundleRegistrationError.resource retains the exact BundleCreate request, including the published manifest and records references. When error.rejected is true, the API definitively rejected creation (invalid input or denied access); correct the request or permissions before retrying. Otherwise creation may have succeeded: retrieve error.resource.identifier first, then client.bundles.create(error.resource) only if it is absent. Neither case requires re-uploading. The original error remains in __cause__. Records are stored before the bundle is published, so a failed publication leaves at most a small records object and shards for the caller to reconcile.

scratch_dir and concurrency are optional: omitting scratch_dir uses VXS3_CACHE_DIR, and omitting concurrency uses 8. If neither a scratch path nor a non-blank VXS3_CACHE_DIR is configured, upload raises ValueError before packing or uploading. The scratch location must be outside the input directory, with space for the packed bundle plus any shards read back for verification. Temporary local files are removed on exit. Zarr stores may be included unchanged. bucket is also optional: omit it to use the API’s configured default bucket, or pass a configured bucket name; the manifest reference preserves it for later reads.

Opening a bundle reads its exact manifest without an S3 listing. The SDK obtains presigned URLs through the API; vxs3 reads metadata and byte ranges. Manifest reads are bounded and retain an ETag across interrupted-read retries. One presigned URL per shard serves many range batches and is renewed once if rejected. For downloads, renewal restarts the batch with fresh URLs and all previously learned ETags, so an object cannot silently change across renewal. A second rejection fails. General retries remain in vxs3; the SDK adds no transient-error retry loop. Ordinary storage methods retain their physical-object semantics; client.storage never interprets bundles.

Ephemeral local files

storage.open fetches one object into a temp file and yields it opened; the file is removed when the block exits:

with client.storage.open("s3://bucket/g/model.pt") as f:
    model = torch.load(f)

with client.storage.open("s3://bucket/g/rows.csv", "r") as f:  # UTF-8 text
    for line in f:
        ...

Libraries that want a path can use f.name. Re-entering the context re-downloads — if a loop rereads the same object, download it once instead.

Remote random access

storage.open_remote(url) opens one S3 object as a read-only, seekable binary file without downloading it to disk:

with client.storage.open_remote("s3://bucket/g/volume.tif") as f:
    f.seek(4096)
    header = f.read(128)

The API supplies the exact object’s size with its presigned URL. This needs no prefix listing. Ordinary downloads also request exact sizes for vxs3.

DataFrame materialization

storage.materialize(df, dest_dir) finds s3:// urls in string columns, downloads each unique object once, and rewrites the cells to local paths.

Errors

All SDK exceptions derive from VxDataError: ResourceNotFound, UnexpectedPayload, UnknownPayloadType, TransferError (a blob transfer exhausted its retries), and BundleRegistrationError (publication succeeded but catalog registration failed; rejected distinguishes a definitive create rejection from an unconfirmed outcome). Transport-level HTTP failures surface as httpx errors / RuntimeError with the API’s detail message.

TLS and download transport

API requests and existing uploads use the SDK’s httpx client, which trusts OS certificates or SSL_CERT_FILE/SSL_CERT_DIR. Client(http_client=...) overrides that client; injected clients remain yours to close. API authentication is sent only to the API.

Downloads and bundle metadata use vxs3’s obstore transport. An injected httpx client, Python SSLContext, hooks and mocks do not configure these requests. Vxs3 supplements its backend’s default roots with SSL_CERT_FILE and hashed SSL_CERT_DIR certificates. Explicit options take precedence:

from pathlib import Path
from vxdata.sdk import Client, TransferConfig

client = Client(
    storage_options={
        "root_certificate": Path("./storage-ca.pem").read_bytes(),
        "timeout": "10m",
    }
)
# Per-call options override matching client-wide options.
client.storage.download(
    url,
    "./cache",
    transfer_config=TransferConfig(
        client_options={"read_timeout": "60s"},
    ),
)

Vxs3 defaults are 5 seconds to connect, 300 seconds per read stall, and 300 seconds for the complete request including its body. The request timeout differs from httpx’s phase timeouts. Proxy settings can be supplied through the same options. Use allow_http=True explicitly for local HTTP stores. open_remote retains its separate fsspec transport and TLS behavior.

Module layout

src/vxdata/sdk/
├── client.py             # Client: config resolution, namespace wiring, ping/whoami
├── requests.py           # APIRequestConfig + request(); TLS-trusting default httpx client
├── transport.py          # non-storage API endpoint functions
├── namespaces/           # ergonomic layer: generic + typed CRUD, lineage, storage
│   └── registry.py       #   typed-namespace name -> payload type map
├── s3/                   # internal S3 control plane and transfer engines
│   ├── __init__.py       #   internal subsystem interface
│   ├── control.py        #   control-plane protocol and API implementation
│   ├── bundles.py        #   pack/publish/open convenience wrappers
│   ├── adapters.py       #   vxs3 storage adapters and URL renewal
│   ├── upload.py         #   parallel upload engine (single PUT / multipart)
│   ├── download.py       #   existing-directory/return-path adaptation
│   └── transfer.py       #   shared transfer config
├── query_builder.py      # lazy query builder (filter/sort/limit/select/collect)
├── filters.py            # F / ANY / ALL / has_ancestor filter DSL
├── config.py             # explicit > env (.env) > default value resolution
└── exceptions.py         # VxDataError hierarchy

Development

pixi install -e dev
pixi run test         # pytest (tests/)
pixi run lint         # ruff check
pixi run format       # ruff format
pixi run typecheck    # ty
Navigation

Type to search…

↑↓ navigate↵ selectEsc close