Typed Python client for the VxData API (apps/vxdata-api), stainless-style:
one Client exposing resource namespaces, with request/response types shared
with the server via vxdata-schemas.
Quick start
from vxdata.sdk import Client
vxd = Client() # API_URL and VXDATA_TOKEN from the environment or a .env file
vxd = Client(base_url="http://localhost:8000", token="...") # explicit
vxd.ping() # connectivity check (raises)
me = vxd.whoami() # authenticated principal + grants
patient = vxd.resources.retrieve("patient/bb01") # generic, any payload type
volumes = vxd.volumes.retrieve(["volume/a", "volume/b"]) # typed responsesConfiguration resolves explicit argument > environment (.env included) >
default. The constructor does no I/O. A client-level timestamp= is injected
as as_of into every read/query unless overridden per call. Use the client as
a context manager (or call close()) to release pooled connections.
For file collections, see explicit bundles: client.bundles
packs, publishes and registers a Bundle with an optional records table, and reads
files back selectively by byte range.
The six basic examples cover uploading and downloading
files, bundles and directories.
Namespaces
vxd.resources— generic CRUD over any payload type:retrieve,create,update,delete(soft, cascading),restore,query().- Typed namespaces — one per payload type (
vxd.volumes,vxd.patients,vxd.imaging_studies,vxd.artefacts, …; the full name map lives innamespaces/registry.py). Same verbs with typed create/update/response models. Extras where the platform has them:vxd.patients.register/resolve(external_uid, datasource_id)andvxd.imaging_studies.register/resolve(external_uid, patient_identifier)— server-side identifier allocation.vxd.artefacts.attach_files(...)/vxd.artefacts.download(...).vxd.bundles— catalog CRUD/query for published bundles plusupload,read_manifest,read_records,openandmaterialize; see explicit bundles.
vxd.lineage.parents(...)/vxd.lineage.children(...)— ancestry walks.vxd.storage— blob transfer (below).
All verbs mirror the shape of their input: scalar in yields scalar out, list yields list, dict yields dict.
Queries
query() returns a lazy builder; nothing hits the server until collect:
from vxdata.sdk import ANY, F, has_ancestor
df = (
vxd.pirads_lesion_assessments.query()
.filter(F.score >= 4, has_ancestor("patient/bb01"))
.sort("created_at", descending=True)
.limit(100)
.collect() # polars DataFrame (server-side parquet)
)
models = vxd.volumes.query().filter(...).collect_as_pydantic() # typed models
# Subqueries: pass a query builder to is_in, or compare via ANY/ALL
top = vxd.pirads_assessments.query().filter(F.score == 5).select("score")
worse = vxd.pirads_assessments.query().filter(F.score >= ANY(top)).collect()F.<field> builds filters with comparison operators plus contains,
startswith, endswith, between, before/after, is_null,
is_in (lists, Series, or subqueries), and more; has_ancestor filters by
lineage. head(n) is shorthand for limit(n).collect().
On JSON (dict/list) fields, JSON null counts as unset and ==/!=/is_in
compare JSON values. contains tests list membership, a dict subset, or (with a
string on a dict) text search; overlaps matches any list element. Ordering
and startswith/endswith on JSON fields return a 400.
Blob storage (client.storage)
S3 transfer is presign-only: the SDK asks the API for presigned URLs and moves bytes directly against the object store; it never holds S3 credentials.
Listing objects and folders
listing = client.storage.list_objects("s3://vxdata/source-data/", recursive=False)
print(listing.prefixes) # immediate folders, e.g. ["vxdata/source-data/leopard/"]
for obj in listing.objects:
print(obj.key, obj.size) # immediate objects; sizes in bytesRecursive listing is the default; pass
recursive=False to avoid traversing subfolders. Folder prefixes and object
keys include the bucket but omit s3://. Use a trailing / for folder contents.
Transfers
url = client.storage.upload("weights.pt", group="my-project") # -> s3:// url
local = client.storage.download(url, "./cache") # Path
paths = client.storage.download([url_a, url_b], "./cache") # list[Path]
root = client.storage.download("s3://bucket/g/run/", "./cache") # whole prefixZarr stores can use these ordinary directory transfers to preserve their existing
layout; keep the trailing / on the download URI. This does not optimise or
validate Zarr or create a completion record. See the directory examples
and vxs3 native Zarr workflow.
Downloads request a batch of presigned URLs and exact sizes from the API, then
delegate transfers to vxs3’s DownloadPool. By default, requests go through the
signed gateway URL without D3N discovery or worker processes. To enable direct
D3N workers, set TransferConfig(download_routing="d3n"); see
download routing below.
Vxs3 owns request ranges, concurrency, ETag/hash verification, staging and
transient retries. The SDK renews rejected URLs once, retaining learned ETags
and the pool’s node health state. The SDK preserves scalar/list/dict results and
downloads into existing directories; files are replaced only after the transfer
batch succeeds.
There is no transaction across the final individual file replacements.
Synchronous downloads work inside notebook event loops. Ctrl-C cancels gateway
downloads and waits for cancellation before staging cleanup. When using D3N in scripts, put
downloads and bundle operations inside if __name__ == "__main__": so spawned
workers can import the script safely; the examples show
this structure.
Uploads use the existing HTTP uploader: small files stream as one PUT, with multipart parts above 1 GiB. Both directions accept:
from vxdata.sdk import TransferConfig
client.storage.download(urls, dest, transfer_config=TransferConfig(max_streams=32))max_streams (default 16) caps concurrent transfer workers across a call.
timeout, max_attempts, and backoff now configure uploads only. Downloads use
vxs3’s current retry policy; set download HTTP options through client_options
(or client-wide storage_options, described under TLS below).
Downloads and bundles require vxs3[workers] >=1.2.0, including obstore and
pybreaker. The conda SDK installs these dependencies; Python metadata exposes
storage and bundles extras for internal wheel installations. NumPy and Zarr
are optional vxs3 dependencies. Ordinary catalog imports remain lazy and do not
load the transport.
Download routing
Choose routing per call with TransferConfig.download_routing. All modes use
vxs3/obstore and retain range requests, retries and integrity checks.
download_routing |
Behavior |
|---|---|
"gateway" (default) |
Use the signed gateway URL, skipping discovery and worker processes. |
"d3n" |
Require a configured or mounted D3N inventory; fail if none is available. |
"auto" |
Use discovered D3N instances; use the gateway when no inventory is configured. |
To opt into parallel D3N downloads from a script:
from vxdata.sdk import Client, TransferConfig
def main():
with Client() as client:
client.storage.download(
["s3://bucket/path/a.tif", "s3://bucket/path/b.tif"],
"./downloads",
transfer_config=TransferConfig(download_routing="d3n", max_streams=8),
)
if __name__ == "__main__":
main()Vxs3 discovers instances from VXS3_D3N_INSTANCES, then the mounted
/etc/virdx/d3n/d3n_instances.json. In d3n mode, missing inventory is an error;
the bundled development map is never implicit production membership.
D3N endpoints must be reachable from the client, and routed presigned URLs must use HTTPS.
For local MinIO with HTTP URLs, select gateway (or auto without an inventory)
and set client_options={"allow_http": True}.
An explicitly configured missing or invalid inventory raises an error in
d3n and auto modes; gateway ignores inventory configuration.
The API keeps signing the gateway hostname. Vxs3 preserves the signed Host while connecting to the selected D3N node; no S3 credentials are needed. See vxs3 worker routing for deployment details.
The setting also applies to open, materialize, bundles.upload readback and
shard downloads from bundles.open. Pass transfer_config when opening the
bundle; subsequent streaming reads and downloads reuse it. Bundle.read_range
and open_zarr_store honor routing, using aligned stripes in D3N mode.
Bundle metadata and the separate materialize/extract read_ranges path stay
on the gateway. See D3N cache behavior and request policy.
Each SDK client creates its download pool lazily and reuses its processes and
node health across downloads, bundle readback and bulk shard reads. Batches sharing
the pool run serially. Changing concurrency, routing or transport options replaces
the pool. Use with Client() as client: or call client.close() to join workers;
perform bundle reads while their client remains open.
Streaming has a separate shared transport session with bounded reusable HTTP
clients and its own node health; it does not use the bulk worker processes.
Explicit bundles
The vxs3 README
documents the underlying format and direct S3 interfaces; this section covers
the authenticated vxdata workflow. A Bundle is one vxs3 bundle (tar shards plus
a parquet manifest of files) registered as bundle/<uuid4>, optionally with a
records table: your own rows, of any grain, whose path_* columns name bundle
files. Query the records, then fetch only the files they name.
import polars as pl
from vxdata.sdk import Client
records = pl.DataFrame(
{"subject": [0, 1], "path_image": ["sub-0/image.nii.gz", "sub-1/image.nii.gz"]}
)
with Client() as client:
resource = client.bundles.upload(
"./input",
group="example",
title="example",
records=records,
records_meta={"generator": "example"},
pack_by=["subject"], # Optional: files read together sit together.
scratch_dir="./scratch", # Optional; defaults to VXS3_CACHE_DIR.
concurrency=8, # Optional; defaults to 8.
)
print(resource.identifier) # bundle/<uuid4>upload packs ordinary local files or a directory, uploads and reads back the
shards for verification, publishes and verifies the manifest last, and only then
registers the resource. Every path_* value must name a source file. pack_by
sorts a one-row-per-file view (path, size, path_column and the naming
records row) to choose the packing order; without it, files named by records
are packed first in records order. Optional parent_identifier, license,
access_level and derived_from use the normal resource metadata rules, and
registration requires permission to write Bundle resources.
Reading starts from the registered resource, whose manifest_url and
manifest_sha256 (and records_url/records_sha256) pin exact stored bytes:
with Client() as client:
bundle = client.bundles.retrieve("bundle/<uuid4>")
records = client.bundles.read_records(bundle) # your table, verified
manifest = client.bundles.read_manifest(bundle) # one row per file/directory
picked = records.filter(pl.col("subject") == 0)
local = client.bundles.materialize(bundle, "./out", paths=picked) # path_* -> ./out/...
files = client.bundles.materialize(bundle, "./out", paths=["sub-1/image.nii.gz"])
client.bundles.materialize(bundle, "./out") # everything
opened = client.bundles.open(bundle) # vxs3 Bundle: list_entries, open, download, ...materialize writes files under their bundle paths into dest, reading only
their byte ranges; identical existing files are kept and differing ones raise
FileExistsError. open returns the vxs3 Bundle for discovery
(list_entries, get_member), cached member reads (materialize, open) and
whole restores (download). Keep the client open while reading. Bundles
registered before parquet manifests, whose manifest_url ends in
manifest.json.gz, read the same way.
Streaming Zarr inside a bundle
Zarr stores can be uploaded inside ordinary bundles, including alongside other files. Their bytes, metadata and chunking are preserved without optimisation. Install NumPy/Zarr alongside vxs3 as described in the vxs3 setup guide, then:
import zarr
from vxdata.sdk import Client
with Client() as client:
bundle = client.bundles.retrieve("bundle/<uuid4>")
with client.bundles.open_zarr_store(bundle, "images/example.zarr") as store:
group = zarr.open_group(store=store, mode="r")
tile = group["0"][0:512, 0:512]Reads fetch requested encoded values/ranges into memory. They require no local
extraction or VXS3_CACHE_DIR; the Client must remain open while consuming the
store. The adapter supports Zarr v2/v3 and native sharding through the installed
Zarr codecs, and rejects writes. Existing records path_* columns still refer
to individual files; pass the store’s bundle-relative directory as prefix.
Optional max_read_bytes (64 MiB), concurrency (8) and cache_bytes (16 MiB)
bound each encoded read, active requests and the small-read memory cache.
Larger chunks/full-shard reads need a larger explicit read budget; decoded
arrays need additional memory. Complete members verify SHA-256. Partial reads
validate ranges and pin shard ETags, but cannot verify the whole-member hash.
See streaming semantics.
Pass transfer_config=TransferConfig(download_routing="d3n") to
open_zarr_store for direct D3N caching, or "auto" to use discovered D3N
instances when present. Import TransferConfig from vxdata.sdk. In D3N mode,
wire reads cover complete 4 MiB stripes and return only the selected bytes;
the default "gateway" retains exact ranges. The SDK reuses HTTP clients and
preserves ETags across stripes and URL renewal. Physical overfetch and decoded
arrays are additional to max_read_bytes; see the D3N guide.
Writing one bundle from many workers
A data-parallel job, such as an Argo Workflows batch, can have every worker
publish its own bundle and still end up with one registered Bundle. Each worker
publishes beneath items/<id>/ of a prefix chosen for the run; a finaliser then
describes all of them with a single manifest at that prefix and registers it. No
shard is read, moved or rewritten in the process, and nothing published is ever
replaced: the parts stay valid bundles of their own, and the merged manifest is
only metadata pointing at them where they are.
Publication uses vxs3 over direct S3 credentials, since the API’s upload
allocation always returns a fresh random directory (see the note at the end of
this section). In a worker, after writing results to ./output:
from obstore.store import S3Store
from vxs3.bundle.io import publish_bundle
from vxs3.bundle.local import pack
from vxs3.transport.s3 import S3ExactDestination
store = S3Store( # credentials from the environment
"workflow-output",
endpoint="https://s3.fra.virdx.dev",
region="eu-central-1",
virtual_hosted_style_request=False,
retry_config={"max_retries": 0},
)
run, item = "alex/run-1", "0042" # the run's prefix and this worker's id
manifest = pack("./output", "/cache/pack") # local shards + manifest
ref = publish_bundle(manifest, S3ExactDestination(store, f"{run}/items/{item}"))
print(ref.manifest_uri, ref.sha256) # s3://workflow-output/alex/run-1/items/0042/manifest.parquetpublish_bundle uploads and ETag-verifies the shards, then writes the
item’s manifest.parquet last, so its presence marks the item complete. A retry
after a partial upload must use a fresh prefix or clean up first:
S3ExactDestination refuses a prefix that already holds objects.
When the workers have finished, the finaliser merges the item manifests, writes the run’s root manifest last, and registers it with a records table of its own (here one row per item):
import polars as pl
from vxdata.sdk import Client
from vxs3.bundle.io import open_bundle, publish_manifest
from vxs3.bundle.manifest import merge_manifests
from vxs3.errors import StorageError
items = ["0001", "0002", "0042"] # the ids the run expected
parts, rows = {}, []
for item in items:
uri = f"s3://workflow-output/{run}/items/{item}/manifest.parquet"
try:
bundle = open_bundle(uri, store=store) # pin with sha256=... if the worker reported it
except StorageError: # no complete item manifest: the worker failed
rows.append({"item": item, "status": "failed"})
continue
parts[f"items/{item}"] = bundle.inventory
rows.append({"item": item, "status": "succeeded"})
root = publish_manifest(merge_manifests(parts), S3ExactDestination(store, run))
with Client() as client:
resource = client.bundles.register(
root.manifest_uri,
root.sha256,
title="run-1",
records=pl.DataFrame(rows),
group="example", # where the records table is stored
)
print(resource.identifier) # bundle/<uuid4>merge_manifests prefixes each part’s paths and shard keys with its items/<id>
and sums provenance; offsets and hashes are untouched because the shards are.
publish_manifest lists the run’s prefix once, requires every referenced shard
at its exact size, and then writes manifest.parquet at the root and reads it
back. Readers of the result see one tree: items/0042/... lists that worker’s
files, and client.bundles.materialize reads only the ranges it needs across
items. Items the finaliser leaves out are simply absent from the tree, so record
which items were expected, as the records table above does; a root manifest
written with some parts missing is a deliberate partial result. The parts’
manifests are not members of the merged bundle. Reading a composed bundle needs
vxs3 3.1 or later.
register reads the manifest and checks it against the hash before anything
is written, stores records like upload does, and registers with the manifest’s
footer (without shards) as manifest_meta. It uploads nothing else.
The API’s upload allocation always returns a fresh random directory, so the SDK
cannot yet place parts as siblings or a manifest at their parent; an
exact-prefix allocation endpoint is the follow-up that would let client.bundles
publish a composed bundle without raw S3 credentials.
Publication and catalog registration are separate operations. If registration or
its read-after-write fails, BundleRegistrationError.resource retains the exact
BundleCreate request, including the published manifest and records references.
When error.rejected is true, the API definitively rejected creation (invalid
input or denied access); correct the request or permissions before retrying.
Otherwise creation may have succeeded: retrieve error.resource.identifier
first, then client.bundles.create(error.resource) only if it is absent.
Neither case requires re-uploading. The original error remains in __cause__.
Records are stored before the bundle is published, so a failed publication
leaves at most a small records object and shards for the caller to reconcile.
scratch_dir and concurrency are optional: omitting scratch_dir uses
VXS3_CACHE_DIR, and omitting concurrency uses 8. If neither a scratch path
nor a non-blank VXS3_CACHE_DIR is configured, upload raises ValueError before
packing or uploading. The scratch location must be outside the input directory,
with space for the packed bundle plus any shards read back for verification. Temporary local files
are removed on exit. Zarr stores may be included unchanged. bucket is also
optional: omit it to use the API’s configured default bucket, or pass a
configured bucket name; the manifest reference preserves it for later reads.
Opening a bundle reads its exact manifest without an S3 listing. The SDK obtains
presigned URLs through the API; vxs3 reads metadata and byte ranges. Manifest
reads are bounded and retain an ETag across interrupted-read retries. One
presigned URL per shard serves many range batches and is renewed once if
rejected. For downloads, renewal restarts the batch with fresh URLs and all
previously learned ETags, so an object cannot silently change across renewal. A
second rejection fails. General retries remain in vxs3; the SDK adds no
transient-error retry loop. Ordinary storage methods retain their
physical-object semantics; client.storage never interprets bundles.
Ephemeral local files
storage.open fetches one object into a temp file and yields it opened; the
file is removed when the block exits:
with client.storage.open("s3://bucket/g/model.pt") as f:
model = torch.load(f)
with client.storage.open("s3://bucket/g/rows.csv", "r") as f: # UTF-8 text
for line in f:
...Libraries that want a path can use f.name. Re-entering the context
re-downloads — if a loop rereads the same object, download it once instead.
Remote random access
storage.open_remote(url) opens one S3 object as a read-only, seekable binary
file without downloading it to disk:
with client.storage.open_remote("s3://bucket/g/volume.tif") as f:
f.seek(4096)
header = f.read(128)The API supplies the exact object’s size with its presigned URL. This needs no prefix listing. Ordinary downloads also request exact sizes for vxs3.
DataFrame materialization
storage.materialize(df, dest_dir) finds s3:// urls in string columns,
downloads each unique object once, and rewrites the cells to local paths.
Errors
All SDK exceptions derive from VxDataError: ResourceNotFound,
UnexpectedPayload, UnknownPayloadType, TransferError (a blob transfer
exhausted its retries), and BundleRegistrationError (publication succeeded but
catalog registration failed; rejected distinguishes a definitive create
rejection from an unconfirmed outcome). Transport-level HTTP failures surface as
httpx errors / RuntimeError with the API’s detail message.
TLS and download transport
API requests and existing uploads use the SDK’s httpx client, which trusts OS
certificates or SSL_CERT_FILE/SSL_CERT_DIR. Client(http_client=...) overrides
that client; injected clients remain yours to close. API authentication is sent
only to the API.
Downloads and bundle metadata use vxs3’s obstore transport. An injected httpx
client, Python SSLContext, hooks and mocks do not configure these requests.
Vxs3 supplements its backend’s default roots with SSL_CERT_FILE and hashed
SSL_CERT_DIR certificates. Explicit options take precedence:
from pathlib import Path
from vxdata.sdk import Client, TransferConfig
client = Client(
storage_options={
"root_certificate": Path("./storage-ca.pem").read_bytes(),
"timeout": "10m",
}
)
# Per-call options override matching client-wide options.
client.storage.download(
url,
"./cache",
transfer_config=TransferConfig(
client_options={"read_timeout": "60s"},
),
)Vxs3 defaults are 5 seconds to connect, 300 seconds per read stall, and 300 seconds
for the complete request including its body. The request timeout differs from
httpx’s phase timeouts. Proxy settings can be supplied through the same options.
Use allow_http=True explicitly for local HTTP stores. open_remote retains its
separate fsspec transport and TLS behavior.
Module layout
src/vxdata/sdk/
├── client.py # Client: config resolution, namespace wiring, ping/whoami
├── requests.py # APIRequestConfig + request(); TLS-trusting default httpx client
├── transport.py # non-storage API endpoint functions
├── namespaces/ # ergonomic layer: generic + typed CRUD, lineage, storage
│ └── registry.py # typed-namespace name -> payload type map
├── s3/ # internal S3 control plane and transfer engines
│ ├── __init__.py # internal subsystem interface
│ ├── control.py # control-plane protocol and API implementation
│ ├── bundles.py # pack/publish/open convenience wrappers
│ ├── adapters.py # vxs3 storage adapters and URL renewal
│ ├── upload.py # parallel upload engine (single PUT / multipart)
│ ├── download.py # existing-directory/return-path adaptation
│ └── transfer.py # shared transfer config
├── query_builder.py # lazy query builder (filter/sort/limit/select/collect)
├── filters.py # F / ANY / ALL / has_ancestor filter DSL
├── config.py # explicit > env (.env) > default value resolution
└── exceptions.py # VxDataError hierarchyDevelopment
pixi install -e dev
pixi run test # pytest (tests/)
pixi run lint # ruff check
pixi run format # ruff format
pixi run typecheck # ty