How data should be laid out and moved on this cluster, and the tools that do it.
inspect_data applies the vxs3 bulk_nvme_v1 policy; this page is the human
readable version. Facts first, recommendations second.
For the public Python interfaces, read read_doc(“vx/cluster/vxs3”) (direct S3 and native or bundled Zarr),
read_doc(“vx/cluster/vxs3-storage”) (verification, caching and transfer details), and
read_doc(“vx/cluster/vxdata-sdk”) (authenticated transfers and Bundle resources).
read_doc(“vx/cluster/vxdata-api”) covers the bundle allocation and catalog contract. These are
vendored from the merged source; check the installed SDK and deployed API support
the interfaces before changing a job. Downloads and bundles in the new SDK need
vxs3[workers] >=1.2.0; Bundle registration also needs the API’s bundle
migration and write permission. A source merge alone does not establish deployment.
The single most effective change for most jobs: store and read fewer, larger objects. Bundle ordinary files into tar shards; for Zarr, review native sharding or immutable bundles with a compatible reader before a job depends on them. See Fewer, larger objects.
Storage choices
| Tier | Path | Good for | Bad for |
|---|---|---|---|
| Node-local NVMe | /cache ($VXRUN_CACHE_DIR, request with --local-cache 64Gi) |
hot training data, framework caches, scratch | anything that must survive the Pod |
| Object storage (Ceph RGW) | s3://vxdata/... via vxdata-sdk and vxs3 |
datasets, model weights, published results | per-step tiny writes |
| Scratch bucket (Ceph RGW) | s3://scratch/<username>/<experiment-or-workflow>/ via the [scratch] profile |
development experiments, intermediate data, temporary checkpoints | permanent data or repeatable workflow inputs and outputs; objects are garbage-collected after two weeks (14 days) |
| NFS (legacy) | /mnt/storage (read-only in jobs), /mnt/artifacts (run-image /output, read-write) |
existing inputs not yet published to vxdata; run-image outputs you have not moved yet |
anything new; random reads in a training loop; frequent large checkpoint rewrites. Not mounted in run-batch |
Pattern for a job: download once from object storage into /cache, train from
/cache, checkpoint to the scratch bucket (or /output when checkpoints are
small and rare), then publish data and results you want to keep as permanent
vxdata resources. In a run-batch item, /checkpoints ($VXRUN_CHECKPOINTS_DIR)
is the node-local directory shared by that item’s retries: cheap for frequent
saves, but gone with the item’s claim, so it does not cover a resubmitted batch
or a lost node. Keep the scratch bucket for those.
NFS is legacy; run-batch is the S3-native path
NFS (/mnt/storage, /mnt/artifacts) is the legacy tier. It stays mounted in
run-image jobs and Coder workspaces for existing data, but nothing new should
depend on it: exports can be full or read-only, Coder startup no longer writes
to them, and the platform direction is to remove them. Treat every NFS path a
job reads or writes as something to migrate, not a design choice.
run-batch is the replacement and has no NFS or hostPath mounts:
- Inputs come from S3 (vxdata resources or the scratch bucket) and are staged
once into
/cache, node-local NVMe. - Each item writes
/outputon a temporary per-item TopoLVM claim on the node’s NVMe (16Gi withlocal_cache: "0", otherwise thelocal_cachesize). - The
packstep publishes that output tos3://workflow-output/...as a vxs3 bundle, the exit handler completes the batch, andvxrun batch resultscollects it. S3 is the only durable copy; the claim is deleted after the item.
This is faster as well as cleaner. Reads and writes hit local NVMe instead of a shared network filesystem that every job on the cluster contends for, so the GPU is not waiting on file-system latency. Publication uploads a few large shards in parallel rather than leaving thousands of small files on an export, and the result is immutable, addressable by prefix and discoverable through the catalog instead of a mutable directory someone has to remember.
What this means in practice:
- New batch or sweep work: use
vxrun batch; outputs publish themselves. - A single
run-imagejob: read from S3 into/cache, checkpoint to scratch, and publish results withvxrun storage uploadrather than leaving them on/mnt/artifacts. - Existing inputs on
/mnt/storage: bundle them once withvxrun storage uploadand point jobs at the vxdata resource. - Existing outputs on
/mnt/artifacts: publish what you want to keep the same way.
Where the S3 credentials are
You never need to ask for S3 keys. Every Coder workspace and every run-image
or run-batch Pod carries the shared credentials file; AWS_SHARED_CREDENTIALS_FILE
points at it (fall back to ~/.aws/credentials). The same profiles exist in
Coder workspaces and in run-image/run-batch Pods: [scratch] reads and
writes the scratch bucket; [workflow-output] reads the workflow-output
bucket (the RGW user coder-workspace, read-only; the pack and finalise steps
hold a separate writer file); [default] is the same identity as
[workflow-output], so a client with no profile set can read published batch
results. Each profile carries
endpoint_url = https://s3.fra.virdx.dev, so boto3.Session(profile_name=...),
s3fs and AWS_PROFILE=... aws s3 ... work unchanged. [default] is no longer
real-AWS credentials; older comments that say so are stale. s3://vxdata goes through the vxdata SDK with VXDATA_TOKEN, not S3
keys. So when a job or a migration from NFS needs S3 access, look in the
environment first; the only thing to copy is a profile name, never a key.
Fewer, larger objects: bundle files, reshard Zarr
Every object costs the same fixed work whether it holds 4 KiB or 1 GiB: a catalog or listing entry, a presigned URL, at least one HTTP request to Ceph RGW, a verification read and a retry slot. Object storage and the vxdata catalog scale with object count, not bytes; the GPU waits on request latency, not bandwidth. A dataset of 200,000 PNGs is 200,000 round trips per epoch read directly, and 200,000 entries for every listing, grant check and GC pass. The same bytes in 1 GiB tar shards are a few hundred objects, four range requests each.
What the tooling does about it:
- Bundling (ordinary files): vxs3 packs a directory into uncompressed tar shards targeting 1 GiB plus a parquet manifest. Whole restores fetch shards on the 256 MiB range grid, so throughput is bounded by concurrency rather than per-file latency. D3N caches the underlying tail stripes, typically 4 MiB each; one tar shard can occupy many cache entries. Selective materialization fetches the chosen files’ byte ranges, merged per shard. The manifest answers “what is in here” and “where is file X” without listing the bucket or touching a shard.
- Resharding (Zarr): a store with small chunks is one object per chunk and
one request per chunk.
vxrun storage optimiserewrites it as a Zarr v3 copy whose shards target 1 GiB while preserving the inner chunk shape, values and codecs, so readers keep their access pattern but issue far fewer requests. - Bundled Zarr: packing preserves existing encoded files and metadata.
vxs3.zarr.store.BundleStoreorclient.bundles.open_zarr_store(...)translates Zarr reads, including native shard ranges, into bundle range requests without extracting files. Readers must support this adapter; ordinary Zarr S3 readers cannot open the tar directly. Bundling does not validate metadata or optimise chunks. The bundle stays immutable and the store is read-only. With SDKdownload_routing="d3n", streaming fetches complete 4 MiB Ceph stripes through Ketama and slices the requested bytes. Gateway mode retains exact ranges. Read the D3N guide linked fromread_doc("vx/cluster/vxs3-storage")before assuming partial reads reuse warmed server caches or setting memory budgets. - Keep a few large files as they are: a 2 GiB weights file is already one object, and bundling it buys nothing.
inspect_data and vxrun data inspect apply exactly this policy: objects
below 4 MiB count as small, any selection with more than one payload object
and a small one can get bundle. Zarr metadata candidates get
review_zarr_storage; native layout findings may instead recommend
investigate_zarr_sharding. Review metadata, reader compatibility and access
patterns before choosing a layout. Packing eligibility does not establish that
bundling is the right choice, and object counts do not establish a speedup.
| Data | Temporary (scratch) | Permanent (vxdata) |
|---|---|---|
| Directory of ordinary files | vxs3.bundle.local.pack + publish_bundle to scratch |
client.bundles.upload(...) (SDK, registers a Bundle) or vxrun storage upload DIR (worker, always bundles) |
| Zarr store | optimize_local_zarr before upload |
vxrun storage optimise s3://.../store.zarr/ writes the resharded copy to vxdata-optimised |
| Zarr with BundleStore consumers | pack unchanged files + publish_bundle to scratch |
client.bundles.upload(...); stream with client.bundles.open_zarr_store(...) |
| A few large files | keep | keep; vxrun storage upload accepts a standalone file of at least 100 MiB |
Two consequences to design around. Reading members of a bundle fetches their
byte ranges merged per shard, so files read together are cheapest when they
sit side by side. Packing walks paths in sorted order, so lay the source out so
such files share a directory or name prefix (by split, class or sample id), or
pass pack_by= (or a records table) to client.bundles.upload to set the order.
And resharded Zarr needs a Zarr v3 capable reader and a grant on
vxdata-optimised; check both before pointing a job at the optimised copy.
Scratch is for development
Use scratch while developing and experimenting with data, models and results.
Its two-week (14-day) garbage-collection policy makes it temporary storage.
Do not use scratch as the permanent source for a dataset, model or repeatable
workflow. Checkpoint/resume can use it within that retention window; restarting
a job does not exempt its objects from GC.
Once the developer is happy with the data, publish it to vxdata and create a
permanent resource. Future workflows should use that resource. The retained
bytes must also be in vxdata storage: registering a resource that points to a
scratch URL does not preserve the data. See Publishing data
below; use --retention permanent for the permanent resource and finish the
upload before the scratch objects expire.
Access scratch through the [scratch] profile, not a filesystem mount.
Coder workspaces and run-image/run-batch jobs always mount the shared
credentials file and set AWS_SHARED_CREDENTIALS_FILE (see “Where the S3
credentials are” above). Its
[scratch] profile carries endpoint_url = https://s3.fra.virdx.dev, so
boto3.Session(profile_name="scratch"), s3fs.S3FileSystem(profile="scratch")
or AWS_PROFILE=scratch aws s3 ... work unchanged. For a boto3 connection check
with the internal CA, see read_doc(“vx/cluster/submitting-jobs”), “Scratch storage for development”.
The identity is shared, so the <username>/<experiment-or-workflow>/ prefix is
a convention, not an access boundary. The bucket has a shared size quota;
delete temporary data you no longer need. A new submission has a new workflow
name, so resuming an earlier run means pointing the job at the old prefix
(or the old /output directory), while its data is still available.
Review scratch layout and request frequency
Before uploading, call inspect_data on the local source with scratch=true
(and repo_path if the source is under the repository), or run:
vxrun data inspect ./data --scratchThis mode gives temporary-storage advice instead of suggesting immediate
publication. vxrun review also emits an advisory finding for scratch URLs,
profiles and bucket references. Neither check blocks a job. A few large files
or occasional checkpoints can be fine; look at object counts, sizes and access
patterns before changing the layout.
For many small records, consolidate into a suitable native format such as Parquet if the readers support it. For independent ordinary files, pack locally into vxs3 tar shards before upload:
from vxs3.bundle.local import pack
pack("./data", "./data-bundle") # destination must be new; target 1 GiB shardsPass workers=8 to write shards in worker processes, or set VXS3_WORKERS=8
for every pack and extract in the job (default 0); scripts that use
workers need an if __name__ == "__main__": guard.
Publish the packed bundle with vxs3.bundle.io.publish_bundle and an
S3Destination configured for scratch. It verifies each shard against the store’s ETag as it uploads
(reading back only what the store cannot attest) and publishes the manifest last.
vxs3.bundle.io.upload_bundle packs and publishes in one pass, uploading shards
as they are written and holding only shards in flight in scratch. Keep the returned manifest URI
and hash; use open_bundle(ref, store=store) to restore once onto /cache or
materialize selected members into a shared local cache. See read_doc(“vx/cluster/vxs3”) for the
complete direct-S3 example. All objects in this bundle still expire under the
scratch policy; the reference does not extend retention. Keep restored data
on local NVMe across epochs and workers when using materialisation. For Zarr,
review chunking/sharding, metadata and reader compatibility: native stores and
unchanged stores in immutable bundles are both options. BundleStore can stream
selected chunks without disk staging. Batch small writes and avoid repeated listings or each worker
independently downloading the same inputs. Bound aggregate transfer concurrency.
Prefer the local source: the analyzer itself should not add unnecessary Ceph
load. Scratch mode defaults to 10,000 objects, 32 sources, 64 metadata probes,
4 MiB of metadata and a 30-second cooperative time budget. LIST/HEAD requests
are outside the probe budget and in-flight requests can overrun the deadline.
Whole-bucket scans are refused; select only the relevant prefix or object.
Remote analysis uses environment-based S3 credentials and endpoint, not the
boto3 session shown above; AWS_PROFILE=scratch alone is insufficient.
Report partial/failed coverage explicitly: it does not establish that the remaining data is well laid out. Object shape and estimated transfer requests are not measured IOPS. Review GET/LIST/PUT frequency in the job as well.
How transfers actually work
- vxdata-sdk (
client.storage.download) obtains presigned URLs and exact object sizes from the vxdata API, then delegates downloads to vxs3’s 256 MiB range grid, shared concurrency budget, verification and retry logic. It never holds S3 keys.TransferConfig(max_streams=...)defaults to 16; choose concurrency across the whole job, including worker processes. The SDK renews rejected URLs once while preserving learned ETags. Existing uploads use the SDK’s HTTP uploader. Download TLS/timeouts usestorage_optionsorTransferConfig(client_options=...), independently of the API’s httpx client.TransferConfig(download_routing=...)picks the path:"gateway"(the SDK default) uses the signed gateway URL with no worker processes;"d3n"requires the D3N inventory (VXS3_D3N_INSTANCES, or the/etc/virdx/d3n/d3n_instances.jsonthat Kyverno mounts into workflow Pods and Coder workspaces) and fans ranges out to D3N nodes in worker processes;"auto"uses D3N when an inventory is configured. Note the defaults differ: vxs3’s ownDownloadPooldefaults torouting="d3n". The same setting applies toopen_bundle,materializeandopen; pass it when opening the bundle. In scripts, keep D3N downloads underif __name__ == "__main__":so spawned workers can import the module. - vxs3 packs a directory into uncompressed tar shards targeting 1 GiB
(4 GiB ceiling, 100,000 entries per shard) plus a Parquet manifest that
records each file’s byte range. A 1 GiB shard needs 4 payload range requests,
before metadata, retries or verification reads.
vxs3 analyzecalls objects below 4 MiB “small” and can suggestbundlewhen an ordinary-file selection has more than one payload object and any of them is small. - Native Zarr optimisation:
vxrun storage optimise s3://.../store.zarr/writes a separate, resharded Zarr v3 copy at the same path in thevxdata-optimisedbucket. The source is untouched; readers need a grant on that bucket and must be Zarr v3 capable. - Bundled Zarr streaming uses exact member-relative ranges through
BundleStore, with a read-size limit, concurrency limit and bounded memory cache. Full member reads verify SHA-256; partial reads pin a strong shard ETag for version consistency without verifying the whole-member hash. It does not use the bulk downloader’s 256 MiB grid or require a disk cache.
bundle.list_entries() and get_member() use the manifest without shard I/O.
materialize() and open() fetch and verify complete selected members,
merging their byte ranges per shard, then reuse an immutable local cache.
Group files by access pattern and share the cache across
workers on the same node. Copy a cached file before editing it. There is no
automatic cache eviction. bundle.download() restores all files into a new
destination separately from the member cache. Allow disk for staging and output;
publication also reads uploaded bytes back for verification.
Ordinary SDK directory transfers preserve physical object layout. They do not automatically unpack bundles, reshard Zarr or produce native Zarr completion records. Use the explicit bundle or native Zarr interfaces for those operations.
Publishing a Bundle from local files
Use the SDK for ordinary local files or directories when a durable, discoverable bundle resource is wanted. Keep the source immutable during publication, and put local scratch outside it on a volume with room for packed data and verification:
import polars as pl
from vxdata.sdk import Client
records = pl.DataFrame({"subject": [0, 1], "path_image": ["sub-0/t2w.nii.gz", "sub-1/t2w.nii.gz"]})
with Client() as client:
resource = client.bundles.upload(
"./data",
group="example",
title="example",
records=records, # optional: your own index of the files
pack_by=["subject"], # optional: files read together sit together
bucket="vxdata",
scratch_dir="/cache/publish",
concurrency=8,
)
print(resource.identifier) # bundle/<uuid4>This stores the records table, publishes verified bytes to permanent storage,
then registers a Bundle pinning the exact manifest and records by URL and
SHA-256; the identifier is allocated for you. Here scratch_dir is temporary
local disk, not the S3 scratch bucket. This API has no --retention flag;
do not select the expiring scratch bucket for retained inputs. For a remote
scratch prefix, use the worker upload path below or stage it locally before SDK
publication.
If registration fails after publication, BundleRegistrationError.resource
retains the create request and reference. With rejected=True, fix the rejected
request or permissions. Otherwise retrieve the identifier first to reconcile an
uncertain write, and create the resource only if absent. Do not upload again.
See read_doc(“vx/cluster/vxdata-sdk”) for the full recovery and authorization contract. The API
stores the publisher’s reference; registration itself does not verify shards.
Batch outputs (run-batch publication)
A run-batch Workflow publishes by default (publish: "true"): after each item’s
command, a pack step turns that attempt’s /output into a vxs3 bundle at
s3://workflow-output/<user>/<workflow>-<uid>/items/<id>/, and the exit handler
writes workflow.json (name, uid, user, status, expected item ids), then
items.parquet (one row per expected item: item, status succeeded|failed,
manifest_uri, manifest_sha256, files, bytes, detail), and then, only
when every item succeeded, the root manifest.parquet: the completion marker
for the whole batch, like a multipart upload that becomes an object only on
completion. The root is one composed bundle: items/<id>/... lists an item’s
files, shards stay where the items published them, and each item’s own manifest
remains a complete bundle. Reading it needs vxs3 3.1 or later. (Finalisers
before vxrun 0.4.7 also wrote a root for partial batches; items.parquet tells
them apart.)
Advice that follows from this:
- Write results under
/output(VXRUN_OUTPUT_DIR), not scratch, when they should outlive the Pod; the pack step publishes exactly that directory. - Each item’s
/outputis a temporary TopoLVM claim on the node (16Gi withlocal_cache: "0", otherwise thelocal_cachesize); there are no NFS mounts, so read inputs from S3/vxdata into/cache. The pack step mounts the claim read-only and assembles the shards in its own/cachebefore upload, solocal_cachemust cover an item’s outputs and the packing space. - A failed item (exhausted retries, or a pack retry after a partial upload) has
no manifest and appears in
items.parquetas failed; the batch then gets no root manifest and no catalog entry. The successful items’ bundles remain and can be fetched one at a time (vxrun batch results --item); the fix is to re-run, which is a new batch under a new prefix. - Terminate skips the exit handler (no
items.parquet, no root); Stop runs it and records which items finished. register: "true"also registers the root as a vxdata Bundle withitems.parquetas its records table, for discovery through the catalog.- Published outputs are immutable from the user’s point of view. The writer
identity (RGW user
workflow-writer: put, get, list, no delete) lives in theworkflow-output-writersecret, mounted only by the controller-createdpackandfinalisepods, which run a vxrun-tools image pinned by digest inside the template; there is notools-imageparameter. The user’smaincontainer getsaws-credentials, whose[workflow-output]profile is read-only on the bucket (RGW usercoder-workspace, the same keys as[default]). Kyverno policyargo-restrict-workflow-output-writerrejects submissions referencing the writer secret, run-batch submissions that addspec.templatesorspec.podSpecPatch, other pods mounting the secret, and in-cluster edits of ArgoCD-managed WorkflowTemplates. Known limits: the writer can overwrite existing keys (software-guarded; bucket versioning would close it), and an item’s output is mutable on its temporary claim until packed. Published output is retained; there is no lifecycle expiry on workflow-output. The claim is deleted after each item (also withpublish: "false"), so S3 is the only durable copy.
Collecting results
vxrun batch results <workflow-name> # prefix, completion, one line per item
vxrun batch results <workflow-name> --json
vxrun batch results <workflow-name> --download ./out # complete batch: ./out/items/<id>/...
vxrun batch results <workflow-name> --download ./out --item trial-0 # one item, works when incomplete
vxrun batch results s3://workflow-output/<user>/<name>-<uid> # by prefixresults derives the prefix from the live Workflow; once the Workflow is
garbage-collected (three days after it finishes) it finds <user>/<name>-<uid>/
in the bucket and reads workflow.json, so results outlive the Workflow. Before
the exit handler has run it reports items as published/pending; after,
the items.parquet rows; complete is the root manifest’s presence (its
SHA-256 is reported and pinned on download). It reads the bucket with the
[workflow-output] profile from AWS_SHARED_CREDENTIALS_FILE or
~/.aws/credentials; Coder workspaces and job containers both carry it
(--profile overrides). Decide from the per-item status before using a batch;
a whole-batch download is refused when the root manifest is absent.
In Python, take prefix or an item’s manifest_uri from --json and open it
with vxs3; vxrun.artifacts.profile_store builds the store from the same
profile:
from vxrun.artifacts import profile_store
from vxs3.bundle.io import open_bundle
store = profile_store("default", "workflow-output", path="~/.aws/credentials") # profile, bucket
root = open_bundle(
f"{prefix}/manifest.parquet", store=store, sha256=sha256
) # complete batches only
root.list_entries("items/trial-0/") # one item's files, no shard I/O
root.materialize(["items/trial-0/metrics.json"]) # {path: cached local file}, byte range only
item = open_bundle(f"{prefix}/items/trial-0/manifest.parquet", store=store) # any published item
item.extract("./out/items/trial-0")When the batch ran with register: "true" and completed, go through the
catalog instead; the finalise pod log prints the Bundle identifier and the title
is the Workflow name:
from vxdata.sdk import Client, F
with Client() as client:
found = client.bundles.query().filter(F.title == "<workflow-name>").collect()
bundle = client.bundles.retrieve(found["identifier"][0]) # or the logged bundle/<uuid4>
items = client.bundles.read_records(bundle) # items.parquet, hash-verified
client.bundles.materialize(bundle, "./out", paths=["items/trial-0/metrics.json"])The same composition works outside run-batch: workers publish parts with
publish_bundle beneath sibling prefixes and a finaliser merges them with
merge_manifests and publish_manifest, then registers with
client.bundles.register. See “Writing one bundle from many workers” in
read_doc(“vx/cluster/vxdata-sdk”).
Publishing data (vxrun storage upload)
vxrun storage upload /mnt/storage/<path> --name <dataset> --project <project> \
--description "<what it is>" --data-role reference --retention review-180d --waitTo keep a finished development dataset from scratch, stop modifying its prefix and publish it as a permanent resource:
vxrun storage upload s3://scratch/<username>/<experiment>/ \
--name <dataset> --project <project> --description "<what it is>" \
--data-role derived --retention permanent --waitAfter the upload succeeds, use the returned vxdata resource and data URI in repeatable workflows. Publication creates a separate stored copy; the scratch source remains subject to GC. S3 uploads require the backend capability documented in read_doc(“vx/cluster/submitting-jobs”), “Vxoptimise service workflows”.
- Source: a path under
/mnt/storageor/mnt/artifacts, or ans3://bucket/prefix/(a directory prefix, never a bucket or single object). - Accepted shapes: a valid root Zarr, a standalone file of at least 100 MiB, or a directory, which is always bundled. Small standalone files are rejected.
- Limits per upload: 256 GiB and 100,000 entries. The source must not change during the run; the worker re-lists and refuses if it did.
--data-role: source, derived, reference, model, results.--retention: temporary-30d, review-180d, permanent.--projectis a lowercase slug. Also--metadata file.json,--dry-run,--timeout.- Result: an immutable
s3://vxdata/storage/<prefix>/and a catalog recordvxupload-<workflow-uid>that turnsreadyonly after read-back verification.vxrun storage status [NAME] [--json]shows your jobs with progress and, on success, the data URI; rows disappear with the Workflow TTL (2 days). Durable lookup:Client().generic_files.retrieve("vxupload-<uid>"), usefile_metadata.data_urionly whenstate == "ready".
Reading data in a job
For a registered Bundle, keep the client open through bundle reads:
import os
from pathlib import Path
import polars as pl
from vxdata.sdk import Client
cache = Path(os.environ.get("VXRUN_CACHE_DIR", "/cache"))
with Client() as client:
bundle = client.bundles.retrieve("bundle/<uuid4>")
client.bundles.materialize(bundle, cache / "data") # every file, by byte range
# Or only what a records query names: path_* columns come back pointing into dest.
picked = client.bundles.read_records(bundle).filter(pl.col("subject") == 0)
local = client.bundles.materialize(bundle, cache / "data", paths=picked)The worker upload path above still returns a GenericFile with file_metadata
and a physical data prefix. It does not produce a Bundle resource. Wait for
its ready state, then use its reported URI. For an ordinary-file worker bundle,
the existing physical-download/unpack path remains valid:
import os
from pathlib import Path
from vxdata.sdk import Client, TransferConfig
from vxs3.bundle.local import unpack
cache = Path(os.environ.get("VXRUN_CACHE_DIR", "/cache"))
with Client() as client: # VXDATA_TOKEN is injected by run-image
client.storage.download(
"s3://vxdata/storage/<prefix>/",
cache / "bundle",
transfer_config=TransferConfig(download_routing="d3n", max_streams=8),
)
data = unpack(cache / "bundle" / "manifest.parquet", cache / "data") # destination must be newdownload_routing="d3n" fails fast when no D3N inventory is configured; use
"auto" for code that also runs on a laptop, or omit it for the gateway. Keep
the call under a main() guard in scripts so D3N workers can spawn.
Do not infer a Bundle reference from shard names or assume that every worker
upload is a tar bundle: a native Zarr upload stays Zarr. Use open_bundle with an
explicit manifest reference when one is available (older bundles with a
manifest.json.gz still open).
Credentials: run-image mounts the optional secret
vxdata-credentials-<run_environment> as VXDATA_TOKEN; no S3 keys are
needed for s3://vxdata. The scratch bucket ([scratch]) and read-only
workflow-output access ([workflow-output], the same keys as [default])
come from
AWS_SHARED_CREDENTIALS_FILE, which is always mounted. When local_cache is set, run-image also exports
VXS3_CACHE_DIR=/cache/.vxs3, so vxs3 calls that omit a destination land on
the node-local volume without further configuration.
Inspecting data (vxrun data inspect, vxs3 analyze)
vxrun data inspect SOURCE [--table] [--scratch] [--json] runs a bounded metadata scan
(no payload downloads) and reports: physical and logical counts and bytes,
p50/p95 object sizes, the share below 4 MiB, estimated range requests, detected
formats with an evidence level (metadata_verified, header_verified,
extension_only, unknown), findings with codes such as bulk.layout and
bundle.zarr_reader_required, and one suggested action: keep, bundle,
investigate_zarr_sharding, review_zarr_storage or unsupported, plus a
command or review advice. Zarr storage review does not automatically upload,
convert or optimise the store.
coverage is complete, partial (a limit such as max_objects was hit;
counts are not extrapolated) or failed. The underlying CLI vxs3 analyze
exits 0, 2 and 1 respectively.
What to say to a user
- Developing or experimenting: “use scratch for temporary data; it is garbage-collected after two weeks. Once you are happy with it, publish a permanent vxdata resource for future or repeatable workflows.”
- Many small scratch files: “inspect locally with
--scratch, then consolidate records or pack ordinary files into 1 GiB tar shards before upload. Every object is a request; a few hundred shards beat a few hundred thousand files. Stage once into/cache; check repeated reads, writes and listings across workers too.” - Many small files on NFS: “bundle it once with
vxrun storage upload, then stage into/cacheat job start; the GPU stops waiting on the file system, and every later job reuses the bundled resource.” - A job that writes results to
/mnt/artifacts, or a new layout on NFS: “NFS is legacy. Run it as avxrun batch(outputs publish to S3 on their own, no NFS involved, faster on node-local NVMe), or write to scratch and publish a vxdata resource. Nothing new should depend on/mnt/*.” - A Zarr store with tiny chunks: “
vxrun storage optimisewrites a resharded v3 copy in vxdata-optimised with 1 GiB shards and the same inner chunks; point the job at that copy and it reads chunks with a fraction of the requests.” - A few large files already in object storage: “keep; download once to
/cache.” - Results of a run-batch: “
vxrun batch results <workflow>shows the prefix, per-item status and whether the batch completed;--downloadfetches a complete batch,--itemone item even when it did not (see Collecting results). Passregister: "true"next time if it should appear in the catalog.” - Analyzer unavailable (
[data]error): show the discovered sources and this page; do not guess sizes.