Skip to content
VIRDX

Storage practices

How data should be laid out and moved on this cluster, and the tools that do it. inspect_data applies the vxs3 bulk_nvme_v1 policy; this page is the human readable version. Facts first, recommendations second.

For the public Python interfaces, read read_doc(“vx/cluster/vxs3”) (direct S3 and native or bundled Zarr), read_doc(“vx/cluster/vxs3-storage”) (verification, caching and transfer details), and read_doc(“vx/cluster/vxdata-sdk”) (authenticated transfers and Bundle resources). read_doc(“vx/cluster/vxdata-api”) covers the bundle allocation and catalog contract. These are vendored from the merged source; check the installed SDK and deployed API support the interfaces before changing a job. Downloads and bundles in the new SDK need vxs3[workers] >=1.2.0; Bundle registration also needs the API’s bundle migration and write permission. A source merge alone does not establish deployment.

The single most effective change for most jobs: store and read fewer, larger objects. Bundle ordinary files into tar shards; for Zarr, review native sharding or immutable bundles with a compatible reader before a job depends on them. See Fewer, larger objects.

Storage choices

Tier Path Good for Bad for
Node-local NVMe /cache ($VXRUN_CACHE_DIR, request with --local-cache 64Gi) hot training data, framework caches, scratch anything that must survive the Pod
Object storage (Ceph RGW) s3://vxdata/... via vxdata-sdk and vxs3 datasets, model weights, published results per-step tiny writes
Scratch bucket (Ceph RGW) s3://scratch/<username>/<experiment-or-workflow>/ via the [scratch] profile development experiments, intermediate data, temporary checkpoints permanent data or repeatable workflow inputs and outputs; objects are garbage-collected after two weeks (14 days)
NFS (legacy) /mnt/storage (read-only in jobs), /mnt/artifacts (run-image /output, read-write) existing inputs not yet published to vxdata; run-image outputs you have not moved yet anything new; random reads in a training loop; frequent large checkpoint rewrites. Not mounted in run-batch

Pattern for a job: download once from object storage into /cache, train from /cache, checkpoint to the scratch bucket (or /output when checkpoints are small and rare), then publish data and results you want to keep as permanent vxdata resources. In a run-batch item, /checkpoints ($VXRUN_CHECKPOINTS_DIR) is the node-local directory shared by that item’s retries: cheap for frequent saves, but gone with the item’s claim, so it does not cover a resubmitted batch or a lost node. Keep the scratch bucket for those.

NFS is legacy; run-batch is the S3-native path

NFS (/mnt/storage, /mnt/artifacts) is the legacy tier. It stays mounted in run-image jobs and Coder workspaces for existing data, but nothing new should depend on it: exports can be full or read-only, Coder startup no longer writes to them, and the platform direction is to remove them. Treat every NFS path a job reads or writes as something to migrate, not a design choice.

run-batch is the replacement and has no NFS or hostPath mounts:

  • Inputs come from S3 (vxdata resources or the scratch bucket) and are staged once into /cache, node-local NVMe.
  • Each item writes /output on a temporary per-item TopoLVM claim on the node’s NVMe (16Gi with local_cache: "0", otherwise the local_cache size).
  • The pack step publishes that output to s3://workflow-output/... as a vxs3 bundle, the exit handler completes the batch, and vxrun batch results collects it. S3 is the only durable copy; the claim is deleted after the item.

This is faster as well as cleaner. Reads and writes hit local NVMe instead of a shared network filesystem that every job on the cluster contends for, so the GPU is not waiting on file-system latency. Publication uploads a few large shards in parallel rather than leaving thousands of small files on an export, and the result is immutable, addressable by prefix and discoverable through the catalog instead of a mutable directory someone has to remember.

What this means in practice:

  • New batch or sweep work: use vxrun batch; outputs publish themselves.
  • A single run-image job: read from S3 into /cache, checkpoint to scratch, and publish results with vxrun storage upload rather than leaving them on /mnt/artifacts.
  • Existing inputs on /mnt/storage: bundle them once with vxrun storage upload and point jobs at the vxdata resource.
  • Existing outputs on /mnt/artifacts: publish what you want to keep the same way.

Where the S3 credentials are

You never need to ask for S3 keys. Every Coder workspace and every run-image or run-batch Pod carries the shared credentials file; AWS_SHARED_CREDENTIALS_FILE points at it (fall back to ~/.aws/credentials). The same profiles exist in Coder workspaces and in run-image/run-batch Pods: [scratch] reads and writes the scratch bucket; [workflow-output] reads the workflow-output bucket (the RGW user coder-workspace, read-only; the pack and finalise steps hold a separate writer file); [default] is the same identity as [workflow-output], so a client with no profile set can read published batch results. Each profile carries endpoint_url = https://s3.fra.virdx.dev, so boto3.Session(profile_name=...), s3fs and AWS_PROFILE=... aws s3 ... work unchanged. [default] is no longer real-AWS credentials; older comments that say so are stale. s3://vxdata goes through the vxdata SDK with VXDATA_TOKEN, not S3 keys. So when a job or a migration from NFS needs S3 access, look in the environment first; the only thing to copy is a profile name, never a key.

Fewer, larger objects: bundle files, reshard Zarr

Every object costs the same fixed work whether it holds 4 KiB or 1 GiB: a catalog or listing entry, a presigned URL, at least one HTTP request to Ceph RGW, a verification read and a retry slot. Object storage and the vxdata catalog scale with object count, not bytes; the GPU waits on request latency, not bandwidth. A dataset of 200,000 PNGs is 200,000 round trips per epoch read directly, and 200,000 entries for every listing, grant check and GC pass. The same bytes in 1 GiB tar shards are a few hundred objects, four range requests each.

What the tooling does about it:

  • Bundling (ordinary files): vxs3 packs a directory into uncompressed tar shards targeting 1 GiB plus a parquet manifest. Whole restores fetch shards on the 256 MiB range grid, so throughput is bounded by concurrency rather than per-file latency. D3N caches the underlying tail stripes, typically 4 MiB each; one tar shard can occupy many cache entries. Selective materialization fetches the chosen files’ byte ranges, merged per shard. The manifest answers “what is in here” and “where is file X” without listing the bucket or touching a shard.
  • Resharding (Zarr): a store with small chunks is one object per chunk and one request per chunk. vxrun storage optimise rewrites it as a Zarr v3 copy whose shards target 1 GiB while preserving the inner chunk shape, values and codecs, so readers keep their access pattern but issue far fewer requests.
  • Bundled Zarr: packing preserves existing encoded files and metadata. vxs3.zarr.store.BundleStore or client.bundles.open_zarr_store(...) translates Zarr reads, including native shard ranges, into bundle range requests without extracting files. Readers must support this adapter; ordinary Zarr S3 readers cannot open the tar directly. Bundling does not validate metadata or optimise chunks. The bundle stays immutable and the store is read-only. With SDK download_routing="d3n", streaming fetches complete 4 MiB Ceph stripes through Ketama and slices the requested bytes. Gateway mode retains exact ranges. Read the D3N guide linked from read_doc("vx/cluster/vxs3-storage") before assuming partial reads reuse warmed server caches or setting memory budgets.
  • Keep a few large files as they are: a 2 GiB weights file is already one object, and bundling it buys nothing.

inspect_data and vxrun data inspect apply exactly this policy: objects below 4 MiB count as small, any selection with more than one payload object and a small one can get bundle. Zarr metadata candidates get review_zarr_storage; native layout findings may instead recommend investigate_zarr_sharding. Review metadata, reader compatibility and access patterns before choosing a layout. Packing eligibility does not establish that bundling is the right choice, and object counts do not establish a speedup.

Data Temporary (scratch) Permanent (vxdata)
Directory of ordinary files vxs3.bundle.local.pack + publish_bundle to scratch client.bundles.upload(...) (SDK, registers a Bundle) or vxrun storage upload DIR (worker, always bundles)
Zarr store optimize_local_zarr before upload vxrun storage optimise s3://.../store.zarr/ writes the resharded copy to vxdata-optimised
Zarr with BundleStore consumers pack unchanged files + publish_bundle to scratch client.bundles.upload(...); stream with client.bundles.open_zarr_store(...)
A few large files keep keep; vxrun storage upload accepts a standalone file of at least 100 MiB

Two consequences to design around. Reading members of a bundle fetches their byte ranges merged per shard, so files read together are cheapest when they sit side by side. Packing walks paths in sorted order, so lay the source out so such files share a directory or name prefix (by split, class or sample id), or pass pack_by= (or a records table) to client.bundles.upload to set the order. And resharded Zarr needs a Zarr v3 capable reader and a grant on vxdata-optimised; check both before pointing a job at the optimised copy.

Scratch is for development

Use scratch while developing and experimenting with data, models and results. Its two-week (14-day) garbage-collection policy makes it temporary storage. Do not use scratch as the permanent source for a dataset, model or repeatable workflow. Checkpoint/resume can use it within that retention window; restarting a job does not exempt its objects from GC.

Once the developer is happy with the data, publish it to vxdata and create a permanent resource. Future workflows should use that resource. The retained bytes must also be in vxdata storage: registering a resource that points to a scratch URL does not preserve the data. See Publishing data below; use --retention permanent for the permanent resource and finish the upload before the scratch objects expire.

Access scratch through the [scratch] profile, not a filesystem mount. Coder workspaces and run-image/run-batch jobs always mount the shared credentials file and set AWS_SHARED_CREDENTIALS_FILE (see “Where the S3 credentials are” above). Its [scratch] profile carries endpoint_url = https://s3.fra.virdx.dev, so boto3.Session(profile_name="scratch"), s3fs.S3FileSystem(profile="scratch") or AWS_PROFILE=scratch aws s3 ... work unchanged. For a boto3 connection check with the internal CA, see read_doc(“vx/cluster/submitting-jobs”), “Scratch storage for development”. The identity is shared, so the <username>/<experiment-or-workflow>/ prefix is a convention, not an access boundary. The bucket has a shared size quota; delete temporary data you no longer need. A new submission has a new workflow name, so resuming an earlier run means pointing the job at the old prefix (or the old /output directory), while its data is still available.

Review scratch layout and request frequency

Before uploading, call inspect_data on the local source with scratch=true (and repo_path if the source is under the repository), or run:

vxrun data inspect ./data --scratch

This mode gives temporary-storage advice instead of suggesting immediate publication. vxrun review also emits an advisory finding for scratch URLs, profiles and bucket references. Neither check blocks a job. A few large files or occasional checkpoints can be fine; look at object counts, sizes and access patterns before changing the layout.

For many small records, consolidate into a suitable native format such as Parquet if the readers support it. For independent ordinary files, pack locally into vxs3 tar shards before upload:

from vxs3.bundle.local import pack

pack("./data", "./data-bundle")  # destination must be new; target 1 GiB shards

Pass workers=8 to write shards in worker processes, or set VXS3_WORKERS=8 for every pack and extract in the job (default 0); scripts that use workers need an if __name__ == "__main__": guard.

Publish the packed bundle with vxs3.bundle.io.publish_bundle and an S3Destination configured for scratch. It verifies each shard against the store’s ETag as it uploads (reading back only what the store cannot attest) and publishes the manifest last. vxs3.bundle.io.upload_bundle packs and publishes in one pass, uploading shards as they are written and holding only shards in flight in scratch. Keep the returned manifest URI and hash; use open_bundle(ref, store=store) to restore once onto /cache or materialize selected members into a shared local cache. See read_doc(“vx/cluster/vxs3”) for the complete direct-S3 example. All objects in this bundle still expire under the scratch policy; the reference does not extend retention. Keep restored data on local NVMe across epochs and workers when using materialisation. For Zarr, review chunking/sharding, metadata and reader compatibility: native stores and unchanged stores in immutable bundles are both options. BundleStore can stream selected chunks without disk staging. Batch small writes and avoid repeated listings or each worker independently downloading the same inputs. Bound aggregate transfer concurrency.

Prefer the local source: the analyzer itself should not add unnecessary Ceph load. Scratch mode defaults to 10,000 objects, 32 sources, 64 metadata probes, 4 MiB of metadata and a 30-second cooperative time budget. LIST/HEAD requests are outside the probe budget and in-flight requests can overrun the deadline. Whole-bucket scans are refused; select only the relevant prefix or object. Remote analysis uses environment-based S3 credentials and endpoint, not the boto3 session shown above; AWS_PROFILE=scratch alone is insufficient.

Report partial/failed coverage explicitly: it does not establish that the remaining data is well laid out. Object shape and estimated transfer requests are not measured IOPS. Review GET/LIST/PUT frequency in the job as well.

How transfers actually work

  • vxdata-sdk (client.storage.download) obtains presigned URLs and exact object sizes from the vxdata API, then delegates downloads to vxs3’s 256 MiB range grid, shared concurrency budget, verification and retry logic. It never holds S3 keys. TransferConfig(max_streams=...) defaults to 16; choose concurrency across the whole job, including worker processes. The SDK renews rejected URLs once while preserving learned ETags. Existing uploads use the SDK’s HTTP uploader. Download TLS/timeouts use storage_options or TransferConfig(client_options=...), independently of the API’s httpx client. TransferConfig(download_routing=...) picks the path: "gateway" (the SDK default) uses the signed gateway URL with no worker processes; "d3n" requires the D3N inventory (VXS3_D3N_INSTANCES, or the /etc/virdx/d3n/d3n_instances.json that Kyverno mounts into workflow Pods and Coder workspaces) and fans ranges out to D3N nodes in worker processes; "auto" uses D3N when an inventory is configured. Note the defaults differ: vxs3’s own DownloadPool defaults to routing="d3n". The same setting applies to open_bundle, materialize and open; pass it when opening the bundle. In scripts, keep D3N downloads under if __name__ == "__main__": so spawned workers can import the module.
  • vxs3 packs a directory into uncompressed tar shards targeting 1 GiB (4 GiB ceiling, 100,000 entries per shard) plus a Parquet manifest that records each file’s byte range. A 1 GiB shard needs 4 payload range requests, before metadata, retries or verification reads. vxs3 analyze calls objects below 4 MiB “small” and can suggest bundle when an ordinary-file selection has more than one payload object and any of them is small.
  • Native Zarr optimisation: vxrun storage optimise s3://.../store.zarr/ writes a separate, resharded Zarr v3 copy at the same path in the vxdata-optimised bucket. The source is untouched; readers need a grant on that bucket and must be Zarr v3 capable.
  • Bundled Zarr streaming uses exact member-relative ranges through BundleStore, with a read-size limit, concurrency limit and bounded memory cache. Full member reads verify SHA-256; partial reads pin a strong shard ETag for version consistency without verifying the whole-member hash. It does not use the bulk downloader’s 256 MiB grid or require a disk cache.

bundle.list_entries() and get_member() use the manifest without shard I/O. materialize() and open() fetch and verify complete selected members, merging their byte ranges per shard, then reuse an immutable local cache. Group files by access pattern and share the cache across workers on the same node. Copy a cached file before editing it. There is no automatic cache eviction. bundle.download() restores all files into a new destination separately from the member cache. Allow disk for staging and output; publication also reads uploaded bytes back for verification.

Ordinary SDK directory transfers preserve physical object layout. They do not automatically unpack bundles, reshard Zarr or produce native Zarr completion records. Use the explicit bundle or native Zarr interfaces for those operations.

Publishing a Bundle from local files

Use the SDK for ordinary local files or directories when a durable, discoverable bundle resource is wanted. Keep the source immutable during publication, and put local scratch outside it on a volume with room for packed data and verification:

import polars as pl
from vxdata.sdk import Client

records = pl.DataFrame({"subject": [0, 1], "path_image": ["sub-0/t2w.nii.gz", "sub-1/t2w.nii.gz"]})
with Client() as client:
    resource = client.bundles.upload(
        "./data",
        group="example",
        title="example",
        records=records,  # optional: your own index of the files
        pack_by=["subject"],  # optional: files read together sit together
        bucket="vxdata",
        scratch_dir="/cache/publish",
        concurrency=8,
    )
    print(resource.identifier)  # bundle/<uuid4>

This stores the records table, publishes verified bytes to permanent storage, then registers a Bundle pinning the exact manifest and records by URL and SHA-256; the identifier is allocated for you. Here scratch_dir is temporary local disk, not the S3 scratch bucket. This API has no --retention flag; do not select the expiring scratch bucket for retained inputs. For a remote scratch prefix, use the worker upload path below or stage it locally before SDK publication.

If registration fails after publication, BundleRegistrationError.resource retains the create request and reference. With rejected=True, fix the rejected request or permissions. Otherwise retrieve the identifier first to reconcile an uncertain write, and create the resource only if absent. Do not upload again. See read_doc(“vx/cluster/vxdata-sdk”) for the full recovery and authorization contract. The API stores the publisher’s reference; registration itself does not verify shards.

Batch outputs (run-batch publication)

A run-batch Workflow publishes by default (publish: "true"): after each item’s command, a pack step turns that attempt’s /output into a vxs3 bundle at s3://workflow-output/<user>/<workflow>-<uid>/items/<id>/, and the exit handler writes workflow.json (name, uid, user, status, expected item ids), then items.parquet (one row per expected item: item, status succeeded|failed, manifest_uri, manifest_sha256, files, bytes, detail), and then, only when every item succeeded, the root manifest.parquet: the completion marker for the whole batch, like a multipart upload that becomes an object only on completion. The root is one composed bundle: items/<id>/... lists an item’s files, shards stay where the items published them, and each item’s own manifest remains a complete bundle. Reading it needs vxs3 3.1 or later. (Finalisers before vxrun 0.4.7 also wrote a root for partial batches; items.parquet tells them apart.)

Advice that follows from this:

  • Write results under /output (VXRUN_OUTPUT_DIR), not scratch, when they should outlive the Pod; the pack step publishes exactly that directory.
  • Each item’s /output is a temporary TopoLVM claim on the node (16Gi with local_cache: "0", otherwise the local_cache size); there are no NFS mounts, so read inputs from S3/vxdata into /cache. The pack step mounts the claim read-only and assembles the shards in its own /cache before upload, so local_cache must cover an item’s outputs and the packing space.
  • A failed item (exhausted retries, or a pack retry after a partial upload) has no manifest and appears in items.parquet as failed; the batch then gets no root manifest and no catalog entry. The successful items’ bundles remain and can be fetched one at a time (vxrun batch results --item); the fix is to re-run, which is a new batch under a new prefix.
  • Terminate skips the exit handler (no items.parquet, no root); Stop runs it and records which items finished.
  • register: "true" also registers the root as a vxdata Bundle with items.parquet as its records table, for discovery through the catalog.
  • Published outputs are immutable from the user’s point of view. The writer identity (RGW user workflow-writer: put, get, list, no delete) lives in the workflow-output-writer secret, mounted only by the controller-created pack and finalise pods, which run a vxrun-tools image pinned by digest inside the template; there is no tools-image parameter. The user’s main container gets aws-credentials, whose [workflow-output] profile is read-only on the bucket (RGW user coder-workspace, the same keys as [default]). Kyverno policy argo-restrict-workflow-output-writer rejects submissions referencing the writer secret, run-batch submissions that add spec.templates or spec.podSpecPatch, other pods mounting the secret, and in-cluster edits of ArgoCD-managed WorkflowTemplates. Known limits: the writer can overwrite existing keys (software-guarded; bucket versioning would close it), and an item’s output is mutable on its temporary claim until packed. Published output is retained; there is no lifecycle expiry on workflow-output. The claim is deleted after each item (also with publish: "false"), so S3 is the only durable copy.

Collecting results

vxrun batch results <workflow-name>                     # prefix, completion, one line per item
vxrun batch results <workflow-name> --json
vxrun batch results <workflow-name> --download ./out    # complete batch: ./out/items/<id>/...
vxrun batch results <workflow-name> --download ./out --item trial-0   # one item, works when incomplete
vxrun batch results s3://workflow-output/<user>/<name>-<uid>          # by prefix

results derives the prefix from the live Workflow; once the Workflow is garbage-collected (three days after it finishes) it finds <user>/<name>-<uid>/ in the bucket and reads workflow.json, so results outlive the Workflow. Before the exit handler has run it reports items as published/pending; after, the items.parquet rows; complete is the root manifest’s presence (its SHA-256 is reported and pinned on download). It reads the bucket with the [workflow-output] profile from AWS_SHARED_CREDENTIALS_FILE or ~/.aws/credentials; Coder workspaces and job containers both carry it (--profile overrides). Decide from the per-item status before using a batch; a whole-batch download is refused when the root manifest is absent.

In Python, take prefix or an item’s manifest_uri from --json and open it with vxs3; vxrun.artifacts.profile_store builds the store from the same profile:

from vxrun.artifacts import profile_store
from vxs3.bundle.io import open_bundle

store = profile_store("default", "workflow-output", path="~/.aws/credentials")  # profile, bucket
root = open_bundle(
    f"{prefix}/manifest.parquet", store=store, sha256=sha256
)  # complete batches only
root.list_entries("items/trial-0/")  # one item's files, no shard I/O
root.materialize(["items/trial-0/metrics.json"])  # {path: cached local file}, byte range only
item = open_bundle(f"{prefix}/items/trial-0/manifest.parquet", store=store)  # any published item
item.extract("./out/items/trial-0")

When the batch ran with register: "true" and completed, go through the catalog instead; the finalise pod log prints the Bundle identifier and the title is the Workflow name:

from vxdata.sdk import Client, F

with Client() as client:
    found = client.bundles.query().filter(F.title == "<workflow-name>").collect()
    bundle = client.bundles.retrieve(found["identifier"][0])  # or the logged bundle/<uuid4>
    items = client.bundles.read_records(bundle)  # items.parquet, hash-verified
    client.bundles.materialize(bundle, "./out", paths=["items/trial-0/metrics.json"])

The same composition works outside run-batch: workers publish parts with publish_bundle beneath sibling prefixes and a finaliser merges them with merge_manifests and publish_manifest, then registers with client.bundles.register. See “Writing one bundle from many workers” in read_doc(“vx/cluster/vxdata-sdk”).

Publishing data (vxrun storage upload)

vxrun storage upload /mnt/storage/<path> --name <dataset> --project <project> \
  --description "<what it is>" --data-role reference --retention review-180d --wait

To keep a finished development dataset from scratch, stop modifying its prefix and publish it as a permanent resource:

vxrun storage upload s3://scratch/<username>/<experiment>/ \
  --name <dataset> --project <project> --description "<what it is>" \
  --data-role derived --retention permanent --wait

After the upload succeeds, use the returned vxdata resource and data URI in repeatable workflows. Publication creates a separate stored copy; the scratch source remains subject to GC. S3 uploads require the backend capability documented in read_doc(“vx/cluster/submitting-jobs”), “Vxoptimise service workflows”.

  • Source: a path under /mnt/storage or /mnt/artifacts, or an s3://bucket/prefix/ (a directory prefix, never a bucket or single object).
  • Accepted shapes: a valid root Zarr, a standalone file of at least 100 MiB, or a directory, which is always bundled. Small standalone files are rejected.
  • Limits per upload: 256 GiB and 100,000 entries. The source must not change during the run; the worker re-lists and refuses if it did.
  • --data-role: source, derived, reference, model, results. --retention: temporary-30d, review-180d, permanent. --project is a lowercase slug. Also --metadata file.json, --dry-run, --timeout.
  • Result: an immutable s3://vxdata/storage/<prefix>/ and a catalog record vxupload-<workflow-uid> that turns ready only after read-back verification. vxrun storage status [NAME] [--json] shows your jobs with progress and, on success, the data URI; rows disappear with the Workflow TTL (2 days). Durable lookup: Client().generic_files.retrieve("vxupload-<uid>"), use file_metadata.data_uri only when state == "ready".

Reading data in a job

For a registered Bundle, keep the client open through bundle reads:

import os
from pathlib import Path
import polars as pl
from vxdata.sdk import Client

cache = Path(os.environ.get("VXRUN_CACHE_DIR", "/cache"))
with Client() as client:
    bundle = client.bundles.retrieve("bundle/<uuid4>")
    client.bundles.materialize(bundle, cache / "data")  # every file, by byte range
    # Or only what a records query names: path_* columns come back pointing into dest.
    picked = client.bundles.read_records(bundle).filter(pl.col("subject") == 0)
    local = client.bundles.materialize(bundle, cache / "data", paths=picked)

The worker upload path above still returns a GenericFile with file_metadata and a physical data prefix. It does not produce a Bundle resource. Wait for its ready state, then use its reported URI. For an ordinary-file worker bundle, the existing physical-download/unpack path remains valid:

import os
from pathlib import Path
from vxdata.sdk import Client, TransferConfig
from vxs3.bundle.local import unpack

cache = Path(os.environ.get("VXRUN_CACHE_DIR", "/cache"))
with Client() as client:  # VXDATA_TOKEN is injected by run-image
    client.storage.download(
        "s3://vxdata/storage/<prefix>/",
        cache / "bundle",
        transfer_config=TransferConfig(download_routing="d3n", max_streams=8),
    )
data = unpack(cache / "bundle" / "manifest.parquet", cache / "data")  # destination must be new

download_routing="d3n" fails fast when no D3N inventory is configured; use "auto" for code that also runs on a laptop, or omit it for the gateway. Keep the call under a main() guard in scripts so D3N workers can spawn.

Do not infer a Bundle reference from shard names or assume that every worker upload is a tar bundle: a native Zarr upload stays Zarr. Use open_bundle with an explicit manifest reference when one is available (older bundles with a manifest.json.gz still open).

Credentials: run-image mounts the optional secret vxdata-credentials-<run_environment> as VXDATA_TOKEN; no S3 keys are needed for s3://vxdata. The scratch bucket ([scratch]) and read-only workflow-output access ([workflow-output], the same keys as [default]) come from AWS_SHARED_CREDENTIALS_FILE, which is always mounted. When local_cache is set, run-image also exports VXS3_CACHE_DIR=/cache/.vxs3, so vxs3 calls that omit a destination land on the node-local volume without further configuration.

Inspecting data (vxrun data inspect, vxs3 analyze)

vxrun data inspect SOURCE [--table] [--scratch] [--json] runs a bounded metadata scan (no payload downloads) and reports: physical and logical counts and bytes, p50/p95 object sizes, the share below 4 MiB, estimated range requests, detected formats with an evidence level (metadata_verified, header_verified, extension_only, unknown), findings with codes such as bulk.layout and bundle.zarr_reader_required, and one suggested action: keep, bundle, investigate_zarr_sharding, review_zarr_storage or unsupported, plus a command or review advice. Zarr storage review does not automatically upload, convert or optimise the store. coverage is complete, partial (a limit such as max_objects was hit; counts are not extrapolated) or failed. The underlying CLI vxs3 analyze exits 0, 2 and 1 respectively.

What to say to a user

  • Developing or experimenting: “use scratch for temporary data; it is garbage-collected after two weeks. Once you are happy with it, publish a permanent vxdata resource for future or repeatable workflows.”
  • Many small scratch files: “inspect locally with --scratch, then consolidate records or pack ordinary files into 1 GiB tar shards before upload. Every object is a request; a few hundred shards beat a few hundred thousand files. Stage once into /cache; check repeated reads, writes and listings across workers too.”
  • Many small files on NFS: “bundle it once with vxrun storage upload, then stage into /cache at job start; the GPU stops waiting on the file system, and every later job reuses the bundled resource.”
  • A job that writes results to /mnt/artifacts, or a new layout on NFS: “NFS is legacy. Run it as a vxrun batch (outputs publish to S3 on their own, no NFS involved, faster on node-local NVMe), or write to scratch and publish a vxdata resource. Nothing new should depend on /mnt/*.”
  • A Zarr store with tiny chunks: “vxrun storage optimise writes a resharded v3 copy in vxdata-optimised with 1 GiB shards and the same inner chunks; point the job at that copy and it reads chunks with a fraction of the requests.”
  • A few large files already in object storage: “keep; download once to /cache.”
  • Results of a run-batch: “vxrun batch results <workflow> shows the prefix, per-item status and whether the batch completed; --download fetches a complete batch, --item one item even when it did not (see Collecting results). Pass register: "true" next time if it should appear in the catalog.”
  • Analyzer unavailable ([data] error): show the discovered sources and this page; do not guess sizes.
Navigation

Type to search…

↑↓ navigate↵ selectEsc close