---
title: "Storage practices"
image: "https://docs.virdx.dev/img/virdx-social-card.png"
---

> Documentation Index
> Fetch the complete documentation index at: https://docs.virdx.dev/llms.txt
> Use this file to discover all available pages before exploring further.

# Storage practices

How data should be laid out and moved on this cluster, and the tools that do it.
`inspect_data` applies the vxs3 `bulk_nvme_v1` policy; this page is the human
readable version. Facts first, recommendations second.

For the public Python interfaces, read read_doc("vx/cluster/vxs3") (direct S3 and native or bundled Zarr),
read_doc("vx/cluster/vxs3-storage") (verification, caching and transfer details), and
read_doc("vx/cluster/vxdata-sdk") (authenticated transfers and Bundle resources).
read_doc("vx/cluster/vxdata-api") covers the bundle allocation and catalog contract. These are
vendored from the merged source; check the installed SDK and deployed API support
the interfaces before changing a job. Downloads and bundles in the new SDK need
`vxs3[workers] >=1.2.0`; Bundle registration also needs the API's bundle
migration and write permission. A source merge alone does not establish deployment.

The single most effective change for most jobs: **store and read fewer, larger
objects**. Bundle ordinary files into tar shards; for Zarr, review native sharding
or immutable bundles with a compatible reader before a job depends on them.
See [Fewer, larger objects](#fewer-larger-objects-bundle-files-reshard-zarr).

## Storage choices

| Tier | Path | Good for | Bad for |
|---|---|---|---|
| Node-local NVMe | `/cache` (`$VXRUN_CACHE_DIR`, request with `--local-cache 64Gi`) | hot training data, framework caches, scratch | anything that must survive the Pod |
| Object storage (Ceph RGW) | `s3://vxdata/...` via vxdata-sdk and vxs3 | datasets, model weights, published results | per-step tiny writes |
| Scratch bucket (Ceph RGW) | `s3://scratch/<username>/<experiment-or-workflow>/` via the `[scratch]` profile | development experiments, intermediate data, temporary checkpoints | permanent data or repeatable workflow inputs and outputs; objects are garbage-collected after two weeks (14 days) |
| NFS (legacy) | `/mnt/storage` (read-only in jobs), `/mnt/artifacts` (`run-image` `/output`, read-write) | existing inputs not yet published to vxdata; `run-image` outputs you have not moved yet | anything new; random reads in a training loop; frequent large checkpoint rewrites. Not mounted in `run-batch` |

Pattern for a job: download once from object storage into `/cache`, train from
`/cache`, checkpoint to the scratch bucket (or `/output` when checkpoints are
small and rare), then publish data and results you want to keep as permanent
vxdata resources. In a `run-batch` item, `/checkpoints` (`$VXRUN_CHECKPOINTS_DIR`)
is the node-local directory shared by that item's retries: cheap for frequent
saves, but gone with the item's claim, so it does not cover a resubmitted batch
or a lost node. Keep the scratch bucket for those.

## NFS is legacy; run-batch is the S3-native path

NFS (`/mnt/storage`, `/mnt/artifacts`) is the legacy tier. It stays mounted in
`run-image` jobs and Coder workspaces for existing data, but nothing new should
depend on it: exports can be full or read-only, Coder startup no longer writes
to them, and the platform direction is to remove them. Treat every NFS path a
job reads or writes as something to migrate, not a design choice.

`run-batch` is the replacement and has **no NFS or hostPath mounts**:

- Inputs come from S3 (vxdata resources or the scratch bucket) and are staged
  once into `/cache`, node-local NVMe.
- Each item writes `/output` on a temporary per-item TopoLVM claim on the
  node's NVMe (16Gi with `local_cache: "0"`, otherwise the `local_cache` size).
- The `pack` step publishes that output to `s3://workflow-output/...` as a vxs3
  bundle, the exit handler completes the batch, and `vxrun batch results`
  collects it. S3 is the only durable copy; the claim is deleted after the item.

This is faster as well as cleaner. Reads and writes hit local NVMe instead of
a shared network filesystem that every job on the cluster contends for, so the
GPU is not waiting on file-system latency. Publication uploads a few large
shards in parallel rather than leaving thousands of small files on an export,
and the result is immutable, addressable by prefix and discoverable through
the catalog instead of a mutable directory someone has to remember.

What this means in practice:

- New batch or sweep work: use `vxrun batch`; outputs publish themselves.
- A single `run-image` job: read from S3 into `/cache`, checkpoint to scratch,
  and publish results with `vxrun storage upload` rather than leaving them on
  `/mnt/artifacts`.
- Existing inputs on `/mnt/storage`: bundle them once with
  `vxrun storage upload` and point jobs at the vxdata resource.
- Existing outputs on `/mnt/artifacts`: publish what you want to keep the same way.

### Where the S3 credentials are

You never need to ask for S3 keys. Every Coder workspace and every `run-image`
or `run-batch` Pod carries the shared credentials file; `AWS_SHARED_CREDENTIALS_FILE`
points at it (fall back to `~/.aws/credentials`). The same profiles exist in
Coder workspaces and in `run-image`/`run-batch` Pods: `[scratch]` reads and
writes the scratch bucket; `[workflow-output]` reads the `workflow-output`
bucket (the RGW user `coder-workspace`, read-only; the pack and finalise steps
hold a separate writer file); `[default]` is the same identity as
`[workflow-output]`, so a client with no profile set can read published batch
results. Each profile carries
`endpoint_url = https://s3.fra.virdx.dev`, so `boto3.Session(profile_name=...)`,
`s3fs` and `AWS_PROFILE=... aws s3 ...` work unchanged. `[default]` is no longer
real-AWS credentials; older comments that say so are stale. `s3://vxdata` goes through the vxdata SDK with `VXDATA_TOKEN`, not S3
keys. So when a job or a migration from NFS needs S3 access, look in the
environment first; the only thing to copy is a profile name, never a key.

## Fewer, larger objects: bundle files, reshard Zarr

Every object costs the same fixed work whether it holds 4 KiB or 1 GiB: a catalog
or listing entry, a presigned URL, at least one HTTP request to Ceph RGW, a
verification read and a retry slot. Object storage and the vxdata catalog scale
with object count, not bytes; the GPU waits on request latency, not bandwidth.
A dataset of 200,000 PNGs is 200,000 round trips per epoch read directly, and
200,000 entries for every listing, grant check and GC pass. The same bytes in
1 GiB tar shards are a few hundred objects, four range requests each.

What the tooling does about it:

- **Bundling** (ordinary files): vxs3 packs a directory into uncompressed tar
  shards targeting **1 GiB** plus a parquet manifest. Whole restores fetch
  shards on the 256 MiB range grid, so throughput is bounded by concurrency
  rather than per-file latency. D3N caches the underlying tail stripes,
  typically 4 MiB each; one tar shard can occupy many cache entries.
  Selective materialization fetches the chosen files' byte ranges, merged
  per shard. The manifest answers "what is in here" and "where is file X"
  without listing the bucket or touching a shard.
- **Resharding** (Zarr): a store with small chunks is one object per chunk and
  one request per chunk. `vxrun storage optimise` rewrites it as a Zarr v3 copy
  whose shards target 1 GiB while preserving the inner chunk shape, values and
  codecs, so readers keep their access pattern but issue far fewer requests.
- **Bundled Zarr**: packing preserves existing encoded files and metadata.
  `vxs3.zarr.store.BundleStore` or `client.bundles.open_zarr_store(...)` translates
  Zarr reads, including native shard ranges, into bundle range requests without
  extracting files. Readers must support this adapter; ordinary Zarr S3 readers
  cannot open the tar directly. Bundling does not validate metadata or optimise
  chunks. The bundle stays immutable and the store is read-only.
  With SDK `download_routing="d3n"`, streaming fetches complete 4 MiB Ceph
  stripes through Ketama and slices the requested bytes. Gateway mode retains
  exact ranges. Read the D3N guide linked from `read_doc("vx/cluster/vxs3-storage")` before
  assuming partial reads reuse warmed server caches or setting memory budgets.
- **Keep** a few large files as they are: a 2 GiB weights file is already one
  object, and bundling it buys nothing.

`inspect_data` and `vxrun data inspect` apply exactly this policy: objects
below **4 MiB** count as small, any selection with more than one payload object
and a small one can get `bundle`. Zarr metadata candidates get
`review_zarr_storage`; native layout findings may instead recommend
`investigate_zarr_sharding`. Review metadata, reader compatibility and access
patterns before choosing a layout. Packing eligibility does not establish that
bundling is the right choice, and object counts do not establish a speedup.

| Data | Temporary (scratch) | Permanent (vxdata) |
|---|---|---|
| Directory of ordinary files | `vxs3.bundle.local.pack` + `publish_bundle` to scratch | `client.bundles.upload(...)` (SDK, registers a Bundle) or `vxrun storage upload DIR` (worker, always bundles) |
| Zarr store | `optimize_local_zarr` before upload | `vxrun storage optimise s3://.../store.zarr/` writes the resharded copy to `vxdata-optimised` |
| Zarr with BundleStore consumers | `pack` unchanged files + `publish_bundle` to scratch | `client.bundles.upload(...)`; stream with `client.bundles.open_zarr_store(...)` |
| A few large files | keep | keep; `vxrun storage upload` accepts a standalone file of at least 100 MiB |

Two consequences to design around. Reading members of a bundle fetches their
byte ranges **merged per shard**, so files read together are cheapest when they
sit side by side. Packing walks paths in sorted order, so lay the source out so
such files share a directory or name prefix (by split, class or sample id), or
pass `pack_by=` (or a records table) to `client.bundles.upload` to set the order.
And resharded Zarr needs a Zarr v3 capable reader and a grant on
`vxdata-optimised`; check both before pointing a job at the optimised copy.

## Scratch is for development

Use `scratch` while developing and experimenting with data, models and results.
Its **two-week (14-day) garbage-collection policy** makes it temporary storage.
Do not use scratch as the permanent source for a dataset, model or repeatable
workflow. Checkpoint/resume can use it within that retention window; restarting
a job does not exempt its objects from GC.

Once the developer is happy with the data, publish it to vxdata and create a
permanent resource. Future workflows should use that resource. The retained
bytes must also be in vxdata storage: registering a resource that points to a
scratch URL does not preserve the data. See [Publishing data](#publishing-data-vxrun-storage-upload)
below; use `--retention permanent` for the permanent resource and finish the
upload before the scratch objects expire.

Access scratch through the `[scratch]` profile, not a filesystem mount.
Coder workspaces and `run-image`/`run-batch` jobs always mount the shared
credentials file and set `AWS_SHARED_CREDENTIALS_FILE` (see "Where the S3
credentials are" above). Its
`[scratch]` profile carries `endpoint_url = https://s3.fra.virdx.dev`, so
`boto3.Session(profile_name="scratch")`, `s3fs.S3FileSystem(profile="scratch")`
or `AWS_PROFILE=scratch aws s3 ...` work unchanged. For a boto3 connection check
with the internal CA, see read_doc("vx/cluster/submitting-jobs"), "Scratch storage for development".
The identity is shared, so the `<username>/<experiment-or-workflow>/` prefix is
a convention, not an access boundary. The bucket has a shared size quota;
delete temporary data you no longer need. A new submission has a new workflow
name, so resuming an earlier run means pointing the job at the old prefix
(or the old `/output` directory), while its data is still available.

### Review scratch layout and request frequency

Before uploading, call `inspect_data` on the local source with `scratch=true`
(and `repo_path` if the source is under the repository), or run:

```sh
vxrun data inspect ./data --scratch
```

This mode gives temporary-storage advice instead of suggesting immediate
publication. `vxrun review` also emits an advisory finding for scratch URLs,
profiles and bucket references. Neither check blocks a job. A few large files
or occasional checkpoints can be fine; look at object counts, sizes and access
patterns before changing the layout.

For many small records, consolidate into a suitable native format such as
Parquet if the readers support it. For independent ordinary files, pack locally
into vxs3 tar shards before upload:

```python
from vxs3.bundle.local import pack

pack("./data", "./data-bundle")  # destination must be new; target 1 GiB shards
```

Pass `workers=8` to write shards in worker processes, or set `VXS3_WORKERS=8`
for every `pack` and `extract` in the job (default 0); scripts that use
workers need an `if __name__ == "__main__":` guard.

Publish the packed bundle with `vxs3.bundle.io.publish_bundle` and an
`S3Destination` configured for scratch. It verifies each shard against the store's ETag as it uploads
(reading back only what the store cannot attest) and publishes the manifest last.
`vxs3.bundle.io.upload_bundle` packs and publishes in one pass, uploading shards
as they are written and holding only shards in flight in scratch. Keep the returned manifest URI
and hash; use `open_bundle(ref, store=store)` to restore once onto `/cache` or
materialize selected members into a shared local cache. See read_doc("vx/cluster/vxs3") for the
complete direct-S3 example. All objects in this bundle still expire under the
scratch policy; the reference does not extend retention. Keep restored data
on local NVMe across epochs and workers when using materialisation. For Zarr,
review chunking/sharding, metadata and reader compatibility: native stores and
unchanged stores in immutable bundles are both options. BundleStore can stream
selected chunks without disk staging. Batch small writes and avoid repeated listings or each worker
independently downloading the same inputs. Bound aggregate transfer concurrency.

Prefer the local source: the analyzer itself should not add unnecessary Ceph
load. Scratch mode defaults to 10,000 objects, 32 sources, 64 metadata probes,
4 MiB of metadata and a 30-second cooperative time budget. LIST/HEAD requests
are outside the probe budget and in-flight requests can overrun the deadline.
Whole-bucket scans are refused; select only the relevant prefix or object.
Remote analysis uses environment-based S3 credentials and endpoint, not the
boto3 session shown above; `AWS_PROFILE=scratch` alone is insufficient.

Report partial/failed coverage explicitly: it does not establish that the
remaining data is well laid out. Object shape and estimated transfer requests
are not measured IOPS. Review GET/LIST/PUT frequency in the job as well.

## How transfers actually work

- **vxdata-sdk** (`client.storage.download`) obtains presigned URLs and exact
  object sizes from the vxdata API, then delegates downloads to vxs3's **256 MiB
  range grid**, shared concurrency budget, verification and retry logic. It
  never holds S3 keys. `TransferConfig(max_streams=...)` defaults to 16; choose
  concurrency across the whole job, including worker processes. The SDK renews
  rejected URLs once while preserving learned ETags. Existing uploads use the
  SDK's HTTP uploader. Download TLS/timeouts use `storage_options` or
  `TransferConfig(client_options=...)`, independently of the API's httpx client.
  `TransferConfig(download_routing=...)` picks the path: `"gateway"` (the SDK
  default) uses the signed gateway URL with no worker processes; `"d3n"`
  requires the D3N inventory (`VXS3_D3N_INSTANCES`, or the
  `/etc/virdx/d3n/d3n_instances.json` that Kyverno mounts into workflow Pods
  and Coder workspaces) and fans ranges out to D3N nodes in worker processes; `"auto"` uses D3N when an inventory is
  configured. Note the defaults differ: vxs3's own `DownloadPool` defaults to
  `routing="d3n"`. The same setting applies to `open_bundle`, `materialize`
  and `open`; pass it when opening the bundle. In scripts, keep D3N downloads
  under `if __name__ == "__main__":` so spawned workers can import the module.
- **vxs3** packs a directory into uncompressed tar shards targeting **1 GiB**
  (4 GiB ceiling, 100,000 entries per shard) plus a Parquet manifest that
  records each file's byte range. A 1 GiB shard needs 4 payload range requests,
  before metadata, retries or verification reads. `vxs3 analyze` calls objects
  **below 4 MiB** "small" and can suggest `bundle` when an ordinary-file
  selection has more than one payload object and any of them is small.
- **Native Zarr optimisation**: `vxrun storage optimise s3://.../store.zarr/`
  writes a *separate*, resharded Zarr v3 copy at the same path in the
  `vxdata-optimised` bucket. The source is untouched; readers need a grant on
  that bucket and must be Zarr v3 capable.
- **Bundled Zarr streaming** uses exact member-relative ranges through
  `BundleStore`, with a read-size limit, concurrency limit and bounded memory
  cache. Full member reads verify SHA-256; partial reads pin a strong shard
  ETag for version consistency without verifying the whole-member hash.
  It does not use the bulk downloader's 256 MiB grid or require a disk cache.

`bundle.list_entries()` and `get_member()` use the manifest without shard I/O.
`materialize()` and `open()` fetch and verify **complete selected members**,
merging their byte ranges per shard, then reuse an immutable local cache.
Group files by access pattern and share the cache across
workers on the same node. Copy a cached file before editing it. There is no
automatic cache eviction. `bundle.download()` restores all files into a new
destination separately from the member cache. Allow disk for staging and output;
publication also reads uploaded bytes back for verification.

Ordinary SDK directory transfers preserve physical object layout. They do not
automatically unpack bundles, reshard Zarr or produce native Zarr completion
records. Use the explicit bundle or native Zarr interfaces for those operations.

## Publishing a Bundle from local files

Use the SDK for ordinary local files or directories when a durable, discoverable
bundle resource is wanted. Keep the source immutable during publication, and
put local scratch outside it on a volume with room for packed data and verification:

```python
import polars as pl
from vxdata.sdk import Client

records = pl.DataFrame({"subject": [0, 1], "path_image": ["sub-0/t2w.nii.gz", "sub-1/t2w.nii.gz"]})
with Client() as client:
resource = client.bundles.upload(
    "./data",
    group="example",
    title="example",
    records=records,  # optional: your own index of the files
    pack_by=["subject"],  # optional: files read together sit together
    bucket="vxdata",
    scratch_dir="/cache/publish",
    concurrency=8,
)
print(resource.identifier)  # bundle/<uuid4>
```

This stores the records table, publishes verified bytes to permanent storage,
then registers a `Bundle` pinning the exact manifest and records by URL and
SHA-256; the identifier is allocated for you. Here `scratch_dir` is temporary
**local disk**, not the S3 scratch bucket. This API has no `--retention` flag;
do not select the expiring scratch bucket for retained inputs. For a remote
scratch prefix, use the worker upload path below or stage it locally before SDK
publication.

If registration fails after publication, `BundleRegistrationError.resource`
retains the create request and reference. With `rejected=True`, fix the rejected
request or permissions. Otherwise retrieve the identifier first to reconcile an
uncertain write, and create the resource only if absent. Do not upload again.
See read_doc("vx/cluster/vxdata-sdk") for the full recovery and authorization contract. The API
stores the publisher's reference; registration itself does not verify shards.

## Batch outputs (run-batch publication)

A run-batch Workflow publishes by default (`publish: "true"`): after each item's
command, a `pack` step turns that attempt's `/output` into a vxs3 bundle at
`s3://workflow-output/<user>/<workflow>-<uid>/items/<id>/`, and the exit handler
writes `workflow.json` (name, uid, user, status, expected item ids), then
`items.parquet` (one row per expected item: `item`, `status` succeeded|failed,
`manifest_uri`, `manifest_sha256`, `files`, `bytes`, `detail`), and then, only
when every item succeeded, the root `manifest.parquet`: the completion marker
for the whole batch, like a multipart upload that becomes an object only on
completion. The root is one composed bundle: `items/<id>/...` lists an item's
files, shards stay where the items published them, and each item's own manifest
remains a complete bundle. Reading it needs vxs3 3.1 or later. (Finalisers
before vxrun 0.4.7 also wrote a root for partial batches; `items.parquet` tells
them apart.)

Advice that follows from this:

- Write results under `/output` (`VXRUN_OUTPUT_DIR`), not scratch, when they
  should outlive the Pod; the pack step publishes exactly that directory.
- Each item's `/output` is a temporary TopoLVM claim on the node (16Gi with
  `local_cache: "0"`, otherwise the `local_cache` size); there are no NFS
  mounts, so read inputs from S3/vxdata into `/cache`. The pack step mounts the
  claim read-only and assembles the shards in its own `/cache` before upload,
  so `local_cache` must cover an item's outputs and the packing space.
- A failed item (exhausted retries, or a pack retry after a partial upload) has
  no manifest and appears in `items.parquet` as failed; the batch then gets no
  root manifest and no catalog entry. The successful items' bundles remain and
  can be fetched one at a time (`vxrun batch results --item`); the fix is to
  re-run, which is a new batch under a new prefix.
- Terminate skips the exit handler (no `items.parquet`, no root); Stop runs it
  and records which items finished.
- `register: "true"` also registers the root as a vxdata Bundle with
  `items.parquet` as its records table, for discovery through the catalog.
- Published outputs are immutable from the user's point of view. The writer
  identity (RGW user `workflow-writer`: put, get, list, no delete) lives in the
  `workflow-output-writer` secret, mounted only by the controller-created `pack`
  and `finalise` pods, which run a vxrun-tools image pinned by digest inside the
  template; there is no `tools-image` parameter. The user's `main` container
  gets `aws-credentials`, whose `[workflow-output]` profile is read-only on
  the bucket (RGW user `coder-workspace`, the same keys as `[default]`). Kyverno policy `argo-restrict-workflow-output-writer`
  rejects submissions referencing the writer secret, run-batch submissions that
  add `spec.templates` or `spec.podSpecPatch`, other pods mounting the secret,
  and in-cluster edits of ArgoCD-managed WorkflowTemplates. Known limits: the
  writer can overwrite existing keys (software-guarded; bucket versioning would
  close it), and an item's output is mutable on its temporary claim until
  packed. Published output is retained; there is no lifecycle expiry on
  workflow-output. The claim is deleted after each item (also with
  `publish: "false"`), so S3 is the only durable copy.

### Collecting results

```sh
vxrun batch results <workflow-name>                     # prefix, completion, one line per item
vxrun batch results <workflow-name> --json
vxrun batch results <workflow-name> --download ./out    # complete batch: ./out/items/<id>/...
vxrun batch results <workflow-name> --download ./out --item trial-0   # one item, works when incomplete
vxrun batch results s3://workflow-output/<user>/<name>-<uid>          # by prefix
```

`results` derives the prefix from the live Workflow; once the Workflow is
garbage-collected (three days after it finishes) it finds `<user>/<name>-<uid>/`
in the bucket and reads `workflow.json`, so results outlive the Workflow. Before
the exit handler has run it reports items as `published`/`pending`; after,
the `items.parquet` rows; `complete` is the root manifest's presence (its
SHA-256 is reported and pinned on download). It reads the bucket with the
`[workflow-output]` profile from `AWS_SHARED_CREDENTIALS_FILE` or
`~/.aws/credentials`; Coder workspaces and job containers both carry it
(`--profile` overrides). Decide from the per-item status before using a batch;
a whole-batch download is refused when the root manifest is absent.

In Python, take `prefix` or an item's `manifest_uri` from `--json` and open it
with vxs3; `vxrun.artifacts.profile_store` builds the store from the same
profile:

```python
from vxrun.artifacts import profile_store
from vxs3.bundle.io import open_bundle

store = profile_store("default", "workflow-output", path="~/.aws/credentials")  # profile, bucket
root = open_bundle(
f"{prefix}/manifest.parquet", store=store, sha256=sha256
)  # complete batches only
root.list_entries("items/trial-0/")  # one item's files, no shard I/O
root.materialize(["items/trial-0/metrics.json"])  # {path: cached local file}, byte range only
item = open_bundle(f"{prefix}/items/trial-0/manifest.parquet", store=store)  # any published item
item.extract("./out/items/trial-0")
```

When the batch ran with `register: "true"` and completed, go through the
catalog instead; the finalise pod log prints the Bundle identifier and the title
is the Workflow name:

```python
from vxdata.sdk import Client, F

with Client() as client:
found = client.bundles.query().filter(F.title == "<workflow-name>").collect()
bundle = client.bundles.retrieve(found["identifier"][0])  # or the logged bundle/<uuid4>
items = client.bundles.read_records(bundle)  # items.parquet, hash-verified
client.bundles.materialize(bundle, "./out", paths=["items/trial-0/metrics.json"])
```

The same composition works outside run-batch: workers publish parts with
`publish_bundle` beneath sibling prefixes and a finaliser merges them with
`merge_manifests` and `publish_manifest`, then registers with
`client.bundles.register`. See "Writing one bundle from many workers" in
read_doc("vx/cluster/vxdata-sdk").

## Publishing data (`vxrun storage upload`)

```sh
vxrun storage upload /mnt/storage/<path> --name <dataset> --project <project> \
  --description "<what it is>" --data-role reference --retention review-180d --wait
```

To keep a finished development dataset from scratch, stop modifying its prefix
and publish it as a permanent resource:

```sh
vxrun storage upload s3://scratch/<username>/<experiment>/ \
  --name <dataset> --project <project> --description "<what it is>" \
  --data-role derived --retention permanent --wait
```

After the upload succeeds, use the returned vxdata resource and data URI in
repeatable workflows. Publication creates a separate stored copy; the scratch
source remains subject to GC. S3 uploads require the backend capability
documented in read_doc("vx/cluster/submitting-jobs"), "Vxoptimise service workflows".

- Source: a path under `/mnt/storage` or `/mnt/artifacts`, or an
  `s3://bucket/prefix/` (a directory prefix, never a bucket or single object).
- Accepted shapes: a valid root Zarr, a standalone file of at least 100 MiB, or
  a directory, which is always bundled. Small standalone files are rejected.
- Limits per upload: 256 GiB and 100,000 entries. The source must not change
  during the run; the worker re-lists and refuses if it did.
- `--data-role`: source, derived, reference, model, results.
  `--retention`: temporary-30d, review-180d, permanent. `--project` is a
  lowercase slug. Also `--metadata file.json`, `--dry-run`, `--timeout`.
- Result: an immutable `s3://vxdata/storage/<prefix>/` and a catalog record
  `vxupload-<workflow-uid>` that turns `ready` only after read-back
  verification. `vxrun storage status [NAME] [--json]` shows your jobs with
  progress and, on success, the data URI; rows disappear with the Workflow
  TTL (2 days). Durable lookup:
  `Client().generic_files.retrieve("vxupload-<uid>")`, use
  `file_metadata.data_uri` only when `state == "ready"`.

## Reading data in a job

For a registered Bundle, keep the client open through bundle reads:

```python
import os
from pathlib import Path
import polars as pl
from vxdata.sdk import Client

cache = Path(os.environ.get("VXRUN_CACHE_DIR", "/cache"))
with Client() as client:
bundle = client.bundles.retrieve("bundle/<uuid4>")
client.bundles.materialize(bundle, cache / "data")  # every file, by byte range
# Or only what a records query names: path_* columns come back pointing into dest.
picked = client.bundles.read_records(bundle).filter(pl.col("subject") == 0)
local = client.bundles.materialize(bundle, cache / "data", paths=picked)
```

The worker upload path above still returns a `GenericFile` with `file_metadata`
and a physical data prefix. It does not produce a Bundle resource. Wait for
its `ready` state, then use its reported URI. For an ordinary-file worker bundle,
the existing physical-download/unpack path remains valid:

```python
import os
from pathlib import Path
from vxdata.sdk import Client, TransferConfig
from vxs3.bundle.local import unpack

cache = Path(os.environ.get("VXRUN_CACHE_DIR", "/cache"))
with Client() as client:  # VXDATA_TOKEN is injected by run-image
client.storage.download(
    "s3://vxdata/storage/<prefix>/",
    cache / "bundle",
    transfer_config=TransferConfig(download_routing="d3n", max_streams=8),
)
data = unpack(cache / "bundle" / "manifest.parquet", cache / "data")  # destination must be new
```

`download_routing="d3n"` fails fast when no D3N inventory is configured; use
`"auto"` for code that also runs on a laptop, or omit it for the gateway. Keep
the call under a `main()` guard in scripts so D3N workers can spawn.

Do not infer a Bundle reference from shard names or assume that every worker
upload is a tar bundle: a native Zarr upload stays Zarr. Use `open_bundle` with an
explicit manifest reference when one is available (older bundles with a
`manifest.json.gz` still open).

Credentials: run-image mounts the optional secret
`vxdata-credentials-<run_environment>` as `VXDATA_TOKEN`; no S3 keys are
needed for `s3://vxdata`. The scratch bucket (`[scratch]`) and read-only
`workflow-output` access (`[workflow-output]`, the same keys as `[default]`)
come from
`AWS_SHARED_CREDENTIALS_FILE`, which is always mounted. When `local_cache` is set, run-image also exports
`VXS3_CACHE_DIR=/cache/.vxs3`, so vxs3 calls that omit a destination land on
the node-local volume without further configuration.

## Inspecting data (`vxrun data inspect`, `vxs3 analyze`)

`vxrun data inspect SOURCE [--table] [--scratch] [--json]` runs a bounded metadata scan
(no payload downloads) and reports: physical and logical counts and bytes,
p50/p95 object sizes, the share below 4 MiB, estimated range requests, detected
formats with an evidence level (`metadata_verified`, `header_verified`,
`extension_only`, `unknown`), findings with codes such as `bulk.layout` and
`bundle.zarr_reader_required`, and one suggested action: `keep`, `bundle`,
`investigate_zarr_sharding`, `review_zarr_storage` or `unsupported`, plus a
command or review advice. Zarr storage review does not automatically upload,
convert or optimise the store.
`coverage` is `complete`, `partial` (a limit such as `max_objects` was hit;
counts are not extrapolated) or `failed`. The underlying CLI `vxs3 analyze`
exits 0, 2 and 1 respectively.

## What to say to a user

- Developing or experimenting: "use scratch for temporary data; it is
  garbage-collected after two weeks. Once you are happy with it, publish a
  permanent vxdata resource for future or repeatable workflows."
- Many small scratch files: "inspect locally with `--scratch`, then consolidate
  records or pack ordinary files into 1 GiB tar shards before upload. Every
  object is a request; a few hundred shards beat a few hundred thousand files.
  Stage once into `/cache`; check repeated reads, writes and listings across
  workers too."
- Many small files on NFS: "bundle it once with `vxrun storage upload`, then
  stage into `/cache` at job start; the GPU stops waiting on the file system,
  and every later job reuses the bundled resource."
- A job that writes results to `/mnt/artifacts`, or a new layout on NFS: "NFS
  is legacy. Run it as a `vxrun batch` (outputs publish to S3 on their own, no
  NFS involved, faster on node-local NVMe), or write to scratch and publish a
  vxdata resource. Nothing new should depend on `/mnt/*`."
- A Zarr store with tiny chunks: "`vxrun storage optimise` writes a resharded
  v3 copy in vxdata-optimised with 1 GiB shards and the same inner chunks;
  point the job at that copy and it reads chunks with a fraction of the requests."
- A few large files already in object storage: "keep; download once to `/cache`."
- Results of a run-batch: "`vxrun batch results <workflow>` shows the prefix,
  per-item status and whether the batch completed; `--download` fetches a
  complete batch, `--item` one item even when it did not (see Collecting
  results). Pass `register: "true"` next time if it should appear in the catalog."
- Analyzer unavailable (`[data]` error): show the discovered sources and this
  page; do not guess sizes.

Source: https://docs.virdx.dev/vx/cluster/storage-practices/index.mdx
