---
title: "Bundles in Ceph and vxData"
image: "https://docs.virdx.dev/img/virdx-social-card.png"
---

> Documentation Index
> Fetch the complete documentation index at: https://docs.virdx.dev/llms.txt
> Use this file to discover all available pages before exploring further.

# Bundles in Ceph and vxData

*Bundles* are an effective way to store and retrieve large amounts of semi-structured data with Ceph and vxData.

VxData typically enforces a high degree of structure and granularity on data, oftentimes for good reason.
In some rather experimental settings however, integrating data in high granularity may not always be the most effective, feasible, or beneficial for tasks at hand. 
Moreover, data generation tasks involving e.g. millions of very small files is counterproductive for the performance of both vxData's postgres and Ceph when done at larger scales.

In particular, we considered scenarios in which:

- the format/schema of the data is subject to frequent changes
- data is considered an experimental "dump" and is rather short-lived
- data is consumed as a single entity and not cross-matched with other data from vxData
- data is consumed by a single person and not necessarily intended for large-scale sharing and publication

For such scenarios, we instead offer *Bundles*: a simple destination for storing arbitrary data on Ceph. 
Bundles are tarballed automatically, meaning any small files you "upload" will be packed together into fewer, much larger files. 
This happens internally in the background of the write and read process here, so you do not need to be aware of this.
By packing many small files into few larger files, we can achieve much higher read and write speeds with our Ceph infrastructure!

Moreover, Bundles allow you to associate your bundle with a dataframe of your choosing: you can for example store a form of case index present in your bundle alongside it. 
This allows for very fast queries and selective retrieval of files from your bundle.

## How to create a bundle?

```python
import polars as pl
from vxdata.sdk import Client

client = Client()

# the *records dataframe* can describe your bundle contents however you'd like
records = pl.DataFrame({
"subject":   [0, 1],
"b_value":   [0.0, 1000.0],
"path_dwi":  ["sub-0/dwi.npy", "sub-1/dwi.npy"],
"path_mask": ["sub-0/mask.npy", "sub-1/mask.npy"],
})

bundle_id = client.bundles.upload(
"./diffsim-dump",  # path to data you want to upload
group="diffsim",  # needed for a coarse storage location
title="diffsim-release-v2.9",  # display title
records=records,  # metadata describing your stored data
records_meta={"diffsim_version": "2.9"},  # top-level metadata of interest
).identifier
```

Pro tip: Additionally, a `pack_by` argument in `client.bundles.upload` allows you to specify the order of files by which you want to pack files. 
This can be used to optimize the layout of files in the tarball: by storing jointly accessed files next to one another, read speeds can be improved!

## How to read a bundle?

Internally, Bundles are a resource like any other in vxData, simply pointing to paths on Ceph.
The SDK's `bundles` namespace does however provide helpful functionality to better work with bundles:

```python
from vxdata.schemas.response import BundleResponse

# load the Bundle pydantic object from vxData
bundle: BundleResponse = client.bundles.retrieve(bundle_id)

# download the entire bundle to disk
client.bundles.materialize(bundle, "./out")

# fetch the bundle's record table 
df_records = client.bundles.read_records(bundle)
# fetch the internal file & sharding table
df_manifest = client.bundles.read_manifest(bundle)

# use your metadata to materialize only parts of the bundle:
# by passing a dataframe into `paths`, only files mentioned in `path` / `path_*` columns will be materialized
client.bundles.materialize(bundle, "./out", paths=df_records.filter(pl.col("b_value") == 0))

# or pass explicit  lists of strings in:
client.bundles.materialize(bundle, "./out", paths=["sub-0/dwi.npy"])
```

## Internals: vxs3 allows for vxData-free access to bundles

The underlying package `vxs3` provides more direct ways to access the tarball on Ceph:
```python
from vxs3.bundle.io import Bundle, open_bundle

# via vxData: the manifest is pinned by the catalog entry
b: Bundle = client.bundles.open(bundle)
# or without vxData, straight from S3 (needs S3 credentials in the environment)
b = open_bundle(bundle.manifest_url, sha256=bundle.manifest_sha256)

b.list_entries("sub-0")                    # browse the manifest; no data I/O
files = b.materialize(["sub-0/dwi.npy"])   # {path: cached file}, byte ranges only
with b.open("sub-0/dwi.npy") as stream:    # read-only stream of one file
data = stream.read()
everything = b.download("./restored")      # full verified tree; dir must not exist
```

Source: https://docs.virdx.dev/vxdata-api/guides/bundles/index.mdx