---
title: "2026-08-24 vxData storage pattern inspection"
description: "Inspected recent vxData dataset jobs and storage upload behavior to recommend a minimal PESO and LEOPARD transfer job."
image: "https://docs.virdx.dev/img/virdx-social-card.png"
---

> Documentation Index
> Fetch the complete documentation index at: https://docs.virdx.dev/llms.txt
> Use this file to discover all available pages before exploring further.

# 2026-08-24 vxData storage pattern inspection

## Work Done

- Read the vxData jobs app instructions, current ingestion SOPs, and recent dataset integrations including Frankfurt, CHIMERA, MamaMIA, ODELIA, and BRACS.
- Traced `Client.storage.upload` through the SDK and API presign implementation, confirming timestamp-plus-random-token object prefixes and shared-prefix directory behavior.
- Catalogued existing direct object-storage patterns using `mc mirror`, Ceph/MinIO environment configuration, and Kubernetes Secrets.
- Recommended resumable, checksum-validated local acquisition followed by direct `mc mirror` for stable PESO and LEOPARD raw-source trees; no monorepo files were edited.

## Lessons Learned: Pitfalls

- Legacy raw-ingestion files disagree on bucket layout (`vxdata/source-data/...` versus a `source-data` bucket), so their path literals must not be copied without confirming the current production contract.
- Passing a dictionary of files to SDK upload preserves the Python shape but does not share an object prefix; each file receives a separate timestamp/random prefix. Only directory upload shares a prefix.

## Lessons Learned: Improvements

- The vxData ingestion SOP should document exact current SDK key allocation, the single-segment `group` restriction, and when bulk `mc mirror` is preferable to presigned per-file upload.
- A canonical production raw-source bucket/key convention would prevent new jobs from copying stale, contradictory examples.

Source: https://docs.virdx.dev/knowledge/inbox/2026-08-24-infrastructure-vxdata-storage-pattern-inspection/index.mdx