Read the vxData jobs app instructions, current ingestion SOPs, and recent dataset integrations including Frankfurt, CHIMERA, MamaMIA, ODELIA, and BRACS.
Traced Client.storage.upload through the SDK and API presign implementation, confirming timestamp-plus-random-token object prefixes and shared-prefix directory behavior.
Catalogued existing direct object-storage patterns using mc mirror, Ceph/MinIO environment configuration, and Kubernetes Secrets.
Recommended resumable, checksum-validated local acquisition followed by direct mc mirror for stable PESO and LEOPARD raw-source trees; no monorepo files were edited.
Lessons Learned: Pitfalls
Legacy raw-ingestion files disagree on bucket layout (vxdata/source-data/... versus a source-data bucket), so their path literals must not be copied without confirming the current production contract.
Passing a dictionary of files to SDK upload preserves the Python shape but does not share an object prefix; each file receives a separate timestamp/random prefix. Only directory upload shares a prefix.
Lessons Learned: Improvements
The vxData ingestion SOP should document exact current SDK key allocation, the single-segment group restriction, and when bulk mc mirror is preferable to presigned per-file upload.
A canonical production raw-source bucket/key convention would prevent new jobs from copying stale, contradictory examples.
