Verified PESO v1.1 at Zenodo record 5137717: 102 H&E WSIs plus masks/annotations in 18 deposit objects (138.176 GB), with a machine-readable manifest, MD5 checksums, and CC-BY-NC-SA-4.0 licensing.
Enumerated the public LEOPARD AWS bucket s3://leopard-challenge: 508 WSIs, 508 tissue masks, and the recurrence/follow-up label table (2.142 TB total). Confirmed 0.242535 µm/px WSI resolution, matching masks, CC-BY-NC-SA terms, and the challenge’s additional publication embargo.
Added mono/apps/vxdata-jobs/src/vxdata/jobs/f_20260824_peso_leopard/copy_to_ceph.py. It writes deterministic keys below s3://vxdata/source-data/peso and s3://vxdata/source-data/leopard through the cephvx alias for https://s3.fra.virdx.dev.
PESO uses one resumable, MD5-verified local ZIP at a time and streams each member directly into Ceph. LEOPARD uses one resumable local object at a time and verifies either the single-part MD5 ETag or AWS 8 MiB multipart ETag before upload.
The job validates destination size and retained source-checksum metadata, retries uploads, locks out concurrent runs, keeps verified archives after failure, checks scratch capacity, and persists an append-only transfer manifest after every object to s3://vxdata/source-data/manifests/transfers.jsonl. Final prefix listings are written locally and uploaded as peso.jsonl and leopard.jsonl.
Initially started the transfer locally, then stopped it cleanly after 18 completed objects when the user needed to move. Copied the durable manifest to main.dev.felixknispel.coder, deployed the job there, and resumed it using that host’s ceph-vxdata alias. The operation is idempotent and restart-safe; existing objects are skipped only when size and source-checksum metadata both match.
Validation passed: Ruff format/lint, ty typecheck, Nx vxdata-jobs:test, and monorepo boundary checks.
Lessons Learned: Pitfalls
The vxData SDK storage upload path assigns timestamp/random-token prefixes and reads a local file into memory, so it is inappropriate for stable multi-gigabyte raw-source keys. The direct mc path already established by older vxData jobs is the correct fit here.
Cross-provider S3 CopyObject cannot copy AWS objects into Ceph server-side; a data-plane relay or bounded local staging is required.
Treating an initial “about 200 lines” preference as a hard cap degraded functionality and produced unnatural code. The user explicitly corrected this: rough LOC estimates must never override correctness, reliability, clarity, or conventional CLI design.
Size-only idempotency is insufficient. Existing objects must also carry the expected source checksum/version metadata before being skipped.
Lessons Learned: Improvements
Public-dataset transfer jobs should standardize deterministic source-data/<dataset> prefixes, bounded scratch storage, resumable source downloads, source checksum verification, destination metadata validation, concurrency locking, and durable manifests in Ceph.
Long-lived transfers should persist progress alongside the data after each object rather than relying only on ignored local logs or a final all-or-nothing listing.
LEOPARD’s AWS multipart ETags use 8 MiB source parts; verifying the multipart ETag provides source-integrity checking before the Ceph upload even though the ETag itself is not a simple whole-file MD5.
