---
title: "2026-08-24 PESO and LEOPARD Ceph transfer"
description: "Researched PESO and LEOPARD and started a resumable, integrity-checked copy into deterministic vxData Ceph prefixes."
image: "https://docs.virdx.dev/img/virdx-social-card.png"
---

> Documentation Index
> Fetch the complete documentation index at: https://docs.virdx.dev/llms.txt
> Use this file to discover all available pages before exploring further.

# 2026-08-24 PESO and LEOPARD Ceph transfer

## Work Done

- Verified PESO v1.1 at [Zenodo record 5137717](https://zenodo.org/records/5137717): 102 H&E WSIs plus masks/annotations in 18 deposit objects (138.176 GB), with a machine-readable manifest, MD5 checksums, and CC-BY-NC-SA-4.0 licensing.
- Enumerated the public LEOPARD AWS bucket `s3://leopard-challenge`: 508 WSIs, 508 tissue masks, and the recurrence/follow-up label table (2.142 TB total). Confirmed 0.242535 µm/px WSI resolution, matching masks, CC-BY-NC-SA terms, and the challenge's additional publication embargo.
- Added `mono/apps/vxdata-jobs/src/vxdata/jobs/f_20260824_peso_leopard/copy_to_ceph.py`. It writes deterministic keys below `s3://vxdata/source-data/peso` and `s3://vxdata/source-data/leopard` through the `cephvx` alias for `https://s3.fra.virdx.dev`.
- PESO uses one resumable, MD5-verified local ZIP at a time and streams each member directly into Ceph. LEOPARD uses one resumable local object at a time and verifies either the single-part MD5 ETag or AWS 8 MiB multipart ETag before upload.
- The job validates destination size and retained source-checksum metadata, retries uploads, locks out concurrent runs, keeps verified archives after failure, checks scratch capacity, and persists an append-only transfer manifest after every object to `s3://vxdata/source-data/manifests/transfers.jsonl`. Final prefix listings are written locally and uploaded as `peso.jsonl` and `leopard.jsonl`.
- Initially started the transfer locally, then stopped it cleanly after 18 completed objects when the user needed to move. Copied the durable manifest to `main.dev.felixknispel.coder`, deployed the job there, and resumed it using that host's `ceph-vxdata` alias. The operation is idempotent and restart-safe; existing objects are skipped only when size and source-checksum metadata both match.
- Validation passed: Ruff format/lint, ty typecheck, Nx `vxdata-jobs:test`, and monorepo boundary checks.

## Lessons Learned: Pitfalls

- The vxData SDK storage upload path assigns timestamp/random-token prefixes and reads a local file into memory, so it is inappropriate for stable multi-gigabyte raw-source keys. The direct `mc` path already established by older vxData jobs is the correct fit here.
- Cross-provider S3 `CopyObject` cannot copy AWS objects into Ceph server-side; a data-plane relay or bounded local staging is required.
- Treating an initial “about 200 lines” preference as a hard cap degraded functionality and produced unnatural code. The user explicitly corrected this: rough LOC estimates must never override correctness, reliability, clarity, or conventional CLI design.
- Size-only idempotency is insufficient. Existing objects must also carry the expected source checksum/version metadata before being skipped.

## Lessons Learned: Improvements

- Public-dataset transfer jobs should standardize deterministic `source-data/<dataset>` prefixes, bounded scratch storage, resumable source downloads, source checksum verification, destination metadata validation, concurrency locking, and durable manifests in Ceph.
- Long-lived transfers should persist progress alongside the data after each object rather than relying only on ignored local logs or a final all-or-nothing listing.
- LEOPARD's AWS multipart ETags use 8 MiB source parts; verifying the multipart ETag provides source-integrity checking before the Ceph upload even though the ETag itself is not a simple whole-file MD5.

Source: https://docs.virdx.dev/knowledge/inbox/2026-08-24-histo-peso-leopard-ceph-transfer/index.mdx