---
title: "2026-09-17 Gleason S3 training validation"
description: "Enabled S3-backed Gleason image reads, repaired legacy map resolution, and passed bounded CPU integration."
image: "https://docs.virdx.dev/img/virdx-social-card.png"
---

> Documentation Index
> Fetch the complete documentation index at: https://docs.virdx.dev/llms.txt
> Use this file to discover all available pages before exploring further.

# 2026-09-17 Gleason S3 training validation

## Work Done

- Released `vxdata-sdk 4.3.0` through mono PR #403 with a plain-fsspec `Client().storage.open_remote()` implementation for seekable, bounded-cache remote reads. Histo now depends on `vxdata-sdk >=4.3,<5`.
- Extended virdx-ome TIFF metadata support and used it in `scripts/gleason_training/generate_artifact.py` so artifact generation can inspect S3 TIFF pyramids without downloading full slides.
- Pushed the Gleason workflow work to Histo PR #339 on `fk-gleason-primary-workflow`. The final validated Histo commit was `e0cca73a` (`fix: resolve legacy Gleason map paths`).
- Built the final image as `build-histo-h8zpj`, digest `sha256:377ff844d4574b874a3a7aef6367127374bb01104f0e443e48aa2213b54ece03`.
- Generated the bounded 10-row AGGC/PANDA artifact `artefact/gleason-primary-smoke-7dbaab3` using five scans per dataset and the 10x target.
- Diagnosed vxData materialization end to end. The platform map URLs looked like S3 Zarr prefixes, but all 34 prefixes were empty. The corresponding maps remained on shared storage: 11 under `/mnt/artifacts/data/...` and 23 under `/mnt/artifacts/...`; none existed under `/mnt/storage/data/...`.
- Added a read-only `/mnt/artifacts` training mount and a contained compatibility redirect that applies only to map columns. Image TIFFs remain S3-backed and are materialized through vxData.
- Passed CPU integration in workflow `gleason-cpu-128gi-e0cca73-tstrb`: HF access, ClearML upload/download, real Hydra composition, materialization and split (10 rows; 7 train, 3 validation), data-module setup (1,100 train patches), and real train/validation batches (`[1,3,448,448]` images and `[1,1,448,448]` labels). No GPU was used and the model was not constructed.
- GPU checks were intentionally deferred until CPU integration passed. Earlier attempts could not schedule because all Ada GPUs were reserved. No full training was launched.
- Opened infra_k8s PR #228 to inject the optional personal HF token into shared Argo `run-image` CPU/GPU main containers, following the existing template convention. The token value was never inspected or written locally.
- Addressed Histo PR #345 review comments and pushed `42414655`, inlining GrandQC default weight resolution and removing the redundant resolver API, test, and README section.

## Lessons Learned: Pitfalls

- Treating every legacy path as if it had been copied to `s3://vxdata/nfs-copy/` was incorrect. TIFF image objects existed, but Zarr map directories did not. Appending `/` fixed prefix semantics but only changed an HTTP 500 into an empty successful listing.
- Artifact regeneration could not restore original map paths because the platform database already returned the incomplete S3 mirror URLs.
- A successful prefix materialization call does not prove data exists: the SDK returned the expected local root for an empty listing. Validate returned path existence before downstream image opening when diagnosing migrations.
- Large cache requests obscured the first failure. The 128 GiB pod was sufficient; the error was control-plane/path resolution, not capacity.
- Diagnostic errors can contain paths or signed URLs. Emit fixed stage names, exception classes, aggregate counts, and sanitized structural details instead of raw exceptions or pod objects.

## Lessons Learned: Improvements

- Document the legacy storage split explicitly: remote TIFF images are available in the vxData S3 mirror, while current Gleason maps can still require `/mnt/artifacts`, with both `/mnt/artifacts/data/<key>` and `/mnt/artifacts/<key>` layouts.
- Add migration completeness checks before rewriting database URLs. For directory resources, verify a non-empty recursive listing rather than only accepting a syntactically valid S3 prefix.
- Make vxData materialization fail clearly when a requested prefix lists zero objects instead of returning a nonexistent local root.
- Keep the final GPU gate small: reuse the validated artifact and final image, run two batches with a hard deadline, and verify model forward/backward plus checkpoint creation and ClearML upload before declaring PR #339 ready.

Source: https://docs.virdx.dev/knowledge/inbox/2026-09-17-histo-gleason-s3-training-validation/index.mdx