Released vxdata-sdk 4.3.0 through mono PR #403 with a plain-fsspec Client().storage.open_remote() implementation for seekable, bounded-cache remote reads. Histo now depends on vxdata-sdk >=4.3,<5.
Extended virdx-ome TIFF metadata support and used it in scripts/gleason_training/generate_artifact.py so artifact generation can inspect S3 TIFF pyramids without downloading full slides.
Pushed the Gleason workflow work to Histo PR #339 on fk-gleason-primary-workflow. The final validated Histo commit was e0cca73a (fix: resolve legacy Gleason map paths).
Built the final image as build-histo-h8zpj, digest sha256:377ff844d4574b874a3a7aef6367127374bb01104f0e443e48aa2213b54ece03.
Generated the bounded 10-row AGGC/PANDA artifact artefact/gleason-primary-smoke-7dbaab3 using five scans per dataset and the 10x target.
Diagnosed vxData materialization end to end. The platform map URLs looked like S3 Zarr prefixes, but all 34 prefixes were empty. The corresponding maps remained on shared storage: 11 under /mnt/artifacts/data/... and 23 under /mnt/artifacts/...; none existed under /mnt/storage/data/....
Added a read-only /mnt/artifacts training mount and a contained compatibility redirect that applies only to map columns. Image TIFFs remain S3-backed and are materialized through vxData.
Passed CPU integration in workflow gleason-cpu-128gi-e0cca73-tstrb: HF access, ClearML upload/download, real Hydra composition, materialization and split (10 rows; 7 train, 3 validation), data-module setup (1,100 train patches), and real train/validation batches ([1,3,448,448] images and [1,1,448,448] labels). No GPU was used and the model was not constructed.
GPU checks were intentionally deferred until CPU integration passed. Earlier attempts could not schedule because all Ada GPUs were reserved. No full training was launched.
Opened infra_k8s PR #228 to inject the optional personal HF token into shared Argo run-image CPU/GPU main containers, following the existing template convention. The token value was never inspected or written locally.
Addressed Histo PR #345 review comments and pushed 42414655, inlining GrandQC default weight resolution and removing the redundant resolver API, test, and README section.
Lessons Learned: Pitfalls
Treating every legacy path as if it had been copied to s3://vxdata/nfs-copy/ was incorrect. TIFF image objects existed, but Zarr map directories did not. Appending / fixed prefix semantics but only changed an HTTP 500 into an empty successful listing.
Artifact regeneration could not restore original map paths because the platform database already returned the incomplete S3 mirror URLs.
A successful prefix materialization call does not prove data exists: the SDK returned the expected local root for an empty listing. Validate returned path existence before downstream image opening when diagnosing migrations.
Large cache requests obscured the first failure. The 128 GiB pod was sufficient; the error was control-plane/path resolution, not capacity.
Diagnostic errors can contain paths or signed URLs. Emit fixed stage names, exception classes, aggregate counts, and sanitized structural details instead of raw exceptions or pod objects.
Lessons Learned: Improvements
Document the legacy storage split explicitly: remote TIFF images are available in the vxData S3 mirror, while current Gleason maps can still require /mnt/artifacts, with both /mnt/artifacts/data/<key> and /mnt/artifacts/<key> layouts.
Add migration completeness checks before rewriting database URLs. For directory resources, verify a non-empty recursive listing rather than only accepting a syntactically valid S3 prefix.
Make vxData materialization fail clearly when a requested prefix lists zero objects instead of returning a nonexistent local root.
Keep the final GPU gate small: reuse the validated artifact and final image, run two batches with a hard deadline, and verify model forward/backward plus checkpoint creation and ClearML upload before declaring PR #339 ready.
