Rebuilt feat/isup-inference from origin/merge-unify-vits-into-cleanup-vits as a focused implementation using ve2e.vxdata_pipeline, typed cleanup configs, unified DinoFlex forward processing, PHI-safe logging, canonical split preservation, separate biopsy/RPE QWK, and mode-0600 JSON output.
Dropped the obsolete data-pipeline consolidation and Pixi/Docker compatibility commits because cleanup already supplies those facilities.
Pushed focused commit a9636cef and built image feat-isup-inference-a9636ce with Tekton build build-ve2e-k6stx.
Confirmed ClearML model 56d3c7efa29847eba0b250983c2c7abb initially failed under cleanup because its embedded config references removed ve2e.base_configs classes.
Added the explicitly temporary, volume-DinoFlex-only adapter src/ve2e/legacy_checkpoint.py. Native checkpoints remain on the normal load path; fallback occurs only for the removed module. Pushed commit 5205d25c and built image feat-isup-inference-5205d25 with Tekton build build-ve2e-vqqc2.
Validated full deserialization, typed TrainConfig conversion, current DinoFlex reconstruction, EMA state-dict loading, and CUDA placement in Argo workflow run-ve2e-legacy-model-check-f5gz8 on standard rtx-6000. The workflow succeeded.
The first full-cohort attempt exposed a cohort-semantics bug: the exact-one-study guard rejected the whole cohort when any patient was ineligible. Replaced it with unbiased eligibility filtering in commit 66f29d58 and built image feat-isup-inference-66f29d5 with Tekton build build-ve2e-577h8.
Full non-training evaluation succeeded in Argo workflow run-ve2e-isup-full-eligible-bdf7p: 640 eligible patients, biopsy QWK 0.1523730126351761, and RPE QWK 0.08081882876638757.
Verified the result artifact was valid JSON with mode 0600, retrieved it privately to ~/Downloads/ve2e-isup-full-results.json, and deleted the temporary reader pod. No PR was created.
Lessons Learned: Pitfalls
A checkpoint being self-contained does not imply compatibility after config module deletion: Python-pickled config class paths remain runtime dependencies.
The first cleanup image and model implementation were sound, but validation could not pass without either a cleanup-native model or a narrow translation boundary.
Zot image pulls can incur transient TLS handshake timeouts; a later pull may recover without rebuilding.
Raw cleanup did not contain unified DinoFlex. Work must target the combined cleanup-plus-unify tree until the merge lands.
A strict exact-one-study assertion is appropriate for a hand-picked request but not a broad split query. A cohort workflow must filter to patients with exactly one eligible study without using outcome or mask information for ranking.
Lessons Learned: Improvements
Prefer storing checkpoint configs as plain versioned dictionaries rather than pickled config class instances. Include an explicit schema version and migration function.
Keep checkpoint compatibility adapters in one named module with a narrowly guarded fallback. This makes later deletion mechanical.
Produce at least one cleanup-native six-class GG EMA model before removing the adapter. Then delete src/ve2e/legacy_checkpoint.py and the fallback in src/ve2e/isup.py.
Document the cleanup schema mapping for flat legacy DinoFlex fields to encoder, tasks, trainer, clearml, and checkpointing groups.
Add PHI-safe aggregate forward-pass progress. The 640-patient run spent about 81 minutes in inference with no batch counter, which made monitoring less precise than pipeline preprocessing.
