Audited scripts/benchmarking/benchmark_isup.py, the vxData standard pipeline, the clinical dataset/datamodule, minimal PI-CAI inference, and PI-CAI metric utilities.
Identified the expert-safe lesion-mask sources (cspca, isup, and genuine-negative empty) and the need to reject pca, vx1, viseg, and viseg-lesion for spatial ground truth.
Derived a custom benchmark pipeline that selects the cohort without pathology/mask ranking, generates only prostate/anatomy support masks, disables pseudo-lesion inference, converts expert masks to the 3-class target, and materializes all input paths.
Derived direct VxProsDataModule construction from the checkpoint’s frozen preprocessing config after deletion of inference/classification.py.
Identified detection_maps_from_seg, scale_down_secondary_lesions, compute_picai_metrics, and expert/source gating helpers as the existing metric path to reuse.
Lessons Learned: Pitfalls
build_training_table excludes the benchmark split, so it cannot silently serve a true all-non-training benchmark cohort without a benchmark-specific output path.
select_primary_study ranks by lesion-mask availability and ISUP. That leaks ground-truth properties into cohort selection.
standard_expert_masks_only disables pseudo inference but still accepts pca masks through ensure_cspca_masks; pca is not valid csPCa spatial ground truth.
The mandatory clinical workstream KB index and SOP index do not exist, so the start-of-session lookup supplied no project guidance.
Lessons Learned: Improvements
Add a benchmark-specific vxData SOP that states cohort split policy, allowed mask provenance, and the prohibition on pathology/mask-ranked study selection.
Add a reusable inference datamodule builder owned by the data layer, or expose checkpoint preprocessing config through the PI-CAI predictor, so benchmark scripts do not recreate deleted classification helpers.
Add a benchmark table builder that preserves selected non-training splits and emits only required columns without identifiers in public results.
