---
title: "2026-08-31 clinical ve2e PI CAI benchmark audit"
description: "Audited vxData cohort preparation, model preprocessing, and expert-only PI-CAI evaluation after inference helper deletion."
image: "https://docs.virdx.dev/img/virdx-social-card.png"
---

> Documentation Index
> Fetch the complete documentation index at: https://docs.virdx.dev/llms.txt
> Use this file to discover all available pages before exploring further.

# 2026-08-31 clinical ve2e PI CAI benchmark audit

## Work Done

- Audited `scripts/benchmarking/benchmark_isup.py`, the vxData standard pipeline, the clinical dataset/datamodule, minimal PI-CAI inference, and PI-CAI metric utilities.
- Identified the expert-safe lesion-mask sources (`cspca`, `isup`, and genuine-negative `empty`) and the need to reject `pca`, `vx1`, `viseg`, and `viseg-lesion` for spatial ground truth.
- Derived a custom benchmark pipeline that selects the cohort without pathology/mask ranking, generates only prostate/anatomy support masks, disables pseudo-lesion inference, converts expert masks to the 3-class target, and materializes all input paths.
- Derived direct `VxProsDataModule` construction from the checkpoint's frozen preprocessing config after deletion of `inference/classification.py`.
- Identified `detection_maps_from_seg`, `scale_down_secondary_lesions`, `compute_picai_metrics`, and expert/source gating helpers as the existing metric path to reuse.

## Lessons Learned: Pitfalls

- `build_training_table` excludes the `benchmark` split, so it cannot silently serve a true all-non-training benchmark cohort without a benchmark-specific output path.
- `select_primary_study` ranks by lesion-mask availability and ISUP. That leaks ground-truth properties into cohort selection.
- `standard_expert_masks_only` disables pseudo inference but still accepts `pca` masks through `ensure_cspca_masks`; `pca` is not valid csPCa spatial ground truth.
- The mandatory clinical workstream KB index and SOP index do not exist, so the start-of-session lookup supplied no project guidance.

## Lessons Learned: Improvements

- Add a benchmark-specific vxData SOP that states cohort split policy, allowed mask provenance, and the prohibition on pathology/mask-ranked study selection.
- Add a reusable inference datamodule builder owned by the data layer, or expose checkpoint preprocessing config through the PI-CAI predictor, so benchmark scripts do not recreate deleted classification helpers.
- Add a benchmark table builder that preserves selected non-training splits and emits only required columns without identifiers in public results.

Source: https://docs.virdx.dev/knowledge/inbox/2026-08-31-clinical-ve2e-picai-benchmark-audit/index.mdx