Skip to content

2026-09-01 inverse ve2e PI CAI data architecture investigation

Traced the PI-CAI inference data path and identified an immutable benchmark-artifact route that preserves canonical preprocessing.

Work Done

  • Performed a read-only architecture investigation of virdx/ve2e branch feat/isup-inference.
  • Traced src/ve2e/inference_api/picai/data.py through its 24-step vxData pipeline, materialization, Lightning data module, clinical dataset, and canonical resample/crop/pad/z-score preprocessing path.
  • Searched source, scripts, tests, and configs for inference, artifact loading, dataset dumping, materialization, and preprocessing seams.
  • Found that model input preparation, benchmark cohort selection, pathology target construction, expert-mask selection, split routing, and training-table construction are mixed in one 254-line module.
  • Recommended keeping PICAIInference array-only, moving benchmark cohort and target creation out of the general inference API, freezing native-data benchmark manifests as immutable vxData artifacts, and loading them through the existing materialization and clinical dataset preprocessing path.
  • Identified the current shared canonical preprocessing seam in clinical_dataset._preprocess_vols, which is already used by both eager and on-demand clinical dataset loading.
  • Made no changes to the ve2e checkout.

Lessons Learned: Pitfalls

  • The initial workstream inference was too broad. The repository README calls ve2e “the inverse model”, so the inverse workstream index should be read before product or infrastructure context.
  • Repository-wide searches can be noisy because this branch includes a large restructuring. Narrow symbol and AST-based traces were more useful than branch-wide diffs.
  • The current inference-data module is not covered by tests and the sole benchmark caller passes the sample-loader arguments in the opposite positional order. Static API tracing is therefore important before assuming the branch path has run successfully.

Lessons Learned: Improvements

  • The inverse workstream would benefit from a short SOP that records the canonical native-MRI preprocessing boundary and distinguishes immutable training/benchmark artifacts from online inference inputs.
  • A documented benchmark-manifest contract should define cohort immutability, expert-mask provenance, target provenance, and allowed split semantics without overloading the training table builder.
  • The public inference API documentation should state that PICAIInference consumes model-ready arrays only and that data acquisition and benchmark targets belong to separate adapters.
Navigation

Type to search…

↑↓ navigate↵ selectEsc close