Work Done
Added an uncommitted standalone script in mono at apps/vxdata-jobs/src/vxdata/jobs/link_public_pathology_studies.py. It reads the source tables under /mnt/storage/data/mri/public_datasets, queries vxData, and prints proposed root-specimen moves. Writes require --apply; no data was changed during this investigation.
- PICAI source:
picai/picai_original_labels/clinical_information/marksheet.csv. Match patient external UID plus study external UID. All 1,001 pathology event roots map. Patient-only matching is unsafe: some patients have multiple studies. - PRUS source:
prostate_mri_us_biopsy/TCIA-Biopsy-Data_2020-07-14.xlsx. LinkSeries Instance UID (MRI)through DICOMSeries to ImagingStudy. Within each patient, source groups match existing specimens by core count and ordered core labels. Read-only dry run proposes 1,015 moves; 353 groups lack MRI UID and two have ambiguous core-group matches, so 355 roots stay untouched. - Remote native environment:
/home/felixknispel/work/mono/apps/vxdata-jobs/.pixi/envs/default/bin/pythononmain.dev.felixknispel.coder. Ruff passed. Workspace Nx checks could not run because this local worktree lacks installed Nx modules.
Follow-up: the archived data-platform importer at 0ab5da31174cf9533d826345f0cd0bc5b13e0898 grouped all rows without sorting and assigned session numbers in Polars group iteration order. It did not retain the MRI UID on the root specimen. Sorting was added later (f8e1b9c5e876c32d02049ac548594767a4283c60), then null-MRI rows were excluded. The 1,015 core-count/label matches are preliminary, not verified source identities. The script now rejects PRUS --apply; stronger source-attribute validation or original execution artefacts are needed first.
A direct live comparison found zero matches between the 1,017 source patient/MRI-series UID pairs and ImagingStudy.external_uid; a series-to-study translation is needed separately from recovering specimen identity. The official TCIA documentation confirms that the biopsy table supplies series UIDs, blank MRI UIDs indicate irretrievable data or no MRI registration, and multiple ultrasound biopsies can reference one MRI series.
Final implementation: draft mono PR #393,
commit 8db63e53, replaces the provisional script with three modules under
apps/vxdata-jobs/src/vxdata/jobs/link_pathology_studies: prus, picai, and
apply. PRUS now compares the full ordered core attributes plus summaries and
requires a one-to-one match for every group. All 1,370 groups match exactly,
including 24,782 stored child cores; both earlier label ambiguities are resolved.
One single-core group has only summary-level evidence. Volume sidecars (no
DICOMSeries) resolve 1,009 links; 353 lack source MRI UIDs and eight lack matching
volumes. PICAI yields 1,001 links. Combined apply dry-run validates 2,010 moves.
No updates were applied. The old PRUS apply guard applied only to the discarded
provisional script; the new separate updater requires explicit --apply after
mapping review and full live root/patient/study validation.
Generated mapping-prus.json and mapping-picai.json are explicitly excluded
from the PR and ignored in the jobs directory, per user instruction. Validation
artifacts remain under /tmp on the SSH host, not in Git. Native full job tests:
28 passed; Ruff and staged source/test type checks passed. Initial test-fixture
type errors were fixed before commit. Local Nx orchestration remains unavailable
because node_modules is missing; native Pixi checks succeeded.
User review-scope correction: removed all three added test files from draft PR
#393 and condensed the README to 19 bullet-point lines (maximum requested: 30).
Follow-up commit 1223ec14; mapping code and generated results are unchanged.
Rechecked TCIA: MRI UIDs are documented for approximately 70% of the dataset;
blank references mean irretrievable data or no MRI registration. The 353 groups
without MRI references contain 6,607/24,783 core rows (26.7%). The eight absent
volume matches are a separate platform coverage gap, not explained by that
statement. Confirmed the one summary-only singleton is NOT in the 1,009 proposed
updates; every proposed PRUS update has full stored core-detail evidence.
Staging application (explicitly authorized by the user): ran the new updater with
API_URL=https://staging.fra.virdx.dev/vxdata and the two mappings in
/tmp/pathology-staging.QKggCM/. Applied all 2,010 root-parent updates (1,009 PRUS,
1,001 PICAI). Follow-up dry-run returned zero updates and 2,010 already linked.
Verified every target root parent and inherited study_id across 20,802 specimens
and 20,802 assessments. The 361 omitted PRUS roots were not included. Production
was not modified. PR #393 description records this staging-only application.
Production application (subsequently explicitly authorized): regenerated both
mappings against https://data.fra.virdx.dev/ (1,009 PRUS + 1,001 PICAI). Saved
2,010 prior parent IDs in rollback-parent-identifiers.json. Forward maps,
rollback map, and checksum manifest were copied to the user’s Mac and verified
before applying. Durable directories: remote
/home/felixknispel/vxdata-backups/pathology-prod-20260914T144304Z, local
/Users/felix.knispel/vxdata-backups/pathology-prod-20260914T144304Z.
Applied 2,010 production updates. Post-write verification passed for all roots
and inherited study_id across 20,802 specimens + 20,802 assessments; zero updates
remain. Verification JSON is retained with backups. PR #393 description updated;
no source-code changes or generated mappings committed. The 361 omitted PRUS
groups remain excluded (353 without source MRI UID, eight without volumes).
Pitfalls
Current PRUS importer code does not reproduce live session numbering. Null-first and null-last numbering both mismatch stored core groups. Never repair by session ordinal alone. The source has 6,607 rows without MRI series UID; these are present in live pathology despite the current importer filtering them out.
Clinical workstream index/SOP index were absent. Infrastructure vxData SOPs were useful, but SDK version and older method examples there lag current code; consult the SDK README.
Improvements
For future repairs, inspect source linkage and actual stored core groups before relying on importer code. Keep missing MRI links and ambiguous groups explicit; do not infer a study from patient identity alone. Follow-up source evidence is needed for the 355 unresolved PRUS groups.