---
title: "2026-09-14 Public pathology study link proposals"
description: "Read-only PICAI and PRUS source-to-specimen linkage checks and a dry-run repair script."
image: "https://docs.virdx.dev/img/virdx-social-card.png"
---

> Documentation Index
> Fetch the complete documentation index at: https://docs.virdx.dev/llms.txt
> Use this file to discover all available pages before exploring further.

# 2026-09-14 Public pathology study link proposals

## Work Done

Added an uncommitted standalone script in mono at `apps/vxdata-jobs/src/vxdata/jobs/link_public_pathology_studies.py`. It reads the source tables under `/mnt/storage/data/mri/public_datasets`, queries vxData, and prints proposed root-specimen moves. Writes require `--apply`; no data was changed during this investigation.

- PICAI source: `picai/picai_original_labels/clinical_information/marksheet.csv`. Match patient external UID plus study external UID. All 1,001 pathology event roots map. Patient-only matching is unsafe: some patients have multiple studies.
- PRUS source: `prostate_mri_us_biopsy/TCIA-Biopsy-Data_2020-07-14.xlsx`. Link `Series Instance UID (MRI)` through DICOMSeries to ImagingStudy. Within each patient, source groups match existing specimens by core count and ordered core labels. Read-only dry run proposes 1,015 moves; 353 groups lack MRI UID and two have ambiguous core-group matches, so 355 roots stay untouched.
- Remote native environment: `/home/felixknispel/work/mono/apps/vxdata-jobs/.pixi/envs/default/bin/python` on `main.dev.felixknispel.coder`. Ruff passed. Workspace Nx checks could not run because this local worktree lacks installed Nx modules.

Follow-up: the archived data-platform importer at `0ab5da31174cf9533d826345f0cd0bc5b13e0898` grouped all rows without sorting and assigned session numbers in Polars group iteration order. It did not retain the MRI UID on the root specimen. Sorting was added later (`f8e1b9c5e876c32d02049ac548594767a4283c60`), then null-MRI rows were excluded. The 1,015 core-count/label matches are preliminary, not verified source identities. The script now rejects PRUS `--apply`; stronger source-attribute validation or original execution artefacts are needed first.

A direct live comparison found zero matches between the 1,017 source patient/MRI-series UID pairs and `ImagingStudy.external_uid`; a series-to-study translation is needed separately from recovering specimen identity. The [official TCIA documentation](https://www.cancerimagingarchive.net/collection/prostate-mri-us-biopsy/) confirms that the biopsy table supplies series UIDs, blank MRI UIDs indicate irretrievable data or no MRI registration, and multiple ultrasound biopsies can reference one MRI series.

Final implementation: draft [mono PR #393](https://github.com/virdx/mono/pull/393),
commit `8db63e53`, replaces the provisional script with three modules under
`apps/vxdata-jobs/src/vxdata/jobs/link_pathology_studies`: `prus`, `picai`, and
`apply`. PRUS now compares the full ordered core attributes plus summaries and
requires a one-to-one match for every group. All 1,370 groups match exactly,
including 24,782 stored child cores; both earlier label ambiguities are resolved.
One single-core group has only summary-level evidence. Volume sidecars (no
DICOMSeries) resolve 1,009 links; 353 lack source MRI UIDs and eight lack matching
volumes. PICAI yields 1,001 links. Combined apply dry-run validates 2,010 moves.
No updates were applied. The old PRUS apply guard applied only to the discarded
provisional script; the new separate updater requires explicit `--apply` after
mapping review and full live root/patient/study validation.

Generated `mapping-prus.json` and `mapping-picai.json` are explicitly excluded
from the PR and ignored in the jobs directory, per user instruction. Validation
artifacts remain under `/tmp` on the SSH host, not in Git. Native full job tests:
28 passed; Ruff and staged source/test type checks passed. Initial test-fixture
type errors were fixed before commit. Local Nx orchestration remains unavailable
because node_modules is missing; native Pixi checks succeeded.

User review-scope correction: removed all three added test files from draft PR
#393 and condensed the README to 19 bullet-point lines (maximum requested: 30).
Follow-up commit `1223ec14`; mapping code and generated results are unchanged.
Rechecked TCIA: MRI UIDs are documented for approximately 70% of the dataset;
blank references mean irretrievable data or no MRI registration. The 353 groups
without MRI references contain 6,607/24,783 core rows (26.7%). The eight absent
volume matches are a separate platform coverage gap, not explained by that
statement. Confirmed the one summary-only singleton is NOT in the 1,009 proposed
updates; every proposed PRUS update has full stored core-detail evidence.

Staging application (explicitly authorized by the user): ran the new updater with
`API_URL=https://staging.fra.virdx.dev/vxdata` and the two mappings in
`/tmp/pathology-staging.QKggCM/`. Applied all 2,010 root-parent updates (1,009 PRUS,
1,001 PICAI). Follow-up dry-run returned zero updates and 2,010 already linked.
Verified every target root parent and inherited study_id across 20,802 specimens
and 20,802 assessments. The 361 omitted PRUS roots were not included. Production
was not modified. PR #393 description records this staging-only application.

Production application (subsequently explicitly authorized): regenerated both
mappings against `https://data.fra.virdx.dev/` (1,009 PRUS + 1,001 PICAI). Saved
2,010 prior parent IDs in `rollback-parent-identifiers.json`. Forward maps,
rollback map, and checksum manifest were copied to the user's Mac and verified
before applying. Durable directories: remote
`/home/felixknispel/vxdata-backups/pathology-prod-20260914T144304Z`, local
`/Users/felix.knispel/vxdata-backups/pathology-prod-20260914T144304Z`.
Applied 2,010 production updates. Post-write verification passed for all roots
and inherited study_id across 20,802 specimens + 20,802 assessments; zero updates
remain. Verification JSON is retained with backups. PR #393 description updated;
no source-code changes or generated mappings committed. The 361 omitted PRUS
groups remain excluded (353 without source MRI UID, eight without volumes).

## Pitfalls

Current PRUS importer code does not reproduce live session numbering. Null-first and null-last numbering both mismatch stored core groups. Never repair by session ordinal alone. The source has 6,607 rows without MRI series UID; these are present in live pathology despite the current importer filtering them out.

Clinical workstream index/SOP index were absent. Infrastructure vxData SOPs were useful, but SDK version and older method examples there lag current code; consult the SDK README.

## Improvements

For future repairs, inspect source linkage and actual stored core groups before relying on importer code. Keep missing MRI links and ambiguous groups explicit; do not infer a study from patient identity alone. Follow-up source evidence is needed for the 355 unresolved PRUS groups.

Source: https://docs.virdx.dev/knowledge/inbox/2026-09-14-infrastructure-public-pathology-study-links/index.mdx