PHI-safe data handling in vxData

What deliberately holds PHI

DICOMFile payloads store direct DICOM identifiers and PHI-adjacent fields in mono/packages/vxdata-schemas/src/vxdata/schemas/payloads.py: PatientID, PatientName, PatientBirthDate, AccessionNumber, ReferringPhysicianName, InstitutionName, plus raw file_metadata dict. These fields exist by design so downstream processing jobs can correlate studies, but they leak PHI when queried or profiled naively.

Dataset anonymization status is encoded in the license field of each datasource definition in mono/apps/vxdata-jobs/src/vxdata/jobs/source_2026_01_28_ingest_files/dataset_definition.py. Example values: "VIRDX-ANONYMIZED", "VIRDX-PSEUDONYMIZED", "CC-BY-4.0". This field documents the source dataset’s anonymization state but provides no runtime enforcement.

Architectural gaps (as of 2026-08)

• No central PHI-safe projection/redaction layer — there is no allowlist-based view or API wrapper that strips PHI fields from query results.
• No k-anonymity or small-cell suppression guardrails — aggregate queries can return counts ≤ 5 or unique-identifier distributions with no warning or blocking.
• No bounded generic dataset profiler in the repo — no utility that profiles schema/nulls/cardinality while refusing to output values or hashed identifiers.
• No active anonymization pipeline — dicom-anonymizer is not present in the current mono dependency tree; no local pipeline uses it (verified 2026-08).

These gaps mean any ad-hoc profiling, export, or aggregate query can leak PHI unless the operator manually restricts output to PHI-safe summaries.

PHI-safe profiling recipe

Safe for sharing with external agents or in public tickets:

1. Schema inspection: column names, dtypes, nullability → use /schemas endpoint (mono/apps/vxdata-api/src/vxdata/api/routes/meta.py) or parquet preview (/parquet at mono/apps/vxdata-api/src/vxdata/api/routes/storage.py), both return schema + row counts.
2. Null rates: COUNT(*) vs COUNT(column) → value-free.
3. Distinct counts: COUNT(DISTINCT column) or Polars .n_unique() → numeric summary, no literal values.
4. Candidate keys: columns with n_unique == n_rows or high uniqueness ratios → structural, no values.
5. String-shape summaries: regex-based patterns like A-9 (letter-digit) or YYYY-MM-DD, counts-of-counts only, never literal strings.
6. Relationship overlap: FK-like in-memory joins to compute overlap counts/ratios between columns → never output the actual keys.

Never export:

• Sample rows, literal values, or hashed values.
• Frequency distributions that include the literal value (ok: “342 values appear once”, not ok: “PatientID=12345 appears 3 times”).
• Small-cell counts (e.g., “2 patients in this subgroup”) without manual review.

Reference profiler script (local dev artifact)

A single-file PHI-safe dataset profiler exists at ~/work/tries/2026-04-01-data-integrator/vxdata_dataset_profiler.py (verified 2026-08). It uses:

• uv script metadata (PEP 723) with Polars + fastexcel.
• --input <dir/file> --output <dir> CLI.
• Outputs: manifest.json, tables/*.json (per-table schema/nulls/distinct counts/candidate keys), relationships.json (FK overlap metrics, no values), vxdata_payload_catalog.json, vxdata_mapping_suggestions.json, integration_handoff.json, SUMMARY.md.
• Safety invariant: never writes sample rows, literal values, hashed values, or frequency distributions with identifiers — only counts-of-counts and structural signatures.

This script lives in a personal ~/work/tries directory and is not in any virdx repo. Reference the technique (schema + nulls + distinct counts + string shapes + candidate keys + relationship overlap), not the path, when describing PHI-safe profiling to others.

API utilities that already exist (mono, 2026-08)

• mono/apps/vxdata-api/src/vxdata/api/routes/meta.py:
  • /datasources/detailed-stats — datasource-level aggregate stats: counts, distributions, date ranges, scanner metadata.
  • /schemas — field definitions and active resource count for every payload type.
• mono/apps/vxdata-api/src/vxdata/api/routes/storage.py:
  • /parquet — parquet preview returning column schema, row counts, and a limited sample (default 100 rows). This endpoint returns literal data rows; restrict to PHI-free tables or redact columns manually.
• mono/apps/vxdata-jobs/src/vxdata/jobs/f_20251215_dicoms/utils.py + organize_bamberg.py: DICOM metadata extraction + lightweight cardinality/uniqueness diagnostics.
• mono/apps/vxdata-api/src/vxdata/api/mock/mock_data.py: tiny seed dataset for UI/dev smoke tests.

History: data-platform repo (archived 2026-05-27) migrated to mono with systematic renaming: apps/api → apps/vxdata-api, apps/worker → apps/vxdata-jobs, packages/client → packages/vxdata-sdk. The architectural PHI-safety gaps identified in 2026-04 remain unfixed as of 2026-08.