Reviewed the in-progress shared indexer helpers against the current vxData SDK, schemas, API CRUD code, patient registration service, and local API setup.
Confirmed the intended idempotence pattern: exact datasource comparison, one bulk patient preload followed by register-only-missing, diffed bulk patient metadata updates, deterministic resource identifiers, and compare-before-skip resource creation.
Sent concrete recommendations on bulk query column selection, bounded reads and creates, parent ordering, partial-run recovery, authentication, and empty local API validation.
Lessons Learned: Pitfalls
QueryBuilder.select() documentation says metadata is retained, but the parquet implementation returns only the explicitly named columns. Indexers must name identifier, external_uid, license, and access_level when they use them.
patients.register() is create-only and creates a patient with license=None; there is no bulk or get-or-create endpoint. Jobs must preload existing patients, register missing patients one by one, then apply a diffed bulk PatientUpdate.
The SDK chunks create POSTs at 500 but performs one read-after-write over the full input list with child hydration. Passing a very large dataset can still create high response and memory cost.
The local vxd_dev_agent token cannot create ingestion payloads and does not allow the datasets’ share-alike licenses. Empty-stack ingestion tests need vxd_dev_admin.
Lessons Learned: Improvements
Update the vxData SDK guide to state the exact select_columns behavior and the large-create read-after-write behavior.
Add a first-class idempotent or bulk patient registration API if large source indexers become common.
Consider a public create method that returns only the created count, or supports disabling read-after-write and child hydration, for high-volume ingestion.
