---
title: "2026-08-27 ve2e data pipeline consolidation audit"
description: "Audited path, import, package, dependency, Docker, and CI risks when folding the standalone ve2e data pipeline into the main package."
image: "https://docs.virdx.dev/img/virdx-social-card.png"
---

> Documentation Index
> Fetch the complete documentation index at: https://docs.virdx.dev/llms.txt
> Use this file to discover all available pages before exploring further.

# 2026-08-27 ve2e data pipeline consolidation audit

## Work Done

- Audited the in-progress move from `data/src` to `src/ve2e/data_pipeline` and from `data/config` to package-local configuration.
- Inspected imports, config and secret resolution, root and pipeline Pixi manifests, lockfiles, Docker assembly, CI targets, documentation, and ignore rules.
- Reported required fixes to the implementing agent. The highest-risk items were the stale `data/.env.secrets` path in ISUP inference, the root lockfile update required by frozen installs, preservation of the externally used `pixi run generate` command, and newly exposing pipeline code to root mypy.
- Flagged package-data verification for YAML, the missing direct `nibabel` declaration, and runtime-version drift from the former isolated Transformers/PEFT environment.
- Did not edit the ve2e worktree.

## Lessons Learned: Pitfalls

- The knowledge base has no `clinical` workstream although the agent startup instructions list it. Infrastructure was the closest available workstream for this packaging and Docker audit.
- The repository was being edited concurrently, so an initial inventory became stale during the audit. Re-reading `git status` and the unstaged diff was necessary before reporting findings.
- The local machine cannot execute the repository's Linux-only Pixi environments, so the exact mypy command could not be validated locally. Static inspection still showed that the move expands the CI type-check scope.

## Lessons Learned: Improvements

- A project-level consolidation checklist would help: preserve CLI contracts, relocate secret/config paths, declare package data, merge direct dependencies and version constraints, regenerate frozen lockfiles, update container bootstrap layers, and compare CI scope before and after a source-tree move.
- Document the Data Platform's exact command contract for repository pipelines. This would make it clear whether deleting a nested Pixi task is a breaking production change.

Source: https://docs.virdx.dev/knowledge/inbox/2026-08-27-infrastructure-ve2e-data-pipeline-consolidation-audit/index.mdx