---
title: "2026-08-27 clinical ve2e ISUP inference integration"
description: "Consolidated the ve2e data pipeline, rebased ISUP inference onto unify-vits, and validated its container entrypoint."
image: "https://docs.virdx.dev/img/virdx-social-card.png"
---

> Documentation Index
> Fetch the complete documentation index at: https://docs.virdx.dev/llms.txt
> Use this file to discover all available pages before exploring further.

# 2026-08-27 clinical ve2e ISUP inference integration

## Work Done

- Implemented a PHI-redacted `predict_isup` API and CLI for self-contained ClearML EMA models, patient preprocessing, six-class ISUP predictions, and separate biopsy/RPE QWK results on `feat/isup-inference`.
- Moved the standalone `data/` pipeline into `src/ve2e/data_pipeline`, merged its direct dependencies into the root Pixi environment, preserved the `generate` task, and simplified `Dockerfile.argo` to one environment.
- Rebased the feature branch onto the new `unify-vits` base and force-pushed it at `12f69e9d`. Updated inference to use the unified Lightning forward path, transfer/pass `pmask`, and apply model output processors required by DinoFlex background masking and map-derived ISUP grading.
- Validated Ruff, complexity <=8, lockfile consistency, wheel inclusion of YAML configs, and call paths with calldiff. Existing mypy failures remained limited to 15 baseline errors in two unrelated files.
- Built and signed the Tekton image and ran Argo workflow `run-ve2e-smoke-retry-2z9x9`. The workflow succeeded: the CLI imported the consolidated runtime, reached PHI-safe empty-patient validation, returned the expected code 1, and the assertion wrapper returned 0. No PR was created.

## Lessons Learned: Pitfalls

- `kuberun` derives labels from the final command token and the Argo template stores `CMD` inside single quotes. Long Python expressions need a simple final label token, and Python literals must use double quotes or the template strips them.
- `kuberun run` reports a job as healthy after 30 seconds but does not wait for workflow completion. Application success must be checked directly in Argo.
- Manually deleting a pod in `ImagePullBackOff` can leave the Argo node pending without a replacement. Recreating the workflow from the already-built image is more reliable.
- Tekton's build task invokes BuildKit twice and captures the first invocation's successful output. Cold dependency/cache exports can therefore look unexplained and take much longer than visible BuildKit timings.
- Running `git rebase --continue` without `GIT_EDITOR=true` in a non-interactive tool can wait indefinitely for an editor.

## Lessons Learned: Improvements

- Add a documented kuberun smoke-test pattern that uses a checked-in or standard assertion wrapper instead of nested shell/Python quoting.
- Add a kuberun option to wait for final Argo completion and return the workload exit status.
- Fix label sanitization to strip invalid trailing characters after truncation.
- Improve Tekton logs by streaming the first BuildKit invocation and timing cache export separately.
- Document that consolidated data-pipeline inference must call the Lightning module, not the bare architecture, and must transfer `pmask` for unified output processors.

Source: https://docs.virdx.dev/knowledge/inbox/2026-08-27-clinical-ve2e-isup-inference-integration/index.mdx