Implemented a PHI-redacted predict_isup API and CLI for self-contained ClearML EMA models, patient preprocessing, six-class ISUP predictions, and separate biopsy/RPE QWK results on feat/isup-inference.
Moved the standalone data/ pipeline into src/ve2e/data_pipeline, merged its direct dependencies into the root Pixi environment, preserved the generate task, and simplified Dockerfile.argo to one environment.
Rebased the feature branch onto the new unify-vits base and force-pushed it at 12f69e9d. Updated inference to use the unified Lightning forward path, transfer/pass pmask, and apply model output processors required by DinoFlex background masking and map-derived ISUP grading.
Validated Ruff, complexity <=8, lockfile consistency, wheel inclusion of YAML configs, and call paths with calldiff. Existing mypy failures remained limited to 15 baseline errors in two unrelated files.
Built and signed the Tekton image and ran Argo workflow run-ve2e-smoke-retry-2z9x9. The workflow succeeded: the CLI imported the consolidated runtime, reached PHI-safe empty-patient validation, returned the expected code 1, and the assertion wrapper returned 0. No PR was created.
Lessons Learned: Pitfalls
kuberun derives labels from the final command token and the Argo template stores CMD inside single quotes. Long Python expressions need a simple final label token, and Python literals must use double quotes or the template strips them.
kuberun run reports a job as healthy after 30 seconds but does not wait for workflow completion. Application success must be checked directly in Argo.
Manually deleting a pod in ImagePullBackOff can leave the Argo node pending without a replacement. Recreating the workflow from the already-built image is more reliable.
Tekton’s build task invokes BuildKit twice and captures the first invocation’s successful output. Cold dependency/cache exports can therefore look unexplained and take much longer than visible BuildKit timings.
Running git rebase --continue without GIT_EDITOR=true in a non-interactive tool can wait indefinitely for an editor.
Lessons Learned: Improvements
Add a documented kuberun smoke-test pattern that uses a checked-in or standard assertion wrapper instead of nested shell/Python quoting.
Add a kuberun option to wait for final Argo completion and return the workload exit status.
Fix label sanitization to strip invalid trailing characters after truncation.
Improve Tekton logs by streaming the first BuildKit invocation and timing cache export separately.
Document that consolidated data-pipeline inference must call the Lightning module, not the bare architecture, and must transfer pmask for unified output processors.
