---
title: "2026-08-28 clinical ve2e ISUP ClearML patient inference"
description: "Discovered and validated a six-class GG EMA model, then ran PHI-safe inference for two requested patients."
image: "https://docs.virdx.dev/img/virdx-social-card.png"
---

> Documentation Index
> Fetch the complete documentation index at: https://docs.virdx.dev/llms.txt
> Use this file to discover all available pages before exploring further.

# 2026-08-28 clinical ve2e ISUP ClearML patient inference

## Work Done

- Queried the ClearML API directly with bounded requests and found 56 `best_QWK_ema.ckpt` models, including 38 six-class `GG` candidates.
- Selected model `56d3c7efa29847eba0b250983c2c7abb` (`gmt_R-P-100_s44/best_QWK_ema.ckpt`), whose task reported peak validation EMA QWK 0.5984.
- Validated the actual checkpoint on CUDA through `ve2e.inference.isup._load_model`; its self-contained EMA payload, embedded configuration, DinoFlex construction, and weights loaded successfully.
- Submitted a two-patient inference workflow using a Kubernetes Secret and mounted wrapper so patient identifiers were absent from workflow commands and labels. PHI-redacted application logs confirmed preprocessing, inference, evaluation, and result storage for both patients.
- Securely retrieved `results.json`, verified mode `0600`, and removed the temporary reader pod, patient Secret, runner ConfigMap, local result copy, and monitoring heartbeat.

## Lessons Learned: Pitfalls

- The ClearML SDK `Model.query_models` hung despite a healthy API. Direct `auth.login`, `models.get_all`, and `tasks.get_all` calls with explicit timeouts were fast and reliable.
- Embedding the `run-image` WorkflowTemplate requires merging its default arguments (`local_cache`, `run_environment`, and `clearml_server`); parameters supplied by the referencing workflow are not sufficient.
- Shell substitutions stored inside the template's `CMD` variable are not re-evaluated. A mounted executable wrapper is required to read sensitive arguments from a Secret without putting them in the workflow specification.
- Zot manifest TLS timeouts remain intermittent. Retrying the same built image works and avoids unnecessary rebuilds.

## Lessons Learned: Improvements

- Add native sensitive-argument Secret mounts to kuberun or the shared `run-image` template.
- Add a standard private-output retrieval mechanism that checks file mode and avoids printing clinical results into pod logs.
- Prefer bounded direct ClearML API discovery when broad SDK model queries stall.

Source: https://docs.virdx.dev/knowledge/inbox/2026-08-28-clinical-ve2e-isup-clearml-patient-inference/index.mdx