Queried the ClearML API directly with bounded requests and found 56 best_QWK_ema.ckpt models, including 38 six-class GG candidates.
Selected model 56d3c7efa29847eba0b250983c2c7abb (gmt_R-P-100_s44/best_QWK_ema.ckpt), whose task reported peak validation EMA QWK 0.5984.
Validated the actual checkpoint on CUDA through ve2e.inference.isup._load_model; its self-contained EMA payload, embedded configuration, DinoFlex construction, and weights loaded successfully.
Submitted a two-patient inference workflow using a Kubernetes Secret and mounted wrapper so patient identifiers were absent from workflow commands and labels. PHI-redacted application logs confirmed preprocessing, inference, evaluation, and result storage for both patients.
Securely retrieved results.json, verified mode 0600, and removed the temporary reader pod, patient Secret, runner ConfigMap, local result copy, and monitoring heartbeat.
Lessons Learned: Pitfalls
The ClearML SDK Model.query_models hung despite a healthy API. Direct auth.login, models.get_all, and tasks.get_all calls with explicit timeouts were fast and reliable.
Embedding the run-image WorkflowTemplate requires merging its default arguments (local_cache, run_environment, and clearml_server); parameters supplied by the referencing workflow are not sufficient.
Shell substitutions stored inside the template’s CMD variable are not re-evaluated. A mounted executable wrapper is required to read sensitive arguments from a Secret without putting them in the workflow specification.
Zot manifest TLS timeouts remain intermittent. Retrying the same built image works and avoids unnecessary rebuilds.
Lessons Learned: Improvements
Add native sensitive-argument Secret mounts to kuberun or the shared run-image template.
Add a standard private-output retrieval mechanism that checks file mode and avoids printing clinical results into pod logs.
Prefer bounded direct ClearML API discovery when broad SDK model queries stall.
