Read the Factory API schema at https://inference.fra.virdx.dev/openapi.json and queried GET /rules/histo-preprocessing-rpe?cached=false.
Found 3,639 actionable cases and no cases already in flight.
Saved the actionable case objects to apps/factory/histo-preprocessing-rpe-open-cases.json in the mono checkout.
Confirmed from apps/factory/src/factory/services/histo_preprocessing/service.py that histo preprocessing dispatch uses SLIDES_PER_JOB = 25; a 256-case sync would therefore create 11 Argo workflows rather than one.
Submitted the first 25 case UIDs through POST /rules/histo-preprocessing-rpe/sync.
Factory created workflow factory-histo-preprocessing-jsc86: https://argo-workflows.fra.virdx.dev/workflows/argo-workflows/factory-histo-preprocessing-jsc86.
Verified all 25 selected cases appeared in the rule’s live in_flight map, then removed them from the saved queue. The queue now contains 3,614 cases.
Inspected live scheduling controls before considering the remaining backlog. The submitter has an Argo semaphore of 10, but its Kueue ClusterQueue has non-borrowable quotas of 2 GPUs, 96 CPUs, and 512 GiB memory. Since each RPE workflow requests 1 GPU, 4 CPUs, 64 GiB memory, and a 1 TiB local cache, Kueue can admit at most two concurrently. The namespace allows 500 Workflow objects and was using 147.
Filtered the saved queue to 673 scans whose identifiers start with histoscan/FR and end with /whole. Refreshed the rule state and confirmed all 673 were still actionable.
Manually created 27 Factory-compatible workflows under the inference API service-account identity: 26 workflows with 25 cases and one with 23. Their payload, deployed RPE config, image, resource requests, labels, annotations, and queue identity mirror an API-created workflow.
Added the supported virdx.dev/gpu-class: rtx-6000 label, which admission propagates to pods and converts to the hard NVIDIA-RTX-6000-Ada-Generation node selector. Verified the first new pod ran on virdx-compute-002, Kueue held total inference usage to two GPUs, and all 673 selected cases appeared in Factory’s live in-flight map.
Removed the 673 submitted cases from the local queue. It then contained 2,941 cases and no remaining FR.../whole cases.
Follow-up showed the first eight admitted Ada workflows all failed with main: OOMKilled (exit code 137). Their main containers had matching 64 GiB memory requests and limits. Failure times ranged from about 8 minutes to several hours. No selected case was materialized because the runner uploads and registers prepared outputs only after processing the full batch.
Terminated the remaining 19 workflows before their pending pods could run. A fresh uncached rule evaluation showed all 673 selected cases still missing and none in flight, so they were restored to the local queue. It contained 3,614 cases again.
Recovered per-scan preprocessing.log files by mounting the retained generic-ephemeral cache PVCs read-only. All eight failed FR workflows reached Visualizing results after component processing and cleanup, then OOM-killed before saving the first QC PNG. The earlier LE workflow completed ten scans and OOM-killed while transforming a 243426 × 97119 component on its eleventh scan.
Deleted the nine terminal failed pods after log recovery, which garbage-collected their approximately 9 TiB of retained generic-ephemeral PVCs. The PVCs persisted because Argo uses podGC.strategy: OnWorkflowSuccess, so failed Pod objects were never removed.
Submitted one Ada-pinned canary for histoscan/FR00313/rpe/v9/whole with 128 GiB memory: factory-histo-preprocessing-cccf4. Removed that in-flight case from the local queue, leaving 3,613 cases.
Opened mono PR #373, changing Factory’s call to preprocess_sample(..., visualize=False) and bumping the app version from 0.3.0 to 0.3.1. Local format, lint, typecheck, boundary, lock, and pre-commit checks passed.
Lessons Learned: Pitfalls
OpenAPI describes the sync request and response but does not expose the service’s per-workflow case limit. Treating one API call as one workflow could accidentally create many workflows.
The requested 256-case batch size conflicted with the service implementation’s 25-slide workflow size. Submission was paused until the user chose which constraint to preserve.
Mirroring the API’s 64 GiB RPE configuration was not safe for the filtered FR whole-slide cohort. The known level-0 full-array materialization in virdx-ome can require roughly six times a component’s gigapixels in GB, and the campaign was started without a representative canary on this cohort.
Lessons Learned: Improvements
Document service dispatch sizes in the Factory API or dashboard so operators can predict the number of workflows before submitting selected cases.
A dry-run response should ideally include actionable case count and projected workflow count rather than returning only an empty submitted-job list.
Factory should expose the supported virdx.dev/gpu-class choice so pinned submissions do not require reconstructing API-equivalent Workflow objects manually.
Before a large cohort submission, run a one-case canary from the largest expected whole slide and verify peak memory. For this retry, start with the first scan from the eight-minute failure at 128 GiB rather than resubmitting full 25-case batches blindly.
