---
title: "2026-09-04 Factory RPE histo inference batch"
description: "Queued the open RPE histo preprocessing cases and submitted one 25-case Factory workflow."
image: "https://docs.virdx.dev/img/virdx-social-card.png"
---

> Documentation Index
> Fetch the complete documentation index at: https://docs.virdx.dev/llms.txt
> Use this file to discover all available pages before exploring further.

# 2026-09-04 Factory RPE histo inference batch

## Work Done

- Read the Factory API schema at `https://inference.fra.virdx.dev/openapi.json` and queried `GET /rules/histo-preprocessing-rpe?cached=false`.
- Found 3,639 actionable cases and no cases already in flight.
- Saved the actionable case objects to `apps/factory/histo-preprocessing-rpe-open-cases.json` in the mono checkout.
- Confirmed from `apps/factory/src/factory/services/histo_preprocessing/service.py` that histo preprocessing dispatch uses `SLIDES_PER_JOB = 25`; a 256-case sync would therefore create 11 Argo workflows rather than one.
- Submitted the first 25 case UIDs through `POST /rules/histo-preprocessing-rpe/sync`.
- Factory created workflow `factory-histo-preprocessing-jsc86`: `https://argo-workflows.fra.virdx.dev/workflows/argo-workflows/factory-histo-preprocessing-jsc86`.
- Verified all 25 selected cases appeared in the rule's live `in_flight` map, then removed them from the saved queue. The queue now contains 3,614 cases.
- Inspected live scheduling controls before considering the remaining backlog. The submitter has an Argo semaphore of 10, but its Kueue ClusterQueue has non-borrowable quotas of 2 GPUs, 96 CPUs, and 512 GiB memory. Since each RPE workflow requests 1 GPU, 4 CPUs, 64 GiB memory, and a 1 TiB local cache, Kueue can admit at most two concurrently. The namespace allows 500 Workflow objects and was using 147.
- Filtered the saved queue to 673 scans whose identifiers start with `histoscan/FR` and end with `/whole`. Refreshed the rule state and confirmed all 673 were still actionable.
- Manually created 27 Factory-compatible workflows under the inference API service-account identity: 26 workflows with 25 cases and one with 23. Their payload, deployed RPE config, image, resource requests, labels, annotations, and queue identity mirror an API-created workflow.
- Added the supported `virdx.dev/gpu-class: rtx-6000` label, which admission propagates to pods and converts to the hard `NVIDIA-RTX-6000-Ada-Generation` node selector. Verified the first new pod ran on `virdx-compute-002`, Kueue held total inference usage to two GPUs, and all 673 selected cases appeared in Factory's live in-flight map.
- Removed the 673 submitted cases from the local queue. It then contained 2,941 cases and no remaining `FR.../whole` cases.
- Follow-up showed the first eight admitted Ada workflows all failed with `main: OOMKilled (exit code 137)`. Their main containers had matching 64 GiB memory requests and limits. Failure times ranged from about 8 minutes to several hours. No selected case was materialized because the runner uploads and registers prepared outputs only after processing the full batch.
- Terminated the remaining 19 workflows before their pending pods could run. A fresh uncached rule evaluation showed all 673 selected cases still missing and none in flight, so they were restored to the local queue. It contained 3,614 cases again.
- Recovered per-scan `preprocessing.log` files by mounting the retained generic-ephemeral cache PVCs read-only. All eight failed FR workflows reached `Visualizing results` after component processing and cleanup, then OOM-killed before saving the first QC PNG. The earlier LE workflow completed ten scans and OOM-killed while transforming a `243426 × 97119` component on its eleventh scan.
- Deleted the nine terminal failed pods after log recovery, which garbage-collected their approximately 9 TiB of retained generic-ephemeral PVCs. The PVCs persisted because Argo uses `podGC.strategy: OnWorkflowSuccess`, so failed Pod objects were never removed.
- Submitted one Ada-pinned canary for `histoscan/FR00313/rpe/v9/whole` with 128 GiB memory: `factory-histo-preprocessing-cccf4`. Removed that in-flight case from the local queue, leaving 3,613 cases.
- Opened mono PR [#373](https://github.com/virdx/mono/pull/373), changing Factory's call to `preprocess_sample(..., visualize=False)` and bumping the app version from 0.3.0 to 0.3.1. Local format, lint, typecheck, boundary, lock, and pre-commit checks passed.

## Lessons Learned: Pitfalls

- OpenAPI describes the sync request and response but does not expose the service's per-workflow case limit. Treating one API call as one workflow could accidentally create many workflows.
- The requested 256-case batch size conflicted with the service implementation's 25-slide workflow size. Submission was paused until the user chose which constraint to preserve.
- Mirroring the API's 64 GiB RPE configuration was not safe for the filtered FR whole-slide cohort. The known level-0 full-array materialization in `virdx-ome` can require roughly six times a component's gigapixels in GB, and the campaign was started without a representative canary on this cohort.

## Lessons Learned: Improvements

- Document service dispatch sizes in the Factory API or dashboard so operators can predict the number of workflows before submitting selected cases.
- A dry-run response should ideally include actionable case count and projected workflow count rather than returning only an empty submitted-job list.
- Factory should expose the supported `virdx.dev/gpu-class` choice so pinned submissions do not require reconstructing API-equivalent Workflow objects manually.
- Before a large cohort submission, run a one-case canary from the largest expected whole slide and verify peak memory. For this retry, start with the first scan from the eight-minute failure at 128 GiB rather than resubmitting full 25-case batches blindly.

Source: https://docs.virdx.dev/knowledge/inbox/2026-09-04-histo-factory-rpe-inference-batch/index.mdx