---
title: "2026-09-07 Frankfurt RPE preprocessing campaign"
description: "Submitted and completed all 673 Frankfurt whole-slide RPE preprocessing cases with smaller preemptible Ada batches."
image: "https://docs.virdx.dev/img/virdx-social-card.png"
---

> Documentation Index
> Fetch the complete documentation index at: https://docs.virdx.dev/llms.txt
> Use this file to discover all available pages before exploring further.

# 2026-09-07 Frankfurt RPE preprocessing campaign

## Work Done

- Built and deployed Factory 0.3.1 from `virdx/mono#373`, disabling optional preprocessing visualization after the previous 64 GiB jobs OOM-killed during that step.
- Terminated the remaining old-image 128 GiB canary so its case could run with the corrected image.
- Selected all 673 missing `histoscan/FRxxxxx/.../whole` cases and submitted them exactly once as campaign `fr-whole-preemptible-8x-128gi-512gi-20260906`.
- Submitted 85 Argo Workflows: 84 batches of eight scans and one batch of one. Each used one RTX 6000 Ada GPU, 4 CPU, 128 GiB memory, a 512 GiB ephemeral LVM cache, and the shared preemptible Kueue queue.
- Verified the jobs pulled Factory image digest `sha256:abd8e00a9287254b2d131120b890092337767b176bb08aca21efcad45ec21316` and admission injected the intended queue, retry, deadline, identity, and GPU scheduling settings.
- Checkpointed accepted submissions in `apps/factory/histo-preprocessing-rpe-fr-preemptible-submissions.json` and removed only accepted cases from `apps/factory/histo-preprocessing-rpe-open-cases.json`.
- Monitored the campaign to completion. All 85 workflows succeeded. A fresh Factory rule evaluation confirmed all 673 campaign cases done, with zero missing and zero in flight. No cases needed restoration to the queue.

## Lessons Learned: Pitfalls

- The Factory API groups histo preprocessing in fixed batches of 25 and does not expose resource, GPU-class, or preemptibility overrides. The requested operational shape therefore required direct, correctly stamped Argo Workflow submission under the inference API service-account identity.
- Local DNS and OIDC access to the cluster disappeared while the laptop was away, which paused monitoring and produced repeated queued heartbeat prompts. Cluster work continued normally because workflows and queueing are server-side.
- Successful workflow pod GC removes pods quickly, so campaign progress should be read primarily from Workflow objects and Factory rule state rather than expecting all successful pods to remain available.

## Lessons Learned: Improvements

- Keep a reusable operational submission tool that accepts an explicit case set, batch size, resources, GPU class, preemptibility, campaign label, queue path, and receipt path. It should perform a server-side admission dry run and checkpoint each accepted submission atomically.
- Campaign completion must be reconciled against a fresh Factory evaluation. A failed batch can still register partial successes, so restoring every case annotation from a failed workflow would create duplicates.
- For monitoring that must survive a sleeping laptop, use an in-cluster monitor or alert rather than a local agent heartbeat.

Source: https://docs.virdx.dev/knowledge/inbox/2026-09-07-histo-frankfurt-rpe-preprocessing-campaign/index.mdx