Built and deployed Factory 0.3.1 from virdx/mono#373, disabling optional preprocessing visualization after the previous 64 GiB jobs OOM-killed during that step.
Terminated the remaining old-image 128 GiB canary so its case could run with the corrected image.
Selected all 673 missing histoscan/FRxxxxx/.../whole cases and submitted them exactly once as campaign fr-whole-preemptible-8x-128gi-512gi-20260906.
Submitted 85 Argo Workflows: 84 batches of eight scans and one batch of one. Each used one RTX 6000 Ada GPU, 4 CPU, 128 GiB memory, a 512 GiB ephemeral LVM cache, and the shared preemptible Kueue queue.
Verified the jobs pulled Factory image digest sha256:abd8e00a9287254b2d131120b890092337767b176bb08aca21efcad45ec21316 and admission injected the intended queue, retry, deadline, identity, and GPU scheduling settings.
Checkpointed accepted submissions in apps/factory/histo-preprocessing-rpe-fr-preemptible-submissions.json and removed only accepted cases from apps/factory/histo-preprocessing-rpe-open-cases.json.
Monitored the campaign to completion. All 85 workflows succeeded. A fresh Factory rule evaluation confirmed all 673 campaign cases done, with zero missing and zero in flight. No cases needed restoration to the queue.
Lessons Learned: Pitfalls
The Factory API groups histo preprocessing in fixed batches of 25 and does not expose resource, GPU-class, or preemptibility overrides. The requested operational shape therefore required direct, correctly stamped Argo Workflow submission under the inference API service-account identity.
Local DNS and OIDC access to the cluster disappeared while the laptop was away, which paused monitoring and produced repeated queued heartbeat prompts. Cluster work continued normally because workflows and queueing are server-side.
Successful workflow pod GC removes pods quickly, so campaign progress should be read primarily from Workflow objects and Factory rule state rather than expecting all successful pods to remain available.
Lessons Learned: Improvements
Keep a reusable operational submission tool that accepts an explicit case set, batch size, resources, GPU class, preemptibility, campaign label, queue path, and receipt path. It should perform a server-side admission dry run and checkpoint each accepted submission atomically.
Campaign completion must be reconciled against a fresh Factory evaluation. A failed batch can still register partial successes, so restoring every case annotation from a failed workflow would create duplicates.
For monitoring that must survive a sleeping laptop, use an in-cluster monitor or alert rather than a local agent heartbeat.
