Skip to content

2026-09-17 Framework managed Factory worker pods

Centralized worker pod setup and removed host-directory mounts from the histo consumer.

Work Done

Updated framework PR #404 at 871f5210 and rebased/updated draft histo PR #406 at 2cf29dbe. The framework Argo backend now owns the private worker pod builder, registry pull secret, vxData token secret and API URL, GPU runtime/selector, shared memory, and lvm scratch PVC sized by worker_disk. Removed the public raw pod_spec option and the consumer’s large pod-spec helper. No hostPath volumes, /mnt/artifacts or /mnt/storage mounts remain in either implementation.

GrandQC now uses the fixed image-local path /opt/models/grandqc/Tissue_Detection_MPP10.pth. Model weights are not yet baked into the Dockerfile; documented that artifact requirement rather than inventing a model downloader. Old host-path input URLs need accessible storage URLs.

Framework/app lint and type checks pass. Manual local checks confirmed scratch sizing, managed mount paths, no host mounts, credential-free backend construction and six-service Argo-configured OpenAPI startup without heavy imports/network. No tests restored, no cluster submissions, no GPU inference or image build. Updated both README files and PR descriptions. Preserved the original worktree’s uncommitted framework pyproject.toml edit byte-for-byte through rebase autostash.

Pitfalls

Moving resource requests alone is insufficient: worker_disk previously did nothing and scratch provisioning was hidden in the consumer. Removing the model host mount also removes access to the upstream GrandQC default weights location; that cannot be described as a validated runnable GPU image until weights are packaged. The old factory pre-commit hook still matches the factory-histo prefix and prints an unrelated staging warning; own app checks ran separately and passed.

Improvements

Consumers should register services and execution sizes, not hand-write pod specs. Keep service-specific inference/thread settings with the application, but keep common cluster plumbing in the backend. Do not restore arbitrary host mounts or client/test abstractions when finishing model/data provisioning.

Navigation

Type to search…

↑↓ navigate↵ selectEsc close