The Factory | VIRDX

Reconciles vxData against a set of rules, and runs the inference that closes the gap.

Keeps vxData complete: a rule’s check diffs the platform, its produce makes one missing item through nanovirdx. No server: one Argo CronWorkflow per rule runs check.sh once and produce.sh once per batch.

Running it

pixi install -e check                                    # SDK + nanovirdx, no GPU stack; works on a Mac
./check.sh histo-preprocessing-rpe plan/                 # plan/batches.json + plan/{cpu,memory,gpu,disk}
pixi run -e check factory check histo-preprocessing-rpe  # {"pending": [...], "done": [...]}
./produce.sh histo-preprocessing-rpe --ids-file b.json   # one worker batch (Linux + CUDA, env `produce`)
./dispatch.sh histo-preprocessing-rpe 375300a145f1    # one reconcile now; tag = a factory-api build (latest = main)

vxData via API_URL / VXDATA_TOKEN; scratch via FACTORY_TMP_DIR. Logs to stderr, JSON to stdout.

Implemented rules

histo-preprocessing-{rpe,essen02-biopsy,panda,aggc-biopsy,aggc-rpe,chimera-rpe}: raw HistoScans of the cohort with no histo.preprocessing component naming them as parent. 25 slides per pod, 4 CPU, 16-32Gi, 1Ti scratch. A no-tissue slide gets preprocessing_descriptor = "histo.preprocessing:no-tissue" on the source, not a failure.
diffsim-bpmri: HistoDomainReps with histo_version in >=2.0,<3 and no DiffsimScan with diffsim_version in >=0.2.1,<0.3; new scans are labelled v0.2.1. 10 reps per pod, 1 GPU, 32Gi, 256Gi scratch. Volumes T2, DWI b100, DWI b1000, ADC; b-values in s/m².

Implementing a rule

class MyRule(Rule):                                # rules/my_rule.py
    batch, cpu, memory, gpu, disk = 25, 4, "32Gi", 0, "0"

    def check(self, client: Client) -> tuple[list[str], list[str]]: ...   # pending, done
    def produce(self, client: Client, identifier: str, workdir: Path) -> None: ...  # no-op if it exists

Register results as you go; add an instance to RULES in registry.py and a CronWorkflow to deployments/cronworkflows.yaml.

Deployment

deployments/build.sh                                                   # Tekton -> zot.fra.virdx.dev:5000/factory-api
kubectl apply -f deployments/factory.workflowtemplate.yaml -f deployments/cronworkflows.yaml

Pods pull :latest (pin image_tag in a CronWorkflow to hold a version), read VXDATA_TOKEN from secret vxdata-token-admin, and trust the on-prem CA from ConfigMap ipa-ca-cert.