VIPS_CONCURRENCY fix (current as of 2026-08-06)
libvips is not cgroup-aware: it sizes its threadpool from the host core count, not the container’s cpu limit. On virdx’s 512-core nodes this spawned 512 workers regardless of the pod’s 4-core or 8-core quota → oversubscription, throttling, and inflated peak memory.
Fix: set VIPS_CONCURRENCY env var to the cpu budget before import pyvips (libvips caches the default at init; setting it after import is a no-op).
Current location (verified 2026-08-06): mono/apps/inference/src/inference/api/workflow.py line 51, in the run-image Argo template’s podSpecPatch:
"env": [
{"name": "VIPS_CONCURRENCY", "value": threads},
{"name": "OMP_NUM_THREADS", "value": threads},
...
]This sets the env var before the container starts, so pyvips imports the correct value. threads is derived from the job’s cpu request.
History: mono PR #196 (2026-07-07) initially set it in entrypoint.py; PR #199 superseded it and moved the fix to workflow.py podSpecPatch.
pyvips 3.1.1 has no concurrency_set() API — the env var is the only mechanism. A C-level vips_concurrency_set() exists but requires separate cffi setup.
CPU/memory ablation (2026-07-07 batch)
Tested via /histo-preprocessing endpoint → Argo run-image template, varying cpu/memory. Timing from app logs (Finished processing <uuid>.tiff in <N>s) for the one resource that completed end-to-end.
CPU sweep (mem=32Gi):
cpu=1: 282scpu=2: 158scpu=4: 146scpu=8: 87scpu=16: 60s (first batch) / 123-125s (later, contended batch)cpu=32: 86s (first) / 101-138s (later)
Diminishing returns past ~8-16 cores. High variance across batches (2x) due to node contention, which the VIPS_CONCURRENCY fix should reduce.
Memory floor (cpu=8):
2Gi,4Gi,8Gi: OOMKilled (exit 137)16Gi: survives but slower under contention (105-131s vs 87s at 32Gi)32Gi: safe default
Recommendation: cpu=8 (or 16), memory=32Gi, gpu=0. Avoid cpu≤2 and memory<32Gi.
Open virdx-ome bugs (verified unfixed 2026-08-06)
Both bugs are in virdx/histo/virdx-ome/ (virdx-ome migrated from standalone archived repo into virdx/histo subdir on 2026-07-02).
Bug 1: convert_zarr_to_tiff full-array materialization
Location: virdx-ome/src/virdx_ome/creation.py line 167:
return create_tiff_from_array(
array=np.asarray(base[:]), # <-- materializes entire zarr level 0
dest_path=tiff_path,
...
)This loads the full warped component into memory. An AGGC component (47995, 91747)×3 = ~13GB in one alloc → OOMKills a 12Gi pod. The transform itself streams into a scratch zarr fine; this hop throws it away.
Rule of thumb: peak memory ~= 6 × (max-component gigapixels) GB.
Workaround: set output_level>=1 (quarters pixels) or budget memory >= 6 × GP.
A draft PR (virdx/histo#175) was mentioned in 2026-07-07 notes but does not exist in git log (unverified; never merged). The bug remains unfixed.
Bug 2: JPEG-only compression whitelist corrupts masks
Location: virdx-ome/src/virdx_ome/tiff_core/constants.py line 3:
type TiffCompression = Literal["JPEG"]
SUPPORTED_TIFF_COMPRESSION_MAPPING: dict[TiffCompression, str] = {"JPEG": "jpeg"}
COMPRESSION_QUALITY_MAPPING: dict[TiffCompression, int] = {"JPEG": 80}TiffImage.make_pyramidal() reads the source’s compression tag and validates it against this whitelist. Label/mask TIFFs use LZW (lossless) → Unsupported compression: LZW. Supported: ['JPEG'] error.
libvips can read and round-trip LZW losslessly. JPEG is lossy and corrupts label IDs (empirically: ~80% of pixels in a uint8 mask).
Fix (not yet applied): extend whitelist to Literal["JPEG","LZW","DEFLATE","NONE"], map to libvips lzw/deflate/none. img.tiffsave already forwards compression straight through. Masks must never be JPEG-encoded.
Pointers
- Endpoint:
https://inference.fra.virdx.dev/histo-preprocessing(unverified runtime endpoint) - Template:
kubectl -n argo-workflows get workflowtemplate run-image -o yaml - Code:
mono/apps/inference/src/inference/services/histo_preprocessing/+workflow.py - virdx-ome:
virdx/histo/virdx-ome/ - Job logs:
argo -n argo-workflows logs <wf> | grep 'Finished processing'