---
title: "Histo preprocessing libvips container performance"
description: "libvips threadpool oversubscription fix (VIPS_CONCURRENCY), cpu/memory ablation, and two unfixed virdx-ome bugs (convert_zarr_to_tiff materialization, JPEG-only compression)."
image: "https://docs.virdx.dev/img/virdx-social-card.png"
---

> Documentation Index
> Fetch the complete documentation index at: https://docs.virdx.dev/llms.txt
> Use this file to discover all available pages before exploring further.

# Histo preprocessing libvips container performance

## VIPS_CONCURRENCY fix (current as of 2026-08-06)

libvips is **not cgroup-aware**: it sizes its threadpool from the host core count, not the container's cpu limit. On virdx's 512-core nodes this spawned 512 workers regardless of the pod's 4-core or 8-core quota → oversubscription, throttling, and inflated peak memory.

**Fix:** set `VIPS_CONCURRENCY` env var to the cpu budget **before** `import pyvips` (libvips caches the default at init; setting it after import is a no-op).

**Current location (verified 2026-08-06):** `mono/apps/inference/src/inference/api/workflow.py` line 51, in the `run-image` Argo template's `podSpecPatch`:

```python
"env": [
{"name": "VIPS_CONCURRENCY", "value": threads},
{"name": "OMP_NUM_THREADS", "value": threads},
...
]
```

This sets the env var before the container starts, so pyvips imports the correct value. `threads` is derived from the job's cpu request.

**History:** mono PR #196 (2026-07-07) initially set it in `entrypoint.py`; PR #199 superseded it and moved the fix to `workflow.py` podSpecPatch.

**pyvips 3.1.1 has no `concurrency_set()` API** — the env var is the only mechanism. A C-level `vips_concurrency_set()` exists but requires separate cffi setup.

## CPU/memory ablation (2026-07-07 batch)

Tested via `/histo-preprocessing` endpoint → Argo `run-image` template, varying cpu/memory. Timing from app logs (`Finished processing <uuid>.tiff in <N>s`) for the one resource that completed end-to-end.

**CPU sweep** (mem=32Gi):

- `cpu=1`: 282s
- `cpu=2`: 158s
- `cpu=4`: 146s
- `cpu=8`: 87s
- `cpu=16`: 60s (first batch) / 123-125s (later, contended batch)
- `cpu=32`: 86s (first) / 101-138s (later)

**Diminishing returns past ~8-16 cores.** High variance across batches (2x) due to node contention, which the VIPS_CONCURRENCY fix should reduce.

**Memory floor** (cpu=8):

- `2Gi`, `4Gi`, `8Gi`: OOMKilled (exit 137)
- `16Gi`: survives but slower under contention (105-131s vs 87s at 32Gi)
- `32Gi`: safe default

**Recommendation:** `cpu=8` (or 16), `memory=32Gi`, `gpu=0`. Avoid cpu≤2 and memory<32Gi.

## Open virdx-ome bugs (verified unfixed 2026-08-06)

Both bugs are in `virdx/histo/virdx-ome/` (virdx-ome migrated from standalone archived repo into `virdx/histo` subdir on 2026-07-02).

### Bug 1: `convert_zarr_to_tiff` full-array materialization

**Location:** `virdx-ome/src/virdx_ome/creation.py` line 167:

```python
return create_tiff_from_array(
array=np.asarray(base[:]),  # <-- materializes entire zarr level 0
dest_path=tiff_path,
...
)
```

This loads the full warped component into memory. An AGGC component `(47995, 91747)`×3 = ~13GB in one alloc → OOMKills a 12Gi pod. The transform itself streams into a scratch zarr fine; this hop throws it away.

**Rule of thumb:** peak memory ~= 6 × (max-component gigapixels) GB.

**Workaround:** set `output_level>=1` (quarters pixels) or budget `memory >= 6 × GP`.

A draft PR (`virdx/histo#175`) was mentioned in 2026-07-07 notes but **does not exist in git log** (unverified; never merged). The bug remains unfixed.

### Bug 2: JPEG-only compression whitelist corrupts masks

**Location:** `virdx-ome/src/virdx_ome/tiff_core/constants.py` line 3:

```python
type TiffCompression = Literal["JPEG"]

SUPPORTED_TIFF_COMPRESSION_MAPPING: dict[TiffCompression, str] = {"JPEG": "jpeg"}
COMPRESSION_QUALITY_MAPPING: dict[TiffCompression, int] = {"JPEG": 80}
```

`TiffImage.make_pyramidal()` reads the source's compression tag and validates it against this whitelist. Label/mask TIFFs use LZW (lossless) → `Unsupported compression: LZW. Supported: ['JPEG']` error.

libvips **can** read and round-trip LZW losslessly. JPEG is **lossy** and corrupts label IDs (empirically: ~80% of pixels in a uint8 mask).

**Fix (not yet applied):** extend whitelist to `Literal["JPEG","LZW","DEFLATE","NONE"]`, map to libvips `lzw`/`deflate`/`none`. `img.tiffsave` already forwards `compression` straight through. Masks must never be JPEG-encoded.

## Pointers

- Endpoint: `https://inference.fra.virdx.dev/histo-preprocessing` (unverified runtime endpoint)
- Template: `kubectl -n argo-workflows get workflowtemplate run-image -o yaml`
- Code: `mono/apps/inference/src/inference/services/histo_preprocessing/` + `workflow.py`
- virdx-ome: `virdx/histo/virdx-ome/`
- Job logs: `argo -n argo-workflows logs <wf> | grep 'Finished processing'`

Source: https://docs.virdx.dev/knowledge/wiki/workstreams/histo/sops/histo-preprocessing-performance/index.mdx