Histo domain representations

A domain representation is what the histo pipeline produces for one H&E whole-slide image: five label maps that describe the slide down to single cells. This page is for teams that want to use them.

There are three ways in:

1. Read the published outputs from the data platform. This is the normal route. You need vxdata-sdk and zarr, but not the histo package.
2. Production runs on the cluster, which publish new outputs to the platform.
3. Local runs, which write outputs to shared storage for your own experiments.

1. Read published outputs

What is published

Each processed scan has one HistoDomainRep bundle in the histo_domain_reps namespace. The bundle points to five HistoMaps.

Identifiers carry the pipeline version, so several versions can exist side by side:

histodomainrep/<scan path>/v2.0
histomap/<scan path>/tissue_seg-v2.0

<scan path> is the source HistoScan identifier without the histoscan/ prefix. The bundle’s parent_identifier is that HistoScan, so you can find the H&E the maps were made from.

Which scans have a bundle changes with every run, so read it from the platform. For example, bundles per dataset:

reps = client.histo_domain_reps.query().filter(F.histo_version == "v2.0")
reps.select("datasource_id").collect()["datasource_id"].value_counts()

Load one slide

from pathlib import Path

import zarr
from vxdata.sdk import Client, F

client = Client()

# Pick bundles: by dataset, by scan, or by version.
reps = (
    client.histo_domain_reps.query()
    .filter(F.datasource_id == "datasource/essen01", F.histo_version == "v2.0")
    .collect()
)
rep = reps.row(0, named=True)

# Resolve the five maps to storage urls.
fields = ["tissue_segmentation", "cell_segmentation", "nuclei_segmentation",
          "gleason_segmentation", "structure_tensor"]
maps = client.histo_maps.query().filter(F.identifier.is_in([rep[f] for f in fields])).collect()
urls = dict(zip(maps["task"], maps["url"]))

# Download the maps you need (each url is a zarr directory) and open them.
dest = Path("domain_reps")
tissue_path = Path(client.storage.download(urls["TISSUE_SEGMENTATION"], dest))
cell_path = Path(client.storage.download(urls["CELL_SEGMENTATION"], dest))

tissue_group = zarr.open_group(tissue_path, mode="r")
cell_group = zarr.open_group(cell_path, mode="r")
tissue_seg = tissue_group["scale_1"]          # (H, W) uint8
cell_seg = cell_group["scale_1"]              # (H, W) uint32
cell_lut = cell_group["cell_lut"][:]          # (n_cells, 3) uint32
classes = tissue_group.attrs["virdx_attrs"]["misc"]  # class names, ids, colours

The H&E comes from the parent scan: client.histo_scans.query() .filter(F.identifier == rep["parent_identifier"]). Its url is the preprocessed pyramidal TIFF.

The maps

Every map is an OME-Zarr (zarr v3) group. The image is the array scale_1; there is no pyramid. cell_seg.zarr also holds the cell_lut array. The four label maps share one pixel grid: the H&E pyramid level closest to 1 µm/px. The structure tensor is on a 4x coarser grid.

The pipeline does not resample, so the pixel size follows the component’s pyramid, which depends on the scanner. It ranges from 0.88 to 1.10 µm/px:

Always read the pixel size of each map; never assume one value. It is stored in three places that agree:

• the scale in the zarr’s OME metadata, in micrometers (the axes of maps written so far say unit: null; read that as micrometers too),
• the affine in its virdx_attrs,
• generation_outputs["mpp"] on the map’s HistoMap record (the bundle’s mpp is that of the label maps).

Do not derive it from the parent HistoScan: its resolution is often empty, rounded, or (for Frankfurt) 2x too small.

Each map’s generation_outputs also says where its grid sits on the parent HistoScan:

• component_level: the TIFF level of the scan the maps were made from,
• level_downsample: how many times that level is downsampled from level 0, as a whole number (4 for a 0.25 µm/px component read at 1 µm/px). It is the nominal factor, not the ratio of the TIFF level shapes, which rounding makes slightly off,
• stride: 1 for the label maps, 4 for the structure tensor.

Map pixel i is pixel stride * i of level component_level. The translation in the zarr’s OME metadata (and in its affine) is the offset of the map’s first pixel centre from the centre of level-0 pixel (0, 0), on both axes: (d - 1) / 2 · mpp / d µm, where d is level_downsample and mpp is the label maps’ µm/px. For d = 4 at 1 µm/px that is 0.375 µm. The structure tensor has the same translation: its first sample sits on the label maps’ first pixel. Maps written before histo 0.0.162 have translation 0 and lack these three fields.

One id space

tissue_seg, cell_seg and nuclei_seg use the same id space:

Do not hard-code the class table. tissue_seg, cell_seg and nuclei_seg store it in attrs["virdx_attrs"]["misc"]: region_ids, region_classes, region_palette_rgb and id_space. The same table is in their HistoMap’s labels column. gleason_seg stores its own five-class table in the same place, without id_space. The source config is config/domain_representation/taxonomy/fine28.yaml.

Cells

cell_seg.zarr/cell_lut has one row per cell: [cell_id, base_class, gleason_grade]. base_class is a 0–27 id. gleason_grade is 3, 4 or 5 for graded epithelium cells and 0 for all others.

The maps agree by construction:

• On cell pixels, tissue_seg is the cell’s class, graded if the cell has a grade.
• Off cell pixels, cell_seg == tissue_seg.
• Every nucleus id in nuclei_seg is also a cell id in cell_seg.
• Every cell id in the map has a LUT row. A few LUT rows (about 0.3 % on a test slide) have no pixels, so join from the map to the LUT, not the reverse.

Only these classes hold cells: epithelia (1, 2, 22, 23), stroma (3, 4), inflammation, nerve, vessels and arteries. Lumen, fat, blood and artefact classes have none.

Structure tensor

Channel 0 is the dominant gradient orientation in radians, in [-π/2, π/2]. The fibres run perpendicular to it. Channel 1 is coherence in [0, 1]. Decode with value * scale + offset. The scale and offset are in attrs["virdx_attrs"]["misc"]["structure_tensor_encoding"]. Pixel (r, c) samples full-grid pixel (4r, 4c). With histo installed, histo.domain_representation.representation.decode_structure_tensor(array) does the decode.

Provenance and versions

Each bundle records:

• histo_version: the pipeline version (v2.0). It changes whenever the outputs change meaning.
• tissue_model_id, gleason_model_id, cell_model_id: the ClearML model ids.
• generation_parameters: the full pipeline config, including the Docker image tag and the taxonomy.
• generation_outputs: µm/px (mpp), component_level, level_downsample, cell count, timings, and urls of the run log (run.json) and event log (events.jsonl).

A bundle is written only after its five maps are uploaded. If a bundle exists, the scan is complete.

Which tissue is covered

A bundle does not cover the whole slide. Its maps are made from a preprocessed HistoScan (the bundle’s parent_identifier), and preprocessing keeps only part of the tissue:

1. A tissue detector finds the tissue on the raw slide.
2. The mask is split into connected pieces. Pieces that are too small or look like artefacts are dropped.
3. Each kept piece is cropped, straightened and published as its own scan, histoscan/<raw scan path>/preprocessed/component_<i>, whose parent_identifier is the raw scan.

Dropped tissue is in no bundle, so small fragments are missing. One slide can also have several components, each with its own bundle; group by the raw scan for slide-level numbers.

Current settings. The preprocessing factory (mono apps/factory/src/factory/registry.py) runs histo’s tissue detection (histo.tissue_detection.params) with:

Older components used other settings. Components are not redone when the settings change, so the published bundles come from these setups, and one dataset can mix several:

The 30 % rule matters most: on those RPE slides only the main piece and pieces close to its size were kept.

Read the settings of a bundle from its component scan:

import json

scan = (
    client.histo_scans.query()
    .filter(F.identifier == rep["parent_identifier"])
    .collect()
    .row(0, named=True)
)
params = json.loads(scan["preprocessing_parameters"])
params = params["tissue_detection_params"]

params.get("method")                  # "grandqc"; None for Otsu
params.get("min_component_area_um2")  # µm²; Otsu: min_component_area (px)
params["min_area_frac_of_largest"]    # 0.3 on Otsu RPE runs, else 0
params.get("content")                 # content check settings, if it ran

GrandQC runs also publish the whole-slide mask as the FOREGROUND_MASK map histomap/<raw scan path>/preprocessing_tissue_mask. It is in raw slide coordinates and holds only the kept pieces, so it shows what preprocessing kept, but not what it dropped.

Size and practical notes

• Biopsy slides are small (for example 27k × 6k px). Frankfurt whole-mounts are about 47k × 47k px. There, tissue_seg is about 100 MB and cell_seg several times more. Read windows (tissue_seg[y0:y1, x0:x1]) instead of whole arrays when you can.
• Label maps are stored in 1344 × 1344 px chunks inside shards of 5376 rows × one super-window (26880 px); the structure tensor uses a quarter of each. Every pyramid level is sharded too. All are compressed with zstd.
• storage.download(url, dest) keeps the key path under dest, without the bucket name.

2. Production runs: publish to the data platform

A production run processes HistoScans on the cluster and registers the results on the platform, as described in section 1. The run is the Argo WorkflowTemplate histo-domain-representation-v2, kept in the mono repo (apps/factory/argo/_histo-domain-representation-v2.workflowtemplate.yaml) and deployed from there. It runs the histo image: the image holds the code, the environment and the config, so the image tag decides the pipeline version and the models. The pods read everything from S3, Hugging Face and ClearML, with credentials from cluster secrets; they need no shared storage.

Weekly schedule. The CronWorkflow histo-domain-representation-v2-rpe (same folder) runs the template every Sunday on every preprocessed prostate RPE scan (organ=prostate, so a new prostate datasource is picked up without a change), in production. Only scans without a bundle are processed, so a week without new scans costs one short list step. It pins a histo image tag; change it in a reviewed mono PR. A tag with a new pipeline_version reprocesses every scan (about 300 GPU-hours) and its bundles sit next to the old ones. New model ids need a new pipeline_version: the platform does not replace a bundle, so scans that already have one of the same version are skipped with a warning.

Manual runs. You need argo access to the argo-workflows namespace and a histo image (kuberun build on a clean checkout builds the current commit as histo:<branch>-<sha>). Test on staging first: run_environment=staging (the default) registers the bundles on the staging platform and uploads to the domrep_v2_testing group. Staging holds the same HistoScans as production. run_environment=production writes the official v2.0 bundles that other teams read (upload group domainrep_v2.0). Agree production runs with the histo team.

argo submit -n argo-workflows --from workflowtemplate/histo-domain-representation-v2 \
  -p image_tag=<tag> -p run_environment=staging \
  -p datasources=essen01 -p specimen_types=RPE

# or explicit scans
argo submit -n argo-workflows --from workflowtemplate/histo-domain-representation-v2 \
  -p image_tag=<tag> -p scans="histoscan/<a> histoscan/<b>"

The template calls paths inside the image. If a histo change moves or changes the scripts it calls (scripts/domain_representation/list_domain_representation_worklists.py and generate_domain_representation.py), update the template in the same step.

Platform token. The platform does not let anonymous callers read pseudonymized or CC-licensed scans or create maps. The template sends the VXDATA_TOKEN of the secret vxdata-credentials-<run_environment>. Use an image of histo 0.0.141 or newer: older images lack the list step options the template passes.

What happens:

• A list step finds the scans that have no bundle of this pipeline version, reads each slide’s size from its TIFF header on S3, and packs them into worklists. It fails if no slide can be sized. Resubmitting only processes what is left.
• One GPU pod (128 Gi) per worklist downloads the models, then processes its scans one by one. A failed scan does not stop the others; the pod then exits with code 13.
• Inputs are preprocessed HistoScans. A scan’s FOREGROUND_MASK map is used when it has one; without it every window is embedded, which is slower.

A manual run goes to your own Kueue queue (user-yourname), which may hold only one GPU. To also use idle GPUs elsewhere on the cluster, add -l virdx.dev/preemptible=true to argo submit. Those pods can be stopped and rerun, which is safe because done scans are skipped.

Watch the run with argo list -n argo-workflows, argo get -n argo-workflows <workflow> and argo logs -n argo-workflows <workflow>. Finished scans appear as bundles on the platform.

3. Local runs: outputs for your own experiments

A local run writes the same five maps to shared storage and never touches the platform. Use it to try other settings, or slides that are not on the platform.

You need:

• A GPU. The coder pod’s 24 GB GPU slice works with the local profiles.
• A histo checkout with its environment: pixi install --manifest-path packages/histo/pyproject.toml -e cuda-dev.
• ClearML credentials in ~/clearml.conf. The three heads are downloaded from clearml.fra.virdx.dev on the first run and cached in ~/.clearml/cache.
• /mnt/artifacts mounted. The backbone weights are read from there, so no Hugging Face token is needed.
• A vxData token only if your inputs are platform scans.

Run from the repo root:

SSL_CERT_FILE=/etc/ssl/certs/ca-certificates.crt \
REQUESTS_CA_BUNDLE=/etc/ssl/certs/ca-certificates.crt \
pixi run --manifest-path packages/histo/pyproject.toml -e cuda-dev \
  python scripts/domain_representation/generate_domain_representation.py \
  compute=local 'inputs.paths=[/path/to/slide.tiff]'

The two certificate variables let Python trust the internal CA of clearml.fra.virdx.dev. On the coder pod the system bundle already contains it.

Inputs, one of:

• inputs.paths=[...]: preprocessed pyramidal TIFFs. If <name>_tissue_mask.ome.zarr sits next to <name>.tiff, only tissue is embedded.
• inputs.list_file=slides.json: a JSON list of {"path": ..., "identifier": ...}. identifier is optional and names the output folder.
• inputs.scans=[histoscan/...]: platform scans, downloaded first.

inputs.limit=N processes only the first N slides.

Profiles: compute=local (fits the coder pod) and compute=local_small (smaller windows, for quick checks).

Each slide gets a folder /mnt/artifacts/$USER/histo/domain_representation_v2/<slide>/ with the five maps (tissue_seg.zarr, cell_seg.zarr, …), run.json (settings, models, timings), events.jsonl, and links to the input slide and its tissue mask, so a viewer can open the folder as one case. Set output.scratch_dir=... to write elsewhere. Open the maps with zarr as in section 1; their layout is the same.

A slide whose folder already has a run.json is skipped. Delete the folder, or use a new output.scratch_dir, to compute it again.

Any setting in config/domain_representation/generate.yaml can be changed on the command line, for example cell_postprocess.nucleus_threshold=0.6. Settings are checked on load, and a bad combination stops the run with a message.

Python API

The same run from Python. The public API is what histo.domain_representation exports:

from hydra import compose, initialize_config_dir
from histo.domain_representation import load_models, load_settings, local_scans, process_scans

with initialize_config_dir(version_base=None, config_dir="/abs/path/to/histo/config/domain_representation"):
    cfg = compose("generate", overrides=["compute=local", "inputs.paths=[/data/slide.tiff]"])
settings = load_settings(cfg)
report = process_scans(local_scans(settings.inputs), settings, load_models(settings), platform=None)

The config is in the repo, not in the histo package, so this also needs a histo checkout. The other modules of histo are internal; their interfaces may change without notice.

Contact

Ask the histo team before you depend on anything not listed on this page.