--- title: "Executing jobs / images / runs on the cluster" description: "The pathways for running code on the VIRDX Kubernetes cluster (kuberun, per-job Argo templates, raw Argo, off-the-shelf images), the build systems behind them, and the sharp edges." image: "https://docs.virdx.dev/img/virdx-social-card.png" --- > Documentation Index > Fetch the complete documentation index at: https://docs.virdx.dev/llms.txt > Use this file to discover all available pages before exploring further. # Executing jobs / images / runs on the cluster "Run my code on the cluster" always decomposes into two questions: **how does the image get built** and **how does the run get launched**. Pick a pathway below, then see the build systems and run mechanisms it composes from. Prerequisite reading: the `argo-workflow-guide` agent skill (Kyverno mutations, resource caps, volume allow-list, secret wiring). This SOP is the layer above it: which pathway to choose and why. ## Pathways (choose one) | Pathway | Use when | Builds? | Launches | | ------------------------------------------ | ----------------------------------------------------- | ------------------------- | ------------------------------------- | | **1. kuberun CLI** | Quick one-off run of whatever is in your working tree | yes, per run | `run-image` template | | **2. Baked image + per-job Argo template** | Recurring / parametrized data jobs kept in-repo | yes, once per code change | `argo submit --from` a job template | | **3. Raw Argo Workflow/WorkflowTemplate** | Anything bespoke (multi-step, fan-out, custom pods) | you decide | `argo submit` | | **4. Off-the-shelf image** | Job only needs an existing tool (e.g. `mc`, `rclone`) | no build at all | raw Argo pointing at the public image | ### 1. kuberun CLI `kuberun run` commits + pushes the current branch, builds an image from the repo-root Dockerfile (auto-discovered: `Dockerfile.argo` > `Containerfile.argo` > `Dockerfile` > `Containerfile`, else generated), and submits a run. Defaults: build via **tekton/buildkit**, launch via the **`run-image`** template. - Source: `/opt/kuberun` (separate repo, `github.com/virdx/kuberun`); shim at `/usr/local/bin/kuberun`. - Per-repo defaults: `.kuberun.yaml` at the git root (single file, repo-wide). - Best for: iterating on one project. Weak fit for a monorepo where you want to target a specific subproject's Dockerfile (a `--dockerfile` flag is proposed but not yet merged as of 2026-07). ### 2. Baked image + per-job Argo template (the vxdata-jobs pattern) Build the subproject's image once, then submit a small per-job WorkflowTemplate that runs the baked env. This is the pattern for `apps/vxdata-jobs`. - Build: submit a tekton `buildkit-image` PipelineRun (see below). - Job manifest: `apps/vxdata-jobs/src/vxdata/jobs//workflowtemplate.yaml`. - Submit helper: `apps/vxdata-jobs/scripts/submit-job.sh [--param value]` (turns the WorkflowTemplate into a one-off Workflow, injects params, sets the quota/impersonation labels). Older `scripts/build-image.sh` uses the legacy `buildah` template and is **stale for the monorepo** (see gotchas). - Worked example: `f_20260717_essen02_wsi` (WSI DICOM->TIFF conversion) — bakes the `histo` Pixi env, runs `pixi run --frozen -e histo python -m .main`, reads `/mnt/storage` (RO), writes `/mnt/artifacts` (RW). ### 3. Raw Argo Workflow / WorkflowTemplate Hand-write the manifest and `argo -n argo-workflows submit ...`. Full control; you own all the wiring the `argo-workflow-guide` skill documents. Co-locate the YAML with the job code. Cluster templates live in `infra_k8s`. ### 4. Off-the-shelf image If a public image already does the work, skip building entirely. Example: `histai-mirror` runs `minio/mc:...` directly to mirror a bucket. No Dockerfile, no build pipeline. ## Build systems (how the image is made) Both build cluster-side from a pushed git ref, and both mount `rattler-credentials` for the private conda channel (as `--mount=type=secret,id=rattler-credentials`, file at `/run/build-secrets/credentials.json`). Push to `zot.fra.virdx.dev:5000`. | | tekton `buildkit-image` (preferred) | `buildah` WorkflowTemplate (legacy) | | ------------------ | ---------------------------------------------------------------------- | ------------------------------------------------ | | Namespace | `tekton-builds` (PipelineRun) | `argo-workflows` (Workflow) | | Repo vs image name | **decoupled**: `project` = GitHub repo, `image-name` = OCI name | **coupled**: `project` is both | | Layer caching | registry import/export cache (`cache/mono`) | none | | kuberun default | yes | no | | Definitions | pipeline `buildkit-image` + task `buildkit-build` (`tekton-builds` ns) | WorkflowTemplate `buildah` (`argo-workflows` ns) | For a **monorepo**, buildkit is the only sane choice: `project` must be the repo name (`mono`), and buildah has no way to name the image anything else. See the `build-vxdata-api-*` PipelineRuns for a working template to mirror. **Do not submit `spec.workspaces`, `spec.taskRunTemplate.serviceAccountName`, or the `virdx.dev/requestor` / `virdx.dev/username` labels yourself.** The Kyverno ClusterPolicy `tekton-mutate-pipelinerun` injects all four on every CREATE in `tekton-builds`: the `shared-workspace` volumeClaimTemplate (200Gi, storageClassName `lvm`), `serviceAccountName: tekton-build-sa`, and both identity labels (derived from the submitting user via `inject-requestor`). The mutation is a JSON Patch `op: add`, which **replaces** rather than merges, so a value you set yourself just gets clobbered — there is nothing to usefully configure here. (Confirmed by server dry-run; a prior audit of this doc wrongly concluded a build was broken for "missing" a workspace/SA that Kyverno adds automatically — don't repeat that.) A PipelineRun can also carry an inline `pipelineSpec` instead of referencing a stored `Pipeline` — useful for a one-off recipe that shouldn't need anything deployed to `infra_k8s`. ## Run mechanisms - **`run-image`** (WorkflowTemplate, `argo-workflows`): kuberun's runner. Pulls `zot.fra.virdx.dev:5000/:`, runs entrypoint/cmd. Mounts `/mnt/artifacts`, `/mnt/storage`, `/dev/shm`. Params include `image`, `image_tag`, `entrypoint`, `cmd`, cpu/memory/gpu. - **Per-job WorkflowTemplate**: you define the container/command/volumes yourself and submit with `argo submit --from workflowtemplate/` (pathway 2/3). ## Definitions / pointers - Registry: `zot.fra.virdx.dev:5000`. Pull secret for `zot.../*`: `zot-registry-credentials`. - Image naming convention: **job/worker images -> shared `mono` repo**, tag is descriptive (e.g. `mono:histo--`, `mono:worker--`). **Deployable services get their own repo** (`vxdata-api`, `dashboard-*`). - Secrets (in `argo-workflows`): `rattler-credentials` (private conda channel, mount as file + `RATTLER_AUTH_FILE`; env/envFrom blocked by Kyverno), `github-pat` (git clone artifacts only — a fine-grained PAT on a personal GitHub account, `AlexLeakeQC`; every Tekton build clones through this one engineer's credential, so an offboarding or rotation breaks all builds at once — follow-up would be moving it to a machine identity), `zot-registry-credentials` (image pull). - Quota label on every Workflow: `workflows.argoproj.io/creator-preferred-username=` where `` comes from `kubectl auth whoami -o json | jq -r '.status.userInfo.username | split("#")[-1]'`. - Volumes: `/mnt/storage` (RO inputs), `/mnt/artifacts` and `/mnt/scratch` (RW outputs), `/dev/shm` (mandatory for torch DataLoaders). ## Gotchas (learned the hard way) - **No `pixi-cache` PVC exists in `argo-workflows`.** So installing a Pixi env at _runtime_ is a cold, full download every run, and needs channel auth at runtime (else 401 on `quantco.jfrog.io`). Therefore **bake the env into the image**; at runtime just activate it. - **At runtime use `pixi run --frozen`** (uses the baked env, no re-solve, no network). Plain `pixi run` re-solves and fails on the run pod (no channel auth). On CPU nodes running a CUDA-pinned env, also `export CONDA_OVERRIDE_CUDA=12.0`. - **`--frozen` vs `--locked` at build time:** `--locked` aborts if `pixi.lock` is out of sync with `pyproject.toml` ("lock-file not up-to-date"); `--frozen` installs the lock as-is. Prefer keeping the lock in sync, but `--frozen` unblocks a stale lock. - **buildkit's shared registry cache can serve a poisoned/empty layer.** A `RUN pixi install ...` came back `CACHED` with an _empty_ env, and the build still reported success. Guard install steps with a verification that fails the build (e.g. `&& test -x .../.pixi/envs//bin/python`), and cache-bust by changing the RUN command string. - **`buildah` couples repo and image name** via `project` -> for the monorepo it tries to clone `github.com/virdx/` and fails "repository not found". Use tekton buildkit with a separate `image-name`. - **`run-image` does not mount `pixi-cache` or `rattler-credentials`** — it assumes a fully-baked image. Don't rely on runtime installs there. - **`check-image` derives its idempotency tag from exactly one repo's commit**: `sha256({commit,context,dockerfile,project[,imageRepository]})[:12]`, and the downstream `build` step only runs `when exists == false`. Nothing documents this, and it's the central trap for any image built from more than one input (e.g. `docs`, built from `mono` + this knowledge repo): a change to the second input alone produces no new tag, so a naive nightly rebuild is a silent no-op. Fix is to feed a **composite `commit`** (e.g. `-`) so the tag moves when either input does, while still short-circuiting when neither did. - **A ServiceAccount cannot create a PipelineRun in `tekton-builds` today.** `inject-requestor` builds the `virdx.dev/requestor` label via `replace_all(username, '.', '-')`; a SA's username is `system:serviceaccount::`, which contains colons — illegal in a label value — so the API server rejects the object (reproduced live). Every PipelineRun in the cluster today is created by a human OIDC user; Tekton Triggers is installed (v0.36.0) with no EventListener using it, almost certainly for this reason. The one-line fix is to strip the SA prefix before the dot-swap, mirroring the sibling `virdx.dev/username` rule — this blocks all build automation, including standing up Triggers. - **`git-clone` hardcodes its destination to `/workspace/source/repo`.** Two invocations in the same PipelineRun collide; a second checkout (e.g. a multi-repo build like `docs`) has to be an inline step rather than a second `git-clone` Task reference, unless you're willing to change a Task every build in the cluster depends on. ## Related - Agent skill: `argo-workflow-guide` (low-level Argo/Kyverno mechanics). - Repos: `github.com/virdx/mono` (job code + Dockerfiles + job templates), `github.com/virdx/kuberun` (CLI), `infra_k8s` (cluster templates/pipelines). ## Argo `creator-preferred-username` label vs exec-based auth (Investigated 2026-04-28, policy evolved 2026-05): the `workflows.argoproj.io/creator-preferred-username` label is Argo-owned (set by Argo Server in `workflow/common/common.go`), not a generic user label. Kyverno policies that read this label (e.g., queue injection, usage metrics) fail when the label is missing. **When the label is populated:** - Argo Server submission with SSO/OIDC: Argo recovers `preferred_username` from the OIDC token claims and sets the label. - `argo submit --from workflowtemplate/...`: server `SubmitWorkflow()` path runs `ApplySubmitOpts()` after creator labeling, so passed labels survive. **When the label is stripped:** - Direct `argo submit workflow.yaml` with exec-based kubeconfig auth (kubectl exec credentials): Argo's `creator.LabelCreator(...)` runs before CREATE but cannot recover user claims locally → removes creator labels before Kyverno sees them. **Current policy (as of 2026-07):** `infra_k8s/current/argo-workflows/policies/workflow-identity-injection.yaml` (renamed from `workflow-queue-injection.yaml` on 2026-05-13) has three identity-injection rules: 1. `inject-username-from-argo-server`: reads `creator-preferred-username` when the submitter is `system:serviceaccount:argo:*` (Argo Server SSO). 2. `inject-username-from-autoresearch-sa`: collapses autoresearch SA workflows to username `"autoresearch"`. 3. `inject-username-from-apiserver`: uses `request.userInfo.username` (API-server-authenticated) for direct kubectl/SDK OIDC submission; strips the `system:serviceaccount:` prefix if present to keep just the SA name. All three set `virdx.dev/username` (the VIRDX-owned label driving queue assignment and metrics). Rules 2 and 3 also back-populate `creator-preferred-username` so the Argo UI shows a consistent identity field. **General rule:** do not build admission policy (Kyverno mutate/generate) on labels set by a non-admission component (Argo Server, a custom controller) unless you control that component's entire auth path. Argo cannot populate `creator-preferred-username` when it has no access to user claims. If the policy must work for both SSO and direct submission, read from `request.userInfo` (always available in admission context) or introduce a VIRDX-owned label that Kyverno itself populates. The `workflow-identity-injection` policy is the reference implementation: it covers SSO (via `creator-preferred-username`), ServiceAccount submission (via `request.userInfo`), and direct OIDC kubectl (also via `request.userInfo`), and ensures `virdx.dev/username` is always set. Source: https://docs.virdx.dev/knowledge/wiki/workstreams/infrastructure/sops/executing-jobs-on-the-cluster/index.mdx