Executing jobs / images / runs on the cluster

“Run my code on the cluster” always decomposes into two questions: how does the image get built and how does the run get launched. Pick a pathway below, then see the build systems and run mechanisms it composes from.

Prerequisite reading: the argo-workflow-guide agent skill (Kyverno mutations, resource caps, volume allow-list, secret wiring). This SOP is the layer above it: which pathway to choose and why.

Pathways (choose one)

1. kuberun CLI

kuberun run commits + pushes the current branch, builds an image from the repo-root Dockerfile (auto-discovered: Dockerfile.argo > Containerfile.argo > Dockerfile > Containerfile, else generated), and submits a run. Defaults: build via tekton/buildkit, launch via the run-image template.

• Source: /opt/kuberun (separate repo, github.com/virdx/kuberun); shim at /usr/local/bin/kuberun.
• Per-repo defaults: .kuberun.yaml at the git root (single file, repo-wide).
• Best for: iterating on one project. Weak fit for a monorepo where you want to target a specific subproject’s Dockerfile (a --dockerfile flag is proposed but not yet merged as of 2026-07).

2. Baked image + per-job Argo template (the vxdata-jobs pattern)

Build the subproject’s image once, then submit a small per-job WorkflowTemplate that runs the baked env. This is the pattern for apps/vxdata-jobs.

• Build: submit a tekton buildkit-image PipelineRun (see below).
• Job manifest: apps/vxdata-jobs/src/vxdata/jobs/<job>/workflowtemplate.yaml.
• Submit helper: apps/vxdata-jobs/scripts/submit-job.sh <job> [--param value] (turns the WorkflowTemplate into a one-off Workflow, injects params, sets the quota/impersonation labels). Older scripts/build-image.sh uses the legacy buildah template and is stale for the monorepo (see gotchas).
• Worked example: f_20260717_essen02_wsi (WSI DICOM->TIFF conversion) — bakes the histo Pixi env, runs pixi run --frozen -e histo python -m <job>.main, reads /mnt/storage (RO), writes /mnt/artifacts (RW).

3. Raw Argo Workflow / WorkflowTemplate

Hand-write the manifest and argo -n argo-workflows submit .... Full control; you own all the wiring the argo-workflow-guide skill documents. Co-locate the YAML with the job code. Cluster templates live in infra_k8s.

4. Off-the-shelf image

If a public image already does the work, skip building entirely. Example: histai-mirror runs minio/mc:... directly to mirror a bucket. No Dockerfile, no build pipeline.

Build systems (how the image is made)

Both build cluster-side from a pushed git ref, and both mount rattler-credentials for the private conda channel (as --mount=type=secret,id=rattler-credentials, file at /run/build-secrets/credentials.json). Push to zot.fra.virdx.dev:5000.

For a monorepo, buildkit is the only sane choice: project must be the repo name (mono), and buildah has no way to name the image anything else. See the build-vxdata-api-* PipelineRuns for a working template to mirror.

Do not submit spec.workspaces, spec.taskRunTemplate.serviceAccountName, or the virdx.dev/requestor / virdx.dev/username labels yourself. The Kyverno ClusterPolicy tekton-mutate-pipelinerun injects all four on every CREATE in tekton-builds: the shared-workspace volumeClaimTemplate (200Gi, storageClassName lvm), serviceAccountName: tekton-build-sa, and both identity labels (derived from the submitting user via inject-requestor). The mutation is a JSON Patch op: add, which replaces rather than merges, so a value you set yourself just gets clobbered — there is nothing to usefully configure here. (Confirmed by server dry-run; a prior audit of this doc wrongly concluded a build was broken for “missing” a workspace/SA that Kyverno adds automatically — don’t repeat that.) A PipelineRun can also carry an inline pipelineSpec instead of referencing a stored Pipeline — useful for a one-off recipe that shouldn’t need anything deployed to infra_k8s.

Run mechanisms

• run-image (WorkflowTemplate, argo-workflows): kuberun’s runner. Pulls zot.fra.virdx.dev:5000/<image>:<tag>, runs entrypoint/cmd. Mounts /mnt/artifacts, /mnt/storage, /dev/shm. Params include image, image_tag, entrypoint, cmd, cpu/memory/gpu.
• Per-job WorkflowTemplate: you define the container/command/volumes yourself and submit with argo submit --from workflowtemplate/<name> (pathway 2/3).

Definitions / pointers

• Registry: zot.fra.virdx.dev:5000. Pull secret for zot.../*: zot-registry-credentials.
• Image naming convention: job/worker images -> shared mono repo, tag is descriptive (e.g. mono:histo-<branch>-<sha>, mono:worker-<branch>-<sha>). Deployable services get their own repo (vxdata-api, dashboard-*).
• Secrets (in argo-workflows): rattler-credentials (private conda channel, mount as file + RATTLER_AUTH_FILE; env/envFrom blocked by Kyverno), github-pat (git clone artifacts only — a fine-grained PAT on a personal GitHub account, AlexLeakeQC; every Tekton build clones through this one engineer’s credential, so an offboarding or rotation breaks all builds at once — follow-up would be moving it to a machine identity), zot-registry-credentials (image pull).
• Quota label on every Workflow: workflows.argoproj.io/creator-preferred-username=<user> where <user> comes from kubectl auth whoami -o json | jq -r '.status.userInfo.username | split("#")[-1]'.
• Volumes: /mnt/storage (RO inputs), /mnt/artifacts and /mnt/scratch (RW outputs), /dev/shm (mandatory for torch DataLoaders).

Gotchas (learned the hard way)

• No pixi-cache PVC exists in argo-workflows. So installing a Pixi env at runtime is a cold, full download every run, and needs channel auth at runtime (else 401 on quantco.jfrog.io). Therefore bake the env into the image; at runtime just activate it.
• At runtime use pixi run --frozen (uses the baked env, no re-solve, no network). Plain pixi run re-solves and fails on the run pod (no channel auth). On CPU nodes running a CUDA-pinned env, also export CONDA_OVERRIDE_CUDA=12.0.
• --frozen vs --locked at build time: --locked aborts if pixi.lock is out of sync with pyproject.toml (“lock-file not up-to-date”); --frozen installs the lock as-is. Prefer keeping the lock in sync, but --frozen unblocks a stale lock.
• buildkit’s shared registry cache can serve a poisoned/empty layer. A RUN pixi install ... came back CACHED with an empty env, and the build still reported success. Guard install steps with a verification that fails the build (e.g. && test -x .../.pixi/envs/<env>/bin/python), and cache-bust by changing the RUN command string.
• buildah couples repo and image name via project -> for the monorepo it tries to clone github.com/virdx/<image-name> and fails “repository not found”. Use tekton buildkit with a separate image-name.
• run-image does not mount pixi-cache or rattler-credentials — it assumes a fully-baked image. Don’t rely on runtime installs there.
• check-image derives its idempotency tag from exactly one repo’s commit: sha256({commit,context,dockerfile,project[,imageRepository]})[:12], and the downstream build step only runs when exists == false. Nothing documents this, and it’s the central trap for any image built from more than one input (e.g. docs, built from mono + this knowledge repo): a change to the second input alone produces no new tag, so a naive nightly rebuild is a silent no-op. Fix is to feed a composite commit (e.g. <mono-sha>-<knowledge-sha>) so the tag moves when either input does, while still short-circuiting when neither did.
• A ServiceAccount cannot create a PipelineRun in tekton-builds today. inject-requestor builds the virdx.dev/requestor label via replace_all(username, '.', '-'); a SA’s username is system:serviceaccount:<ns>:<name>, which contains colons — illegal in a label value — so the API server rejects the object (reproduced live). Every PipelineRun in the cluster today is created by a human OIDC user; Tekton Triggers is installed (v0.36.0) with no EventListener using it, almost certainly for this reason. The one-line fix is to strip the SA prefix before the dot-swap, mirroring the sibling virdx.dev/username rule — this blocks all build automation, including standing up Triggers.
• git-clone hardcodes its destination to /workspace/source/repo. Two invocations in the same PipelineRun collide; a second checkout (e.g. a multi-repo build like docs) has to be an inline step rather than a second git-clone Task reference, unless you’re willing to change a Task every build in the cluster depends on.

Related

• Agent skill: argo-workflow-guide (low-level Argo/Kyverno mechanics).
• Repos: github.com/virdx/mono (job code + Dockerfiles + job templates), github.com/virdx/kuberun (CLI), infra_k8s (cluster templates/pipelines).

Argo creator-preferred-username label vs exec-based auth

(Investigated 2026-04-28, policy evolved 2026-05): the workflows.argoproj.io/creator-preferred-username label is Argo-owned (set by Argo Server in workflow/common/common.go), not a generic user label. Kyverno policies that read this label (e.g., queue injection, usage metrics) fail when the label is missing.

When the label is populated:

• Argo Server submission with SSO/OIDC: Argo recovers preferred_username from the OIDC token claims and sets the label.
• argo submit --from workflowtemplate/...: server SubmitWorkflow() path runs ApplySubmitOpts() after creator labeling, so passed labels survive.

When the label is stripped:

• Direct argo submit workflow.yaml with exec-based kubeconfig auth (kubectl exec credentials): Argo’s creator.LabelCreator(...) runs before CREATE but cannot recover user claims locally → removes creator labels before Kyverno sees them.

Current policy (as of 2026-07): infra_k8s/current/argo-workflows/policies/workflow-identity-injection.yaml (renamed from workflow-queue-injection.yaml on 2026-05-13) has three identity-injection rules:

1. inject-username-from-argo-server: reads creator-preferred-username when the submitter is system:serviceaccount:argo:* (Argo Server SSO).
2. inject-username-from-autoresearch-sa: collapses autoresearch SA workflows to username "autoresearch".
3. inject-username-from-apiserver: uses request.userInfo.username (API-server-authenticated) for direct kubectl/SDK OIDC submission; strips the system:serviceaccount: prefix if present to keep just the SA name.

All three set virdx.dev/username (the VIRDX-owned label driving queue assignment and metrics). Rules 2 and 3 also back-populate creator-preferred-username so the Argo UI shows a consistent identity field.

General rule: do not build admission policy (Kyverno mutate/generate) on labels set by a non-admission component (Argo Server, a custom controller) unless you control that component’s entire auth path. Argo cannot populate creator-preferred-username when it has no access to user claims. If the policy must work for both SSO and direct submission, read from request.userInfo (always available in admission context) or introduce a VIRDX-owned label that Kyverno itself populates.

The workflow-identity-injection policy is the reference implementation: it covers SSO (via creator-preferred-username), ServiceAccount submission (via request.userInfo), and direct OIDC kubectl (also via request.userInfo), and ensures virdx.dev/username is always set.