2026-08-04 infrastructure docs knowledge section two repo build

This repo is now published as the Knowledge section of the docs site (mono#305), which made docs the first image built from two repos.

Work Done

Added this knowledge repo as a second top-level section of the VIRDX docs site (mono/apps/docs, Astro + @cloudflare/nimbus-docs), synced at build time — PR mono#305, branch feat/docs-knowledge-section. Docusaurus/Mintlify split: a horizontal tab strip for the top level, left sidebar scoped one level below.

apps/docs/bin/sync-knowledge.ts normalizes this repo into Nimbus’s strict docs content collection: frontmatter reduced to title/description (+sidebar.order), H1 stripped (Nimbus renders the title from frontmatter), relative *.md links rewritten to extensionless site URLs. Publishes wiki/ and inbox/ only.
Product docs moved under /documentation/* so Documentation and Knowledge are the two tabs; /inference and /vxdata-api redirect. Sidebar collapsed by default, and “Edit this page” suppressed on mirrored pages.
llms.txt now renders the full page tree rather than a flat list.
sources/ is deliberately not published (legacy, superseded by wiki/). Links from wiki/ into it resolve to GitHub blob URLs instead of dead site routes — 15 such links existed.

The interesting half was the build. docs became the first image built from two repos, and knowledge changes far more often than mono does, so nothing in mono’s history moves when the content does.

Settled on: apps/docs/Containerfile does an unconditional COPY .knowledge/ and the sync reads it via KNOWLEDGE_PATH, so no git credentials go inside the image build; whoever builds supplies the checkout. GitHub Actions (Cloudflare Pages) does it with actions/create-github-app-token@v2 against the existing org App — no new secret. For the cluster image, apps/docs/deployments/build.sh now submits an inline pipelineSpec that clones both repos, referencing the existing git-clone, check-image and buildkit-build Tasks unchanged.

Detour worth recording: I first put that recipe in infra_k8s as a docs-image Pipeline plus a nightly CronJob (infra_k8s#174, closed). Inlining it in build.sh needs zero cluster-side changes and puts the recipe next to the COPY .knowledge/ that is its only reason to exist. Nightly rebuilds were dropped; the Cloudflare deploy already refreshes nightly.

Verified with a real build, verify-docs-image-qj4vd, succeeded 4/4: mono@8213fc81 + knowledge@7b1108a8 -> check-image exists=false, tag 45aabe8b046a -> pushed zot.fra.virdx.dev:5000/docs:45aabe8b046a (build/archive/sign all exit 0). additional-tags was blanked so :latest did not move and nothing redeployed.

Lessons Learned: Pitfalls

sops/executing-jobs-on-the-cluster.md is wrong about PipelineRun submission. It says a buildkit-image PipelineRun “needs a shared-workspace volumeClaimTemplate (e.g. 200Gi, storageClassName lvm) and labels virdx.dev/requestor + virdx.dev/username”. In reality the Kyverno ClusterPolicy tekton-mutate-pipelinerun injects all of it — spec.workspaces, spec.taskRunTemplate.serviceAccountName: tekton-build-sa, and both labels — on every CREATE in tekton-builds. You cannot usefully set the labels yourself: the patch is JSON Patch op: add, which replaces. This cost real time: I concluded deployments/build.sh was broken for missing serviceAccountName and workspaces and told the user so, which was wrong. Confirmed by server dry-run showing both injected.
check-image makes builds idempotent on a tag derived from exactly ONE repo’s commit — sha256({commit,context,dockerfile,project[,imageRepository]})[:12] — and build runs only when exists == false. Nothing documents this, and it is the central trap for any multi-input image: a knowledge-only change produces no new image, so a naive nightly cron is a silent no-op. Fix is to feed a composite commit (<mono-sha>-<knowledge-sha>); the tag then moves when either input does, and still short-circuits when neither did.
A ServiceAccount cannot create a PipelineRun in tekton-builds at all. inject-requestor builds virdx.dev/requestor via replace_all(username, '.', '-'). A SA username is system:serviceaccount:<ns>:<name> — no dots, and colons are illegal in a label value, so the API server rejects the object. Reproduced live. Every PipelineRun in the cluster was created by a human OIDC user, and Tekton Triggers is installed (v0.36.0) with no EventListener using it — this is almost certainly why. One-line fix: strip the SA prefix before the dot swap, as the sibling virdx.dev/username rule already does.
git-clone hardcodes its destination as /workspace/source/repo, so two invocations collide. Any second checkout has to be an inline step, unless you change a Task every build in the cluster depends on.
sources/ reads as current but is legacy. index.md describes it neutrally as “any supplementary material” with no staleness marker, and it still holds the guide an agent would reach for on image builds. Only argo-buildah-image-builds.md carries a superseded banner.
Nine sources/monthly-reports/*.md files open at ## with no H1. Cosmetic inside this repo; not cosmetic once published, since the sync falls back to the filename and rendered titles like 01 2026_diffsim. They are unpublished today only because sources/ is excluded.
bun run build does not catch broken internal links — only bun run lint:docs does, and it covers .mdx only. Knowledge pages are .md (plain markdown avoids MDX parse errors on the <placeholder> and {brace} syntax these notes are full of), so they are never link-linted. Moving the product docs to /documentation/* broke 17 links that the build reported as green.

Lessons Learned: Improvements

Correct and extend sops/executing-jobs-on-the-cluster.md — highest-value change. Replace the “PipelineRun needs workspaces + labels” claim with “Kyverno injects taskRunTemplate.serviceAccountName, spec.workspaces (200Gi lvm) and both identity labels; submit neither”. Then add:
- the check-image tag formula and the when exists == false short-circuit, plus the composite-commit trick for multi-input images;
- that a ServiceAccount cannot submit a PipelineRun today, and the one-line policy fix that would change it (blocks all build automation, including the dormant Triggers install);
- that a PipelineRun may carry an inline pipelineSpec, so a one-off recipe needs nothing deployed to infra_k8s — worth stating, since the instinct is to add a Pipeline;
- that git-clone hardcodes /workspace/source/repo.
Add a short “docs site” SOP under wiki/workstreams/infrastructure/sops/: two surfaces (docs.virdx.dev via Cloudflare Pages, docs.fra.virdx.dev via the cluster image, both behind Cloudflare Access, including *.pages.dev previews), that this repo is mirrored in at build time and how, and the rule that content stays here because agents commit here freely while mono is review-gated. Would have removed most of the discovery in this session.
Mark sources/ as legacy in index.md — one clause, e.g. “legacy, superseded by wiki/; not published to the docs site”. Prevents an agent reaching for a stale guide, and now also signals what does and does not reach readers.
Give the nine monthly reports an H1 matching their filename, so published titles are authored rather than slug-derived.
Record that github-pat is a fine-grained PAT on a personal account (AlexLeakeQC), in the secrets list of the cluster SOP. Every Tekton build clones through one engineer’s credential, so an offboarding or rotation breaks all builds at once. Verified it can read this repo. Belongs next to the existing rattler-credentials / zot-registry-credentials notes, ideally with a follow-up to move it to a machine identity.
Note in the docs SOP that only .mdx is link-linted, so a .md page’s links must be checked against dist/ by hand.