vxData production DB restore + cluster storage gotchas

Incident 2026-07-16: compute-002 was silently reprovisioned overnight (fresh, older OS image than its siblings). lvm storageclass (topolvm.io = node-local LVM, no replication, reclaim policy Delete) meant the node rebuild destroyed vxData’s and Gitea’s postgres volumes; the StatefulSets auto-created fresh empty PVCs, postgres re-initialized blank, and Alembic rebuilt an empty schema (36 tables, 0 rows) with no error anywhere.

Any DB on lvm storageclass is one node rebuild away from silent total loss. It needs external backups (vxData/Gitea already have S3 backups) and ideally replication or a non-ephemeral backend — this is not yet fixed cluster-wide.

Restore runbook (Postgres from Ceph S3 backup)

1. Locate the dump: bucket vxdata/vxdata-production-backups, endpoint s3.fra.virdx.dev, creds in secret ceph-vxdata-bucket-credentials. Download with mc --insecure (self-signed cert).
2. Run a one-shot Job: initContainer downloads the dump via mc; main container postgres:17-alpine:
   • terminate active sessions on the target DB,
   • DROP SCHEMA public CASCADE,
   • pg_restore --single-transaction --exit-on-error the dump.
3. Do this against the live DB — don’t try to quiesce the app first.

ArgoCD app-of-apps fights you

vxdata-production is owned by the root app. Patching its syncPolicy.automated to null (to scale the API down during restore) was reverted within seconds by root’s self-heal. Don’t try to scale/edit ArgoCD-managed resources live during an incident — restore against the live DB instead (see above).

Backups can fail silently

The 03:00 UTC backup CronJob failed the day of the outage — pg_dump exited in <1s (DB already unreachable) and produced no file, with no alert. A zero-byte/failed backup should page — this alerting does not exist yet.

Argo ARGO_TEMPLATE 128KB ceiling → spurious configmaps is forbidden

Argo passes each pod’s resolved template via the ARGO_TEMPLATE env var. Past 131072 bytes (maxEnvVarLen), Argo offloads it into a ConfigMap (upstream Argo PR #12325, gated behind helm value controller.rbac.writeConfigMaps). Our workflow-controller ClusterRole only grants get/watch/list on configmaps, not create — so a large inlined job spec (e.g. a 340KB base64 blob passed as a cmd param) fails with configmaps is forbidden, which looks like an RBAC regression but is a symptom of passing large params inline. Fix by either:

• passing the input by reference (file/S3), not as an inline CLI arg — the correct fix for large job specs, or
• enabling controller.rbac.writeConfigMaps: true in the Argo Workflows helm chart (not yet enabled).

worker_processes auto + 512-core nodes = OOM crashloop

nginx’s worker_processes auto reads the host’s core count via sysconf(_SC_NPROCESSORS_ONLN) — it is not cgroup-CPU-limit-aware (long-standing upstream issue, nginx trac #1151 / nginx#855, open since 2018). On virdx’s 512-core nodes this forks 512 workers, each preallocating buffers, which OOMKills a small-limit pod (observed: a 512Mi single-replica edge proxy for data.fra.virdx.dev, OOMKilling every ~1 minute after traffic resumed post-restore).

• NGINX_ENTRYPOINT_WORKER_PROCESSES_AUTOTUNE=1 does not work on pods with readOnlyRootFilesystem: true — the tuner needs to rewrite /etc/nginx/nginx.conf and fails (silently) under a read-only rootfs. All virdx edge proxies run read-only-rootfs, so this env var is the wrong fix here even though it looks like the obvious one.
• Correct fix: mount a full nginx.conf with worker_processes pinned to an explicit small number, via a subPath volume mount (the pattern stitoo/vxseq already use). staging-services instead uses a sed-to-/tmp hack. These should converge on the subPath mount as the one correct mechanism — currently there are three different hacks in use across services.
• Every new nginx/edge deployment on this cluster must pin worker_processes explicitly — never rely on auto.

Related PRs (as of 2026-07-16)

• virdx/infra_k8s#154 (merged): edge replicas 1→2, maxUnavailable: 0, memory 512Mi→2Gi — buys headroom but does not fix the worker-count root cause (its autotune env var is a no-op here).
• virdx/infra_k8s#155 (open at time of writing): mounts the correct nginx.conf via subPath — the actual fix; not yet merged, so the edge currently relies on the #154 replica/memory cushion.

Private registry images need matching imagePullSecrets

(2026-03-30 incident): Removing imagePullSecrets from deployments pulling from ghcr.io/virdx/* private repos caused immediate ImagePullBackOff in staging/production/public namespaces. GHCR package visibility = private → kubelet 401s without a valid credential → pod stays Pending forever.

Fast diagnosis: kubectl describe pod <name> → events show ghcr.io/virdx/<image>: not found (misleading message; the image exists, auth failed). ArgoCD app shows Synced + Degraded (sync succeeded but pods won’t start).

Fix: ensure spec.imagePullSecrets: [{name: ghcr-pull}] (or the registry-specific pull secret) exists in every deployment/statefulset/job pulling private images. The secret must live in the same namespace as the pod.

Registry mirrors are not pull-through caches

(2026-04-01 incident, resolved 2026-04-21): k3s nodes mirrored ghcr.io to a local registry/registry deployment via /etc/rancher/k3s/registries.yaml. The mirror had no pull-through / remote-proxy config, so it returned 404 for any repo not already in its local catalog. kubelet never reached GHCR auth; it failed against the mirror first.

The local registry mirror was removed 2026-04-21 (infra_k8s commit 9f2af4f). If image pulls fail again with mysterious 404s on images that definitely exist upstream:

1. kubectl debug node/<node> --image=busybox:1.36 -- chroot /host cat /etc/rancher/k3s/registries.yaml — check whether a mirror/rewrite rule exists.
2. Use crictl on the node to test the pull path: sudo crictl pull <image> will show whether it’s hitting a mirror or going direct.
3. Check the mirror registry logs (if it exists): kubectl -n <mirror-ns> logs deploy/<mirror> --since=5m for 404 responses.

A registry mirror with no upstream fallback is a purely local cache; it returns 404 for anything not pre-populated. A pull-through cache proxies misses to the upstream registry. Don’t confuse the two.

nginx-unprivileged auto-tunes to host CPU count (not pod limit)

nginx’s worker_processes auto reads sysconf(_SC_NPROCESSORS_ONLN) (the host’s core count), not the pod’s CPU limit (nginx trac #1151, open since 2018). On VIRDX’s 512-core nodes this forks 512 workers, each preallocating buffers → instant OOM on small-memory pods.

• Observed 2026-04-02 in the now-removed ollama TLS sidecar (nginxinc/nginx-unprivileged:alpine, 128Mi limit, 195 workers, exit 137).
• NGINX_ENTRYPOINT_WORKER_PROCESSES_AUTOTUNE=1 does not work under readOnlyRootFilesystem: true — the autotune script rewrites /etc/nginx/nginx.conf and fails silently. All VIRDX edge proxies run read-only rootfs.

Current fix (as of 2026-07): all active nginx edge deployments (stitoo/edge, vxdata-production/edge, vxseq/edge, staging-services/edge) mount a full nginx.conf via ConfigMap with worker_processes pinned to 2 (subPath volume mount for stitoo/vxdata/vxseq; sed-to-/tmp hack for staging-services). See infra_k8s current/vxdata-production/edge.yaml for the ConfigMap approach.

Every new nginx/edge deployment on this cluster must pin worker_processes explicitly — never rely on auto. The gotcha is documented here because it’s a node characteristic, not a service characteristic: any future nginx pod will hit it.

Bulk S3/object migration with rclone Pods

(MinIO → Ceph migration, complete 2026-07): the production vxdata bucket now lives at https://s3.fra.virdx.dev/ (Ceph-backed); the old jumphost MinIO (http://192.168.10.101:10000) is retired for main storage but still hosts Postgres backup dumps (see BACKUP_S3_ENDPOINT in vxdata-production/configmap.yaml).

Durable in-cluster rclone migration patterns:

• Trailing slashes are mandatory. Source object stores (MinIO, S3) keep zero-byte directory-marker objects. rclone copy src:bucket/prefix dst:bucket/prefix (no trailing /) HEADs the path, sees a “file”, aborts with is a file not a directory. Always use src:.../ and dst:.../.
• Mount the FreeIPA CA. The Ceph S3 endpoint uses a FreeIPA-issued cert. Mount ConfigMap ipa-ca-cert (in vxdata-production; real name is ipa-ca-cert not ipa-ca) at /etc/ipa-ca/, pass --ca-cert /etc/ipa-ca/ca.crt to rclone. Do NOT disable cert verification.
• High --transfers for PUT-latency-bound copies. Per-object PUT latency (~0.18s to Ceph) dominates bandwidth. Throughput scales with concurrency; --transfers 24 --checkers 32 was clean. --transfers 16 showed occasional gateway 502s.
• --fast-list is wrong for slow source listings. It buffers the entire recursive listing before moving a byte; the pod sits at 0 B for many minutes on slow MinIO enumerations. Use default traversal so it transfers as it discovers.
• Verify in-cluster, not over VPN. Laptop-over-VPN rclone check produced thousands of spurious “differences” (really i/o timeout listing errors against MinIO). In-cluster verification (--one-way --size-only) came back 0 differences for all folders.

Run as one-shot Pods (restartPolicy: Never), configure via RCLONE_CONFIG_* env vars (no config file). Creds from secrets in vxdata-production: ceph-vxdata-bucket-credentials exposes S3_ACCESS_KEY/S3_SECRET_KEY for the new Ceph endpoint (unverified: secret not in git, name from 2026-07 migration notes).

Production deployment contract (verified 2026-08-06)

https://data.fra.virdx.dev/ (10.10.0.104) in namespace vxdata-production, managed by Argo CD, TLS from the vxdata-production-tls secret.

All runtime manifests live in infra_k8s/current/vxdata-production/ (api.yaml, edge.yaml, backup-cronjob.yaml, …) — not in mono. An older guide claimed mono/apps/vxdata-api/deployments/k8s owned runtime resources; that directory is gone. mono owns the app, its image, and its dev stack; infra_k8s owns everything the cluster runs. The deployed image is tagged with the app version (zot.fra.virdx.dev:5000/vxdata-api:2.1.0 as of 2026-07-30), not a mutable mono:*-latest tag.