2026-07-16 vxdata prod outage db restore argo edge

Work Done

Three linked incidents on the virdx cluster (k3s, ArgoCD app-of-apps, 512-core nodes), all traceable to an overnight node event.

1. vxData production DB wiped and restored. data.fra.virdx.dev returned 0 resources. Root cause: compute-002 was reprovisioned overnight (52m-old node, older OS image than siblings). vxData postgres + gitea postgres both use storageclass lvm (topolvm.io = node-local LVM, no replication, reclaim Delete). The node rebuild destroyed the local volumes; StatefulSet auto-created fresh empty PVCs (94m old), postgres re-initialized blank, alembic rebuilt the empty schema → 36 tables, 0 rows. Restored from Ceph S3 backup vxdata_20260715_030010_dow3.dump (bucket vxdata/vxdata-production-backups, endpoint s3.fra.virdx.dev, secret ceph-vxdata-bucket-credentials, needs mc --insecure). Ran a one-shot Job (initContainer mc download → postgres:17-alpine: terminate sessions, DROP SCHEMA public CASCADE, pg_restore --single-transaction --exit-on-error). Recovered ~4.1M rows (922K resources). Data loss window: 2026-07-15 03:00 UTC → wipe (~28h). Today’s 03:00 backup had failed (pg_dump exited in <1s = DB already unreachable).

2. Argo Workflows configmaps is forbidden (Linde’s panda preprocessing). inference-histo-preprocessing failed: workflow controller SA (argo:argo-workflows-workflow-controller) can’t create configmaps (ClusterRole has only get/watch/list). NOT a regression, NOT related to the outage. Root cause: her cmd param was a 340KB base64 job spec. Argo passes each pod’s resolved template via the ARGO_TEMPLATE env var; when it exceeds 131072 bytes (maxEnvVarLen) Argo offloads it into a ConfigMap (PR #12325, helm-gated behind controller.rbac.writeConfigMaps). Our chart never enabled it. Fix options: (A) shrink the job — pass input by reference, not inline; (B) helm controller.rbac.writeConfigMaps: true. Left for Linde/user to choose; no change made.

3. vxdata-edge nginx OOMKill crashloop → Connection refused on data.fra.virdx.dev. After the DB restore the inference fleet resumed and hammered the single-replica 512Mi edge proxy, which OOMKilled every ~1min → intermittent connection-refused. Root cause: worker_processes auto is not cgroup-CPU-aware (nginx trac #1151 / nginx#855, open since 2018 — reads host core count via sysconf(_SC_NPROCESSORS_ONLN), ignores CPU limit). On 512-core nodes it forked 512 workers (confirmed 513 procs), each preallocating buffers → OOM. Merged PR virdx/infra_k8s#154 (replicas 1→2, maxUnavailable: 0, 512Mi→2Gi) — but its worker fix (NGINX_ENTRYPOINT_WORKER_PROCESSES_AUTOTUNE=1) is a no-op: the image’s tuner rewrites /etc/nginx/nginx.conf, unwritable under readOnlyRootFilesystem: true, so it exits silently. Corrective PR virdx/infra_k8s#155 mounts a full nginx.conf with worker_processes 2 via subPath (the pattern stitoo/vxseq already use). staging-services uses a sed-to-/tmp hack. All three other edge proxies were already guarded; only vxdata-production was missed.

Lessons Learned: Pitfalls

• topolvm (lvm storageclass) is node-local, unreplicated, reclaim Delete. Any node rebuild permanently destroys that node’s postgres data. vxdata-production AND gitea DBs are both on it — a ticking bomb. DBs on topolvm need external backups (they have them) AND ideally replication or a non-ephemeral backend.
• ArgoCD app-of-apps fights you. vxdata-production is owned by the root app; patching its syncPolicy.automated to null to scale down the API was reverted within seconds by root’s selfHeal. Don’t scale/edit ArgoCD-managed resources live — it self-heals. Do restores against the live DB (terminate sessions + single-transaction) instead of trying to quiesce the app.
• NGINX_ENTRYPOINT_WORKER_PROCESSES_AUTOTUNE=1 silently does nothing on readOnlyRootFilesystem: true pods — the tuner needs to write /etc/nginx/nginx.conf. All virdx edge proxies are read-only-rootfs, so the env-var approach is wrong here; mount a full nginx.conf instead. (I shipped the env var first and had to correct it — check the rootfs mode before recommending autotune.)
• worker_processes auto in any container on the 512-core virdx nodes = 512 workers. Every new nginx/edge deployment must pin worker_processes explicitly.
• Argo ARGO_TEMPLATE env has a 128KB ceiling — large inlined params (base64 blobs) silently trigger a ConfigMap-offload code path that needs RBAC we don’t grant. Pass big job specs by reference (file/S3), not as CLI args.
• The vxData backup CronJob fails silently — the 03:00 dump exited without producing a file and nothing alerted. Backups that produce zero bytes should page.

Lessons Learned: Improvements

• wiki/workstreams/infrastructure/index.md is still empty (also flagged 2026-07-14) — begin-work step 4 dead-ends for infra work.
• Restore runbook worth an SOP: mc (--insecure) + postgres:17-alpine Job, creds in ceph-vxdata-bucket-credentials, dumps at s3.fra.virdx.dev/vxdata/vxdata-production-backups, restore with terminate-sessions + DROP SCHEMA public CASCADE + pg_restore --single-transaction.
• vxdata-edge (and stitoo/vxseq/staging-services) should converge on ONE worker_processes mechanism — currently three different hacks (full-conf mount, sed-to-/tmp, dead env var). The full-conf subPath mount is the clean read-only-rootfs-compatible choice.
• Follow-ups still open: enable Argo writeConfigMaps (or document the input-by-reference pattern for inference); add DB replication/backup-alerting; PR #155 needs merging (until then edge relies on the 2Gi+2-replica cushion).