---
title: "2026-07-16 vxdata prod outage db restore argo edge"
description: "Incident session — restored wiped vxData prod DB from S3 backup, diagnosed Argo configmap-forbidden (128KB template env offload), fixed vxdata-edge nginx OOM crashloop"
image: "https://docs.virdx.dev/img/virdx-social-card.png"
---

> Documentation Index
> Fetch the complete documentation index at: https://docs.virdx.dev/llms.txt
> Use this file to discover all available pages before exploring further.

# 2026-07-16 vxdata prod outage db restore argo edge

## Work Done

Three linked incidents on the `virdx` cluster (k3s, ArgoCD app-of-apps, 512-core nodes), all traceable to an overnight node event.

**1. vxData production DB wiped and restored.**
`data.fra.virdx.dev` returned 0 resources. Root cause: `compute-002` was reprovisioned overnight (52m-old node, older OS image than siblings). vxData postgres + gitea postgres both use storageclass `lvm` (topolvm.io = node-local LVM, **no replication, reclaim `Delete`**). The node rebuild destroyed the local volumes; StatefulSet auto-created fresh empty PVCs (94m old), postgres re-initialized blank, alembic rebuilt the empty schema → 36 tables, 0 rows. Restored from Ceph S3 backup `vxdata_20260715_030010_dow3.dump` (bucket `vxdata/vxdata-production-backups`, endpoint `s3.fra.virdx.dev`, secret `ceph-vxdata-bucket-credentials`, needs `mc --insecure`). Ran a one-shot Job (initContainer mc download → postgres:17-alpine: terminate sessions, `DROP SCHEMA public CASCADE`, `pg_restore --single-transaction --exit-on-error`). Recovered ~4.1M rows (922K resources). Data loss window: 2026-07-15 03:00 UTC → wipe (~28h). Today's 03:00 backup had failed (pg_dump exited in <1s = DB already unreachable).

**2. Argo Workflows `configmaps is forbidden` (Linde's panda preprocessing).**
`inference-histo-preprocessing` failed: workflow controller SA (`argo:argo-workflows-workflow-controller`) can't `create` configmaps (ClusterRole has only get/watch/list). NOT a regression, NOT related to the outage. Root cause: her `cmd` param was a **340KB base64 job spec**. Argo passes each pod's resolved template via the `ARGO_TEMPLATE` env var; when it exceeds **131072 bytes (maxEnvVarLen)** Argo offloads it into a ConfigMap (PR #12325, helm-gated behind `controller.rbac.writeConfigMaps`). Our chart never enabled it. Fix options: (A) shrink the job — pass input by reference, not inline; (B) helm `controller.rbac.writeConfigMaps: true`. Left for Linde/user to choose; no change made.

**3. vxdata-edge nginx OOMKill crashloop → `Connection refused` on data.fra.virdx.dev.**
After the DB restore the inference fleet resumed and hammered the single-replica 512Mi edge proxy, which OOMKilled every ~1min → intermittent connection-refused. Root cause: **`worker_processes auto` is not cgroup-CPU-aware** (nginx trac #1151 / nginx#855, open since 2018 — reads host core count via `sysconf(_SC_NPROCESSORS_ONLN)`, ignores CPU limit). On 512-core nodes it forked **512 workers** (confirmed 513 procs), each preallocating buffers → OOM. Merged PR virdx/infra_k8s#154 (replicas 1→2, `maxUnavailable: 0`, 512Mi→2Gi) — but its worker fix (`NGINX_ENTRYPOINT_WORKER_PROCESSES_AUTOTUNE=1`) is a **no-op**: the image's tuner rewrites `/etc/nginx/nginx.conf`, unwritable under `readOnlyRootFilesystem: true`, so it exits silently. Corrective PR virdx/infra_k8s#155 mounts a full `nginx.conf` with `worker_processes 2` via subPath (the pattern `stitoo`/`vxseq` already use). `staging-services` uses a sed-to-/tmp hack. All three other edge proxies were already guarded; only vxdata-production was missed.

## Lessons Learned: Pitfalls

- **topolvm (`lvm` storageclass) is node-local, unreplicated, reclaim `Delete`.** Any node rebuild permanently destroys that node's postgres data. vxdata-production AND gitea DBs are both on it — a ticking bomb. DBs on topolvm need external backups (they have them) AND ideally replication or a non-ephemeral backend.
- **ArgoCD app-of-apps fights you.** `vxdata-production` is owned by the `root` app; patching its `syncPolicy.automated` to null to scale down the API was reverted within seconds by root's selfHeal. Don't scale/edit ArgoCD-managed resources live — it self-heals. Do restores against the live DB (terminate sessions + single-transaction) instead of trying to quiesce the app.
- **`NGINX_ENTRYPOINT_WORKER_PROCESSES_AUTOTUNE=1` silently does nothing on `readOnlyRootFilesystem: true` pods** — the tuner needs to write `/etc/nginx/nginx.conf`. All virdx edge proxies are read-only-rootfs, so the env-var approach is wrong here; mount a full `nginx.conf` instead. (I shipped the env var first and had to correct it — check the rootfs mode before recommending autotune.)
- **`worker_processes auto` in any container on the 512-core virdx nodes = 512 workers.** Every new nginx/edge deployment must pin `worker_processes` explicitly.
- **Argo `ARGO_TEMPLATE` env has a 128KB ceiling** — large inlined params (base64 blobs) silently trigger a ConfigMap-offload code path that needs RBAC we don't grant. Pass big job specs by reference (file/S3), not as CLI args.
- **The vxData backup CronJob fails silently** — the 03:00 dump exited without producing a file and nothing alerted. Backups that produce zero bytes should page.

## Lessons Learned: Improvements

- `wiki/workstreams/infrastructure/index.md` is still empty (also flagged 2026-07-14) — begin-work step 4 dead-ends for infra work.
- Restore runbook worth an SOP: mc (`--insecure`) + postgres:17-alpine Job, creds in `ceph-vxdata-bucket-credentials`, dumps at `s3.fra.virdx.dev/vxdata/vxdata-production-backups`, restore with terminate-sessions + `DROP SCHEMA public CASCADE` + `pg_restore --single-transaction`.
- `vxdata-edge` (and stitoo/vxseq/staging-services) should converge on ONE worker_processes mechanism — currently three different hacks (full-conf mount, sed-to-/tmp, dead env var). The full-conf subPath mount is the clean read-only-rootfs-compatible choice.
- Follow-ups still open: enable Argo `writeConfigMaps` (or document the input-by-reference pattern for inference); add DB replication/backup-alerting; PR #155 needs merging (until then edge relies on the 2Gi+2-replica cushion).

Source: https://docs.virdx.dev/knowledge/inbox/2026-07-16-infrastructure-vxdata-prod-outage-db-restore-argo-edge/index.mdx