---
title: "vxData production DB restore + cluster storage gotchas"
description: "Postgres restore-from-S3-backup runbook, plus the topolvm-no-replication, ArgoCD self-heal, Argo 128KB ARGO_TEMPLATE limit, and nginx worker_processes gotchas from the 2026-07-16 outage."
image: "https://docs.virdx.dev/img/virdx-social-card.png"
---

> Documentation Index
> Fetch the complete documentation index at: https://docs.virdx.dev/llms.txt
> Use this file to discover all available pages before exploring further.

# vxData production DB restore + cluster storage gotchas

Incident 2026-07-16: `compute-002` was silently reprovisioned overnight
(fresh, older OS image than its siblings). `lvm` storageclass (topolvm.io =
**node-local LVM, no replication, reclaim policy `Delete`**) meant the node
rebuild destroyed vxData's and Gitea's postgres volumes; the StatefulSets
auto-created fresh empty PVCs, postgres re-initialized blank, and Alembic
rebuilt an empty schema (36 tables, 0 rows) with no error anywhere.

**Any DB on `lvm` storageclass is one node rebuild away from silent total
loss.** It needs external backups (vxData/Gitea already have S3 backups) and
ideally replication or a non-ephemeral backend — this is not yet fixed
cluster-wide.

## Restore runbook (Postgres from Ceph S3 backup)

1. Locate the dump: bucket `vxdata/vxdata-production-backups`, endpoint
   `s3.fra.virdx.dev`, creds in secret `ceph-vxdata-bucket-credentials`.
   Download with `mc --insecure` (self-signed cert).
2. Run a one-shot Job: initContainer downloads the dump via `mc`; main
   container `postgres:17-alpine`:
   - terminate active sessions on the target DB,
   - `DROP SCHEMA public CASCADE`,
   - `pg_restore --single-transaction --exit-on-error` the dump.
3. Do this **against the live DB** — don't try to quiesce the app first.

## ArgoCD app-of-apps fights you

`vxdata-production` is owned by the `root` app. Patching its
`syncPolicy.automated` to `null` (to scale the API down during restore) was
reverted within seconds by root's self-heal. Don't try to scale/edit
ArgoCD-managed resources live during an incident — restore against the live
DB instead (see above).

## Backups can fail silently

The 03:00 UTC backup CronJob failed the day of the outage — `pg_dump`
exited in <1s (DB already unreachable) and produced no file, with no alert.
**A zero-byte/failed backup should page** — this alerting does not exist
yet.

## Argo `ARGO_TEMPLATE` 128KB ceiling → spurious `configmaps is forbidden`

Argo passes each pod's resolved template via the `ARGO_TEMPLATE` env var.
Past **131072 bytes** (`maxEnvVarLen`), Argo offloads it into a ConfigMap
(upstream Argo PR #12325, gated behind helm value
`controller.rbac.writeConfigMaps`). Our workflow-controller ClusterRole only
grants get/watch/list on configmaps, not create — so a large inlined job
spec (e.g. a 340KB base64 blob passed as a `cmd` param) fails with
`configmaps is forbidden`, which looks like an RBAC regression but is a
symptom of passing large params inline. Fix by either:

- passing the input by reference (file/S3), not as an inline CLI arg — the
  correct fix for large job specs, or
- enabling `controller.rbac.writeConfigMaps: true` in the Argo Workflows
  helm chart (not yet enabled).

## `worker_processes auto` + 512-core nodes = OOM crashloop

nginx's `worker_processes auto` reads the **host's** core count via
`sysconf(_SC_NPROCESSORS_ONLN)` — it is not cgroup-CPU-limit-aware
(long-standing upstream issue, nginx trac #1151 / nginx#855, open since
2018). On virdx's 512-core nodes this forks **512 workers**, each
preallocating buffers, which OOMKills a small-limit pod (observed: a
512Mi single-replica edge proxy for `data.fra.virdx.dev`, OOMKilling every
~1 minute after traffic resumed post-restore).

- `NGINX_ENTRYPOINT_WORKER_PROCESSES_AUTOTUNE=1` **does not work** on pods
  with `readOnlyRootFilesystem: true` — the tuner needs to rewrite
  `/etc/nginx/nginx.conf` and fails (silently) under a read-only rootfs. All
  virdx edge proxies run read-only-rootfs, so this env var is the wrong fix
  here even though it looks like the obvious one.
- **Correct fix**: mount a full `nginx.conf` with `worker_processes`
  pinned to an explicit small number, via a `subPath` volume mount (the
  pattern `stitoo`/`vxseq` already use). `staging-services` instead uses a
  sed-to-`/tmp` hack. These should converge on the subPath mount as the one
  correct mechanism — currently there are three different hacks in use
  across services.
- Every new nginx/edge deployment on this cluster must pin
  `worker_processes` explicitly — never rely on `auto`.

## Related PRs (as of 2026-07-16)

- `virdx/infra_k8s#154` (merged): edge replicas 1→2, `maxUnavailable: 0`,
  memory 512Mi→2Gi — buys headroom but does **not** fix the worker-count
  root cause (its autotune env var is a no-op here).
- `virdx/infra_k8s#155` (open at time of writing): mounts the correct
  `nginx.conf` via subPath — the actual fix; not yet merged, so the edge
  currently relies on the #154 replica/memory cushion.

## Private registry images need matching `imagePullSecrets`

(2026-03-30 incident): Removing `imagePullSecrets` from deployments pulling
from `ghcr.io/virdx/*` private repos caused immediate `ImagePullBackOff` in
staging/production/public namespaces. **GHCR package visibility = private** →
kubelet 401s without a valid credential → pod stays `Pending` forever.

**Fast diagnosis:** `kubectl describe pod <name>` → events show
`ghcr.io/virdx/<image>: not found` (misleading message; the image exists,
auth failed). ArgoCD app shows `Synced` + `Degraded` (sync succeeded but
pods won't start).

**Fix:** ensure `spec.imagePullSecrets: [{name: ghcr-pull}]` (or the
registry-specific pull secret) exists in every deployment/statefulset/job
pulling private images. The secret must live in the same namespace as the
pod.

## Registry mirrors are not pull-through caches

(2026-04-01 incident, resolved 2026-04-21): k3s nodes mirrored `ghcr.io` to
a local `registry/registry` deployment via `/etc/rancher/k3s/registries.yaml`.
The mirror had no pull-through / remote-proxy config, so it returned `404`
for any repo not already in its local catalog. kubelet never reached GHCR
auth; it failed against the mirror first.

**The local registry mirror was removed 2026-04-21** (infra_k8s commit
9f2af4f). If image pulls fail again with mysterious 404s on images that
definitely exist upstream:

1. `kubectl debug node/<node> --image=busybox:1.36 -- chroot /host cat /etc/rancher/k3s/registries.yaml` — check whether a mirror/rewrite rule exists.
2. Use `crictl` on the node to test the pull path: `sudo crictl pull <image>` will show whether it's hitting a mirror or going direct.
3. Check the mirror registry logs (if it exists): `kubectl -n <mirror-ns> logs deploy/<mirror> --since=5m` for 404 responses.

A registry mirror with no upstream fallback is a purely local cache; it
returns 404 for anything not pre-populated. A pull-through cache proxies
misses to the upstream registry. Don't confuse the two.

## nginx-unprivileged auto-tunes to **host** CPU count (not pod limit)

nginx's `worker_processes auto` reads `sysconf(_SC_NPROCESSORS_ONLN)` (the
host's core count), **not** the pod's CPU limit (nginx trac #1151, open
since 2018). On VIRDX's 512-core nodes this forks 512 workers, each
preallocating buffers → instant OOM on small-memory pods.

- Observed 2026-04-02 in the now-removed `ollama` TLS sidecar
  (`nginxinc/nginx-unprivileged:alpine`, `128Mi` limit, `195` workers, exit
  137).
- `NGINX_ENTRYPOINT_WORKER_PROCESSES_AUTOTUNE=1` **does not work** under
  `readOnlyRootFilesystem: true` — the autotune script rewrites
  `/etc/nginx/nginx.conf` and fails silently. All VIRDX edge proxies run
  read-only rootfs.

**Current fix (as of 2026-07):** all active nginx edge deployments
(`stitoo/edge`, `vxdata-production/edge`, `vxseq/edge`, `staging-services/edge`)
mount a full `nginx.conf` via ConfigMap with `worker_processes` pinned to `2`
(subPath volume mount for stitoo/vxdata/vxseq; sed-to-`/tmp` hack for
staging-services). See `infra_k8s current/vxdata-production/edge.yaml` for
the ConfigMap approach.

**Every new nginx/edge deployment on this cluster must pin `worker_processes`
explicitly — never rely on `auto`.** The gotcha is documented here because
it's a node characteristic, not a service characteristic: any future nginx
pod will hit it.

## Bulk S3/object migration with rclone Pods

(MinIO → Ceph migration, complete 2026-07): the production `vxdata` bucket
now lives at `https://s3.fra.virdx.dev/` (Ceph-backed); the old jumphost
MinIO (`http://192.168.10.101:10000`) is retired for main storage but still
hosts Postgres backup dumps (see `BACKUP_S3_ENDPOINT` in
`vxdata-production/configmap.yaml`).

**Durable in-cluster rclone migration patterns:**

- **Trailing slashes are mandatory.** Source object stores (MinIO, S3) keep
  zero-byte directory-marker objects. `rclone copy src:bucket/prefix
dst:bucket/prefix` (no trailing `/`) HEADs the path, sees a "file", aborts
  with `is a file not a directory`. Always use `src:.../` and `dst:.../`.
- **Mount the FreeIPA CA.** The Ceph S3 endpoint uses a FreeIPA-issued cert.
  Mount ConfigMap `ipa-ca-cert` (in `vxdata-production`; real name is
  `ipa-ca-cert` not `ipa-ca`) at `/etc/ipa-ca/`, pass `--ca-cert
/etc/ipa-ca/ca.crt` to rclone. Do NOT disable cert verification.
- **High `--transfers` for PUT-latency-bound copies.** Per-object PUT latency
  (~0.18s to Ceph) dominates bandwidth. Throughput scales with concurrency;
  `--transfers 24 --checkers 32` was clean. `--transfers 16` showed
  occasional gateway 502s.
- **`--fast-list` is wrong for slow source listings.** It buffers the entire
  recursive listing before moving a byte; the pod sits at `0 B` for many
  minutes on slow MinIO enumerations. Use default traversal so it transfers
  as it discovers.
- **Verify in-cluster, not over VPN.** Laptop-over-VPN `rclone check`
  produced thousands of spurious "differences" (really `i/o timeout` listing
  errors against MinIO). In-cluster verification (`--one-way --size-only`)
  came back `0 differences` for all folders.

Run as one-shot Pods (`restartPolicy: Never`), configure via `RCLONE_CONFIG_*`
env vars (no config file). Creds from secrets in `vxdata-production`:
`ceph-vxdata-bucket-credentials` exposes `S3_ACCESS_KEY`/`S3_SECRET_KEY` for
the new Ceph endpoint (unverified: secret not in git, name from 2026-07
migration notes).

## Production deployment contract (verified 2026-08-06)

`https://data.fra.virdx.dev/` (10.10.0.104) in namespace `vxdata-production`,
managed by Argo CD, TLS from the `vxdata-production-tls` secret.

**All runtime manifests live in `infra_k8s/current/vxdata-production/`**
(`api.yaml`, `edge.yaml`, `backup-cronjob.yaml`, …) — not in `mono`. An older
guide claimed `mono/apps/vxdata-api/deployments/k8s` owned runtime resources;
that directory is gone. `mono` owns the app, its image, and its dev stack;
`infra_k8s` owns everything the cluster runs. The deployed image is tagged with
the app version (`zot.fra.virdx.dev:5000/vxdata-api:2.1.0` as of 2026-07-30),
not a mutable `mono:*-latest` tag.

Source: https://docs.virdx.dev/knowledge/wiki/workstreams/infrastructure/sops/vxdata-db-restore-and-cluster-gotchas/index.mdx