Work Done
Read-only inspection of mono/apps/vxdata-api, infra_k8s/current/vxdata-production, and the live virdx cluster. Production postgres-backup runs daily at 03:00 UTC. 2026-09-09 run completed at 03:01:09 UTC, uploaded vxdata_20260909_030001_dow3.dump (248339863 bytes) to s3://vxdata/vxdata-production-backups/ at https://s3.fra.virdx.dev. S3 listing confirmed 54 nonempty dumps (10.43 GiB), with every date in the last 30 days represented. Range GET confirmed PGDMP archive header. Existing 04:00 UTC staging reset fetched this exact dump and restored it with no reported errors, ending Done at 04:01:01 UTC. No new restore was triggered.
Confirmed defect: upload log ends with mc: <ERROR> Unable to list folder. Requested path `` not found, yet Job is Complete. backup-cronjob.yaml uses
mc find ... --print | while ... under set -eu, so the left-side failure
is masked. Exact mc error cause remains unverified. All stored objects have
mtime within 30 days (older named dumps were copied August 17); object age
and filename snapshot age are different. No overdue-by-mtime deletion was
therefore demonstrated, despite confirmed cleanup-command failure.
Additional gaps: staging reset explicitly accepts pg_restore exit 1, which can represent genuine restore errors; backup lacks an active deadline and repo-defined freshness alert. Daily logical dumps are not WAL/PITR, blob, role, or secret backups. Independent Ceph protection was not audited. No mono/infra working files or cluster resources were modified.
Pitfalls
Production configmap BACKUPS3* and older docs still describe retired MinIO, while the CronJob explicitly targets Ceph. Do not infer backup location from those unused config keys. pg_restore –list only validates archive TOC, not full restore. Job Complete alone is insufficient evidence. The app container requires /app/apps/vxdata-api/.pixi/envs/production/bin/python for read-only MinIO inspection; python is not on exec PATH.
Improvements
Fix retention invocation and propagate pipeline errors; make staging restore fail on any pg_restore error. Add freshness/failure alerts and job deadlines. Update obsolete backup endpoints and make restore target guidance explicit: mono deployment guide seeding example currently names the production DB. Use a dedicated isolated restore validation before claiming disaster-recovery readiness. Keep data inspection value-free: object metadata and operational logs sufficed here; no patient records were accessed or exported.