---
title: "2026-09-09 vxData backup health check"
description: "Production snapshots and staging restore work, but retention failures are masked."
image: "https://docs.virdx.dev/img/virdx-social-card.png"
---

> Documentation Index
> Fetch the complete documentation index at: https://docs.virdx.dev/llms.txt
> Use this file to discover all available pages before exploring further.

# 2026-09-09 vxData backup health check

## Work Done

Read-only inspection of mono/apps/vxdata-api, infra_k8s/current/vxdata-production,
and the live virdx cluster. Production postgres-backup runs daily at 03:00 UTC.
2026-09-09 run completed at 03:01:09 UTC, uploaded
vxdata_20260909_030001_dow3.dump (248339863 bytes) to
s3://vxdata/vxdata-production-backups/ at https://s3.fra.virdx.dev.
S3 listing confirmed 54 nonempty dumps (10.43 GiB), with every date in the
last 30 days represented. Range GET confirmed PGDMP archive header.
Existing 04:00 UTC staging reset fetched this exact dump and restored it with
no reported errors, ending Done at 04:01:01 UTC. No new restore was triggered.

Confirmed defect: upload log ends with `mc: <ERROR> Unable to list folder.
Requested path `` not found`, yet Job is Complete. backup-cronjob.yaml uses
`mc find ... --print | while ...` under `set -eu`, so the left-side failure
is masked. Exact mc error cause remains unverified. All stored objects have
mtime within 30 days (older named dumps were copied August 17); object age
and filename snapshot age are different. No overdue-by-mtime deletion was
therefore demonstrated, despite confirmed cleanup-command failure.

Additional gaps: staging reset explicitly accepts pg_restore exit 1, which
can represent genuine restore errors; backup lacks an active deadline and
repo-defined freshness alert. Daily logical dumps are not WAL/PITR, blob,
role, or secret backups. Independent Ceph protection was not audited.
No mono/infra working files or cluster resources were modified.

## Pitfalls

Production configmap BACKUP*S3*\* and older docs still describe retired MinIO,
while the CronJob explicitly targets Ceph. Do not infer backup location from
those unused config keys. pg_restore --list only validates archive TOC, not
full restore. Job Complete alone is insufficient evidence.
The app container requires /app/apps/vxdata-api/.pixi/envs/production/bin/python
for read-only MinIO inspection; python is not on exec PATH.

## Improvements

Fix retention invocation and propagate pipeline errors; make staging restore
fail on any pg_restore error. Add freshness/failure alerts and job deadlines.
Update obsolete backup endpoints and make restore target guidance explicit:
mono deployment guide seeding example currently names the production DB.
Use a dedicated isolated restore validation before claiming disaster-recovery
readiness. Keep data inspection value-free: object metadata and operational
logs sufficed here; no patient records were accessed or exported.

Source: https://docs.virdx.dev/knowledge/inbox/2026-09-09-infrastructure-vxdata-backup-health-check/index.mdx