Deployment | VIRDX

There are two deployments of vxData on our Kubernetes cluster:

production — available at https://data.fra.virdx.dev/.
staging — available at https://staging.fra.virdx.dev/vxdata.

Additionally, you can run a dev deployment locally on your machine via process-compose. See below for more.

At its core a deployment is a frontend/edge, an API service, and a Postgres database. File (blob) data is not managed by this repo; it lives in our central Ceph and the API hands clients a target bucket to read/write against.

Production (Kubernetes)

The production stack is defined by the flat manifests in infra_k8s/current/vxdata-production/ (moved out of this repo) and synced by ArgoCD. Two Application manifests in infra_k8s/current/argo-cd/ drive the namespace:

vxdata-production.yaml — the manifests in infra_k8s/current/vxdata-production/.
vxdata-production-db.yaml — the Bitnami postgresql chart (Postgres StatefulSet/Service/PVC). Its Service vxdata-production-db-postgresql is the POSTGRES_HOST in configmap.yaml.

Cluster contract:

namespace: vxdata-production
public URL: https://data.fra.virdx.dev/
public IP: 10.10.0.104 (MetalLB LoadBalancer on the edge Service)

Components (infra_k8s/current/vxdata-production/):

api.yaml — API Deployment + ClusterIP Service on port 8000.
edge.yaml — nginx TLS terminator + MetalLB LoadBalancer for the public URL.
backup-cronjob.yaml — nightly pg_dump to S3.
configmap.yaml — non-secret env config, including the S3 S3_ENDPOINT/bucket for blob data.
ipa-ca-cert.yaml — FreeIPA CA mounted for TLS trust against the internal S3 endpoint. Secrets are created out-of-band in the cluster (not tracked in git).

Production contains sensitive patient data and therefore must never be accessed by internet-connected LLMs or other public-facing processes.

Access control and the token migration

The API resolves every request to a principal via API_PRINCIPALS_FILE, which points at principals.json from the vxdata-production-principals secret (vxdata-staging-principals in services-staging), created out-of-band like every other secret here. It is a single table of token to principal:

{
  "tokens": {
    "<secret-token>": {"subject": "read-bot", "grants": [...]},
    "": {
      "subject": "anonymous",
      "grants": [
        {"action": "read",  "payload_types": ["*"], "licenses": ["*"], "access_levels": ["*"]},
        {"action": "write", "payload_types": ["*"], "licenses": ["*"], "access_levels": ["*"]}
      ]
    }
  }
}

The empty-string token is the transitional entry: it is the principal a request with no Authorization header resolves to, so callers on pre-token SDK builds keep working unchanged. It is an ordinary entry in every other respect — no special case in the code. A request carrying a token still gets that token’s policy, and a token that is present but unknown is still a 401; only the bare request falls back. "" can never collide with a real token, because HTTPBearer reports no credentials at all for an empty header value.

Staging and production currently run with an anonymous master grant, which means access control is advisory there: any caller can drop its token and get full access. The API logs anonymous_access_enabled at startup as a standing reminder.

Turning enforcement on

Every request logs principal.subject, and pre-token callers show up as anonymous. Use that to find who still needs migrating:

kubectl -n vxdata-production logs deployment/api \
  | jq -r 'select(.["principal.subject"] == "token:anonymous")
           | [.["http.client_ip"], .["http.user_agent"], .["http.route"]] | @tsv' \
  | sort -u

Once that comes back empty, drop the empty-string token from the secret and restart. load_principals runs once in the lifespan, so editing the secret alone changes nothing until the pod restarts:

kubectl -n vxdata-production create secret generic vxdata-production-principals \
  --from-file=principals.json=./principals.json --dry-run=client -o yaml \
  | kubectl apply -f -
kubectl -n vxdata-production rollout restart deployment/api

The flip is instant and all-or-nothing: every remaining bare caller starts getting 401s at once. Re-adding the key and restarting reverses it.

Releasing a new image

The script at apps/vxdata-api/deployments/build.sh submits a Tekton buildkit pipeline to the Cluster. The current version of the API, as defined in the pyproject.toml in your current checkout, will be included the build request: The image will be built to zot.fra.virdx.dev:5000/vxdata-api:{VERSION} and will use the current state of main on the remote!

Ergo: to release a new API version, merge all your changes into main, make sure to have bumped the version number in pyproject.toml by hand, and then run

bash deployments/build.sh                              # build the latest tag from main

Builds tagged with the same version number will be cached.

Once this image is uploaded to the registry, open a PR in infra_k8s that bumps the requested image version number for the deployed API image:

In current/vxdata-production/api.yaml, specify the desired version for the production deployment
In current/staging-services/vxdata.yaml, specify the desired version for the staging deployment.

Once that PR is merged, ArgoCD will deploy the new image within 5 minutes.

Backups

The nightly CronJob (0 3 * * *) writes a pg_dump to s3://vxdata-production-backups/ and prunes the bucket itself: dumps older than 30 days are deleted on every run, except Sunday dumps (suffix _dow0) which are kept indefinitely.

Seeding from a backup

To seed a Postgres target from the latest S3 dump, kubectl port-forward against service/vxdata-production-db-postgresql and pipe a dump through pg_restore --clean --if-exists.

Dev Deployment

The local dev deployment runs natively on the developer’s machine via process-compose — no Docker. It is defined by apps/vxdata-api/deployments/dev/process-compose.yaml and reads apps/vxdata-api/deployments/dev/.env and .env.secrets for ports and credentials. The stack supervises Postgres, a local MinIO, and the API; all runtime state (Postgres data, MinIO data, sockets, logs) lives under apps/vxdata-api/deployments/dev/.runtime/. The API runs with uvicorn --reload, so source edits hot-reload automatically.

Run from apps/vxdata-api/:

just deploy                                   # start the native Postgres + MinIO + API stack
just inject-mock                              # seed the running stack with mock data
just import-backup-file --path=./my.dump      # restore a local pg_dump into dev Postgres
just cleanup                                  # stop the stack and wipe .runtime

just is intentionally limited to local development. Production operations are handled via the Kubernetes manifests and ArgoCD.