Deployment

Kubernetes cluster deployments

We maintain four deployments of the data-platform, each for different purposes:

• production is the go-to production deployment. It should only contain clean data and should not be written to except for automated, well-tested processes.
• staging acts as the final test-bed for changes before they hit production. Otherwise, the deployment is identical to production.

Both production and staging contain sensitive patient data and therefore must never be accessed by internet-connected LLMs or other public-facing processes.

production, staging instances store blob data on our central S3 store (as of mid-February: our MinIO at jumphost 10000).

Each type of deployment lives in its own Kubernetes namespace:

• data-platform-production: https://production.data.fra.virdx.dev/
• data-platform-staging: https://staging.data.fra.virdx.dev/

Production, staging, and public instances receive the same amount of resources. The previews deployments receive fewer resources, but the overall namespace may grow larger in resource use than the three other ones individually as there may be multiple deployments in parallel.

Injecting the backup into staging

We can trigger the injection of a backup into the staging deployment by running in the CLI:

argo -n argo-workflows submit \
     --from workflowtemplate/data-platform-staging-restore \
     --serviceaccount data-platform-staging-restore-runner \
     -p backup_prefix=staging \
     -p backup_key=latest.dump \  
     -l workflows.argoproj.io/creator-preferred-username=<your.username>

This may take 2 minutes and does break the deployment for the duration of the workflow.

Legacy, docker-compose based deployments

The platform runs as docker-compose-managed Docker containers on the jumphost (192.168.10.101).

The prod deployment is the primary system for people to use. A staging deployment is available for testing new features before pushing them to prod. Additionally, we define a local dev deployment for development and testing. All docker-compose deployment specs live in the repo at apps/vxdata-api/deployments/{env}/. Every deployment has a compose.yaml file that defines containers and services. Staging and production also have deployment-local operational scripts such as deploy.sh, backup.sh, and restore_from_s3.sh.

For both prod and staging, a directory at /opt/virdx/vxplatform exists that contains a checkout of the repo and therefore the deployment specs.

On push to the main branch of this repo, we automatically build Docker images and distribute them via the GitHub Container Registry. At its core, the data platform consists of a frontend service, an API service, and a Postgres database. Storage of file data is provided by our central MinIO deployment and not managed by this repo. Each deployment has a "target S3 bucket" - a bucket address that is provided to the client so it knows where to put files.

A dev deployment can be spun up for local testing on a developer's machine.

Production Deployment

apps/vxdata-api/deployments/frankfurt_prod/compose.yaml:

• Defines three services:
  • frontend at port 2700,
  • api at port 2701,
  • postgres at port 2702.
• Uses GHCR images ghcr.io/virdx/vxplatform-api:{tag} and ghcr.io/virdx/vxplatform-frontend:{tag}.
• The postgres container's data is stored into a mounted disk location.
• Defines the target S3 data destination to be the central MinIO instance in the dataplatform-prod bucket.

apps/vxdata-api/deployments/frankfurt_prod/.env.secrets:

• Contains the secrets for the postgres database as well as S3 access credentials (required by the frontend to visualize files).

How can we trigger deployment? The GitHub action at .github/workflows/deploy_frankfurt_prod.yaml allows manual triggering of a deployment. The workflow calls apps/vxdata-api/deployments/frankfurt_prod/deploy.sh.

How are backups created? There is currently no dedicated production backup workflow in this repo. Operational backup automation currently lives under apps/vxdata-api/deployments/frankfurt_staging/ and .github/workflows/backup_frankfurt_staging.yaml.

Staging Deployment

The staging deployment is nearly identical to production, with the following differences:

• different set of ports: frontend at 2800, api at 2801, postgres at 2802
• stores Postgres data into a named volume - no need for exposing the postgres data on disk here

The purpose of the staging deployment is to provide production-like data, but to allow for more frequent updates in order to test new features before pushing them to prod.

How can we trigger deployment? We re-deploy automatically off of main branch pushes. Additionally, the GitHub action at .github/workflows/deploy_frankfurt_staging.yaml allows manual triggering of a deployment. The workflow calls apps/vxdata-api/deployments/frankfurt_staging/deploy.sh.

How do we populate staging with data? We can trigger the GitHub action at .github/workflows/restore_frankfurt_staging.yaml to populate the staging database from a production backup. This will effectively overwrite any existing data in staging with a copy of production data. References to objects on S3 that are present in the production data will continue to work, because the restored records still point to objects in the dataplatform-prod bucket and staging has read access. The workflow calls apps/vxdata-api/deployments/frankfurt_staging/restore_from_s3.sh.

How are backups created? The GitHub action at .github/workflows/backup_frankfurt_staging.yaml creates routine backups by calling apps/vxdata-api/deployments/frankfurt_staging/backup.sh.

How does this not contaminate prod? The staging database is fully isolated from the production database. Any new files uploaded via staging go to the dataplatform-dev bucket, not dataplatform-prod.

Dev Deployment

The local dev deployment runs natively on the developer's machine via process-compose — no Docker. It is defined by apps/vxdata-api/deployments/dev/process-compose.yaml and reads apps/vxdata-api/deployments/dev/.env and .env.secrets for ports and credentials. Postgres data lives under apps/vxdata-api/deployments/dev/.runtime/. The API runs with uvicorn --reload, so source edits hot-reload automatically.

The dev stack does not include a local MinIO; point S3_ENDPOINT at any reachable S3-compatible endpoint.

Run just deploy from apps/vxdata-api/ to get started. Run just import-backup-file --path=./my-backup.dump to populate this deployment's Postgres from a local pg_dump.

just is intentionally limited to local development. Staging and production operations are handled via deployment-specific scripts and GitHub Actions.