Skip to main content

Setting Up Local Development

This guide walks you through setting up a local development environment for VXPlatform.

Prerequisites​

  • Docker and Docker Compose - for running services in containers
  • Pixi - package manager for Python dependencies (install via curl -fsSL https://pixi.sh/install.sh | bash)
  • Git - for version control
  • Artifactory Token (optional) - for accessing private Python packages

Quick Start​

  1. Install Git hooks:

    ./dev hooks

    This installs pre-commit hooks that auto-generate documentation.

  2. Start the development stack:

    ./dev deploy

    This will:

    • Create a .env file with default settings if it doesn't exist
    • Start the database, API, frontend, and worker services
    • Run database migrations automatically
    • Seed the database with example data

  3. Access the services:

The ./dev Command​

The ./dev script is a task runner that executes scripts from the scripts/ directory.

Available commands:

./dev hooks          # Install pre-commit hooks
./dev deploy         # Start full development stack
./dev db_up          # Start only the database
./dev db_down        # Stop the database

Environment Configuration​

The .env File​

The .env file contains all environment variables used by the development stack. It's automatically created by ./dev deploy if it doesn't exist.

Database Configuration (Internal)​

# Internal container networking
POSTGRES_HOST=localhost
POSTGRES_PORT=5432
POSTGRES_USER=virdx
POSTGRES_PASSWORD=virdx
POSTGRES_DB=vxplatform

These variables configure the PostgreSQL database running inside Docker. The POSTGRES_HOST is localhost when running the API outside Docker, but becomes postgres (the service name) when running inside Docker Compose.

API Configuration (Internal)​

# Used by FastAPI when running outside Docker
FASTAPI_HOST=localhost
FASTAPI_PORT=8000

These configure the FastAPI application's host and port for local development.

Host Port Bindings​

# Ports exposed on your host machine
HOST_URL=localhost
HOST_API_PORT=8000
HOST_FRONTEND_PORT=3000
HOST_POSTGRES_PORT=5432

These control which ports are exposed on your host machine. Containers always use their standard internal ports (8000, 3000, 5432), but these variables let you map them to different host ports if needed (e.g., if port 8000 is already in use).

Storage Configuration​

# Local directory for file storage
STORAGE_LOCAL=./data

This directory is mounted into all containers at /mnt/storage. Any files uploaded to the platform are stored here. The containers reference it via the STORAGE_MOUNT environment variable.

Worker Configuration​

# Prefect workflow orchestration
PREFECT_API_URL=http://localhost:4200

URL for the Prefect API server used by the data worker for workflow orchestration.

Docker Compose Settings​

# Project name for docker-compose
COMPOSE_PROJECT_NAME=vxplatform-local
DEPLOYMENT_NAME=local

The COMPOSE_PROJECT_NAME prefixes all container names and isolates this stack from others on your system.

Example .env File​

See .env.example in the project root for a complete example with all available options.

The Deployment Script​

The bash scripts/deploy.sh script (called via ./dev deploy) manages the full development lifecycle.

Basic Usage​

# Start services (stops existing ones first)
./dev deploy

# Rebuild images and start services
./dev deploy --rebuild

# Start in detached mode (background)
./dev deploy --detached

What It Does​

  1. Environment Check: Verifies .env exists, creates default if missing
  2. Cleanup: Stops any existing containers from previous runs
  3. Build (optional): Rebuilds Docker images with --rebuild flag
  4. Start Services: Launches all services defined in docker-compose.dev.yaml

Command-Line Options​

  • -r or --rebuild - Rebuild Docker images before starting
  • -d or --detached - Run services in background instead of foreground

Default Behavior​

Without flags, ./dev deploy:

  • Stops existing containers but keeps images cached
  • Starts services in foreground (you see live logs)
  • Automatically exits and stops services when you press Ctrl+C

When to Use --rebuild​

Rebuild when:

  • You've changed Dockerfile.* or docker-compose.dev.yaml
  • You've modified pixi.toml dependencies
  • You've pulled changes that affect the build process
  • Services fail to start due to dependency issues

Docker Compose Configuration​

The docker-compose.dev.yaml file defines the development stack.

Services​

vxplatform-api (Backend API)​

ports:
  - "$:8000"
volumes:
  - ./apps/api/src:/app/apps/api/src
  - ./packages:/app/packages
  - $:/mnt/storage:rw
  • Hot Reload: Source code is mounted, Uvicorn watches for changes
  • Migrations: Run automatically on startup via RUN_MIGRATIONS=true
  • Database Seeding: Enabled by default via RUN_SEED=true
  • Storage: Local ./data directory mounted at /mnt/storage

Environment variables you can override:

  • APP_MODULE - Python module path (default: api.app:app)
  • LOG_LEVEL - Logging verbosity (default: debug)
  • RUN_SEED - Whether to seed database on startup (default: true)

vxplatform-frontend (React UI)​

ports:
  - "$:3000"
volumes:
  - ./apps/frontend:/app
  - frontend_node_modules:/app/node_modules
  • Hot Reload: Vite dev server with instant updates
  • Node Modules: Persisted in Docker volume to avoid local conflicts
  • Environment: VITE_API_URL points to backend API

postgres (Database)​

ports:
  - "$:5432"
volumes:
  - type: tmpfs
    target: /var/lib/postgresql/data
  • Temporary Storage: Uses tmpfs (RAM) - data lost on restart
  • Fast Performance: No disk I/O overhead for development
  • Health Checks: Other services wait for database readiness

Note: For persistent data across restarts, replace tmpfs with a named volume in production deployments.

worker (Data Processing)​

environment:
  - PREFECT_API_URL=$
  - VXP_API_URL=http://vxplatform-api:8000
volumes:
  - ./apps/worker/src:/app/apps/worker/src
  - ./apps/worker/flows:/app/apps/worker/flows
  • Workflow Orchestration: Connects to Prefect API
  • Hot Reload: Source code mounted for development
  • Platform Integration: Calls API via internal Docker network

Volume Mounts​

All services mount:

  • Source code - for hot reload during development
  • Packages - shared Python packages across services
  • Storage - shared file storage at /mnt/storage

Health Checks​

Services use health checks to ensure proper startup order:

  1. postgres checks pg_isready
  2. vxplatform-api checks /health endpoint
  3. vxplatform-frontend and worker wait for API health

Docker Build Process​

Multi-Stage Builds​

All Dockerfiles use multi-stage builds for efficiency:

  1. Builder Stage: Uses Pixi to install dependencies
  2. Runtime Stage: Copies installed environment to slim Python image

Backend Dockerfile​

# Stage 1 - Install dependencies with Pixi
FROM ghcr.io/prefix-dev/pixi:0.49.0 AS builder
RUN pixi install -e dev

# Stage 2 - Runtime with installed environment
FROM python:3.12-slim AS runtime
COPY --from=builder /app /app

Benefits:

  • Final image doesn't include Pixi or build tools
  • Smaller image size and faster deployments
  • Consistent dependency resolution via lockfiles

Artifactory Authentication​

Private Python packages require authentication:

# Create token file
echo "your-token-here" > ~/.artifactory.token

# Docker Compose mounts it as a secret
secrets:
  artifactory_token:
    file: $HOME/.artifactory.token

The builder stage uses this to authenticate with pixi auth login.

Development Workflow​

1. Initial Setup​

# Clone repository
git clone <repo-url>
cd data-platform

# Install hooks
./dev hooks

# Start services
./dev deploy

2. Making Changes​

Backend Changes​

Edit files in apps/api/src/ or packages/ - Uvicorn auto-reloads.

Frontend Changes​

Edit files in apps/frontend/src/ - Vite reloads instantly in browser.

Database Migrations​

Create a new migration:

cd apps/api
pixi run -e dev alembic revision -m "description"

Apply migrations (happens automatically on container restart):

pixi run -e dev alembic upgrade head

Worker Changes​

Edit files in apps/worker/src/ or apps/worker/flows/ - restart worker service to apply.

3. Testing Changes​

# Run backend tests
cd apps/api
pixi run -e dev pytest

# Run frontend tests
cd apps/frontend
npm test

4. Viewing Logs​

# All services (when running in foreground)
./dev deploy

# Specific service logs (when detached)
docker compose -p vxplatform-local logs -f vxplatform-api
docker compose -p vxplatform-local logs -f vxplatform-frontend

5. Stopping Services​

# If running in foreground
Press Ctrl+C

# If running detached
docker compose -p vxplatform-local down

Common Tasks​

Reset Database​

# Stop and remove database container
./dev db_down
docker compose -p vxplatform-local rm -f postgres

# Start fresh database (migrations run automatically)
./dev deploy

Rebuild Everything​

./dev deploy --rebuild

Access Database Directly​

docker exec -it vxplatform-local-postgres-1 psql -U virdx -d vxplatform

Run API Without Docker​

cd apps/api

# Start database only
./../../dev db_up

# Update .env to point to localhost
# POSTGRES_HOST=localhost

# Run API with Pixi
pixi run -e dev uvicorn api.app:app --reload --host 0.0.0.0 --port 8000

Clear All Containers and Volumes​

docker compose -p vxplatform-local down -v --remove-orphans

Troubleshooting​

Port Already in Use​

Change port mappings in .env:

HOST_API_PORT=8001
HOST_FRONTEND_PORT=3001
HOST_POSTGRES_PORT=5433

Permission Denied on Storage​

Ensure the storage directory is writable:

mkdir -p ./data
chmod 777 ./data  # or appropriate permissions

Services Not Starting​

Check logs:

docker compose -p vxplatform-local logs

Rebuild if dependencies changed:

./dev deploy --rebuild

Database Connection Failed​

Ensure database is healthy:

docker compose -p vxplatform-local ps postgres

Check database logs:

docker compose -p vxplatform-local logs postgres

Hot Reload Not Working​

Verify volumes are mounted correctly:

docker compose -p vxplatform-local config

Restart services:

docker compose -p vxplatform-local restart vxplatform-api

Next Steps​