---
title: "2026-09-06 vxData production API OOM outage"
description: "Reconstructed the September 5 vxData API outage from edge logs and Prometheus history."
image: "https://docs.virdx.dev/img/virdx-social-card.png"
---

> Documentation Index
> Fetch the complete documentation index at: https://docs.virdx.dev/llms.txt
> Use this file to discover all available pages before exploring further.

# 2026-09-06 vxData production API OOM outage

## Work Done

- Inspected deployments, pods, events, API logs, and both edge-proxy logs in namespace `vxdata-production`.
- Identified one production API incident on 2026-09-05. The last successful edge request was at 17:06:01 UTC, 502 responses ran from 17:06:02 through 17:06:40, and successful responses resumed at 17:06:41 UTC.
- Counted 157 HTTP 502 responses during the 38-second failure window: 147 for `/query/parquet`, five health checks, four `/s3/presign/upload-dir`, and one `/resources/create`. Nginx reported an upstream connection closing, followed by connection refusals to the sole API service endpoint.
- Confirmed from Prometheus that the API container hit its 2 GiB limit and terminated with reason `OOMKilled`. Working-set memory rose from about 1.17 GiB at 15:00 UTC to 1.51 GiB at 16:00, 1.96 GiB at 17:00, and 1.995 GiB immediately before termination. The replacement container started around 17:06:28.
- Confirmed the deployment has one API replica with a 2 GiB memory limit and 600m CPU limit. Both edge replicas proxy to that single backend, so its restart causes complete API unavailability.
- Found one separate edge readiness timeout at 10:08 UTC on 2026-09-06 while an API request took about two seconds; API request logs remained successful and this was not a multi-minute outage.

## Lessons Learned: Pitfalls

- Current API logs for the terminated container were already unavailable and Kubernetes events had expired. Edge logs and Prometheus retained enough evidence to reconstruct the incident.
- The available logs support an approximately 39-second outage, not seven minutes. A separate seven-minute claim needs an exact timestamp or alert source for correlation.
- Edge access logs do not identify the authenticated principal, so the high-rate client could not be attributed from retained logs alone.

## Lessons Learned: Improvements

- Run at least two vxData API replicas so one process OOM does not make production unavailable.
- Investigate the sustained memory growth under repeated query, resource, and upload-presign traffic; raising the memory limit alone only delays recurrence.
- Retain terminated-container logs centrally and include principal/request correlation fields in canonical request events.
- Alert on memory approaching the container limit and on API endpoint count dropping below one ready backend.

Source: https://docs.virdx.dev/knowledge/inbox/2026-09-06-infrastructure-vxdata-api-oom-outage/index.mdx