Inspected deployments, pods, events, API logs, and both edge-proxy logs in namespace vxdata-production.
Identified one production API incident on 2026-09-05. The last successful edge request was at 17:06:01 UTC, 502 responses ran from 17:06:02 through 17:06:40, and successful responses resumed at 17:06:41 UTC.
Counted 157 HTTP 502 responses during the 38-second failure window: 147 for /query/parquet, five health checks, four /s3/presign/upload-dir, and one /resources/create. Nginx reported an upstream connection closing, followed by connection refusals to the sole API service endpoint.
Confirmed from Prometheus that the API container hit its 2 GiB limit and terminated with reason OOMKilled. Working-set memory rose from about 1.17 GiB at 15:00 UTC to 1.51 GiB at 16:00, 1.96 GiB at 17:00, and 1.995 GiB immediately before termination. The replacement container started around 17:06:28.
Confirmed the deployment has one API replica with a 2 GiB memory limit and 600m CPU limit. Both edge replicas proxy to that single backend, so its restart causes complete API unavailability.
Found one separate edge readiness timeout at 10:08 UTC on 2026-09-06 while an API request took about two seconds; API request logs remained successful and this was not a multi-minute outage.
Lessons Learned: Pitfalls
Current API logs for the terminated container were already unavailable and Kubernetes events had expired. Edge logs and Prometheus retained enough evidence to reconstruct the incident.
The available logs support an approximately 39-second outage, not seven minutes. A separate seven-minute claim needs an exact timestamp or alert source for correlation.
Edge access logs do not identify the authenticated principal, so the high-rate client could not be attributed from retained logs alone.
Lessons Learned: Improvements
Run at least two vxData API replicas so one process OOM does not make production unavailable.
Investigate the sustained memory growth under repeated query, resource, and upload-presign traffic; raising the memory limit alone only delays recurrence.
Retain terminated-container logs centrally and include principal/request correlation fields in canonical request events.
Alert on memory approaching the container limit and on API endpoint count dropping below one ready backend.
