Inspected the vxdata-production namespace, deployments, pods, Services, Endpoints, events, resource limits, process layout, and six hours of API and edge logs.
Confirmed 17 HTTP 502 responses in two short bursts. Nginx reported connection refusal from the API ClusterIP while the only API pod was failing one-second health probes.
Correlated the incidents with successful but long API requests lasting roughly 6–11 seconds. The API runs one replica with one Uvicorn process, so probe failures can remove the only Service endpoint.
Confirmed the API container had no restarts, OOM events, or meaningful CPU throttling. Both edge containers were currently ready; their recorded restarts were historical.
Confirmed no further 502s in the initial final 20-minute sample, while probe failures and multi-second requests still occurred. The service was available at that point but not stable under load.
A GPT-6 Astra adversarial follow-up found that nginx proxies /health to the API. Edge liveness/readiness therefore amplify an API stall into edge restarts and LoadBalancer endpoint withdrawal. It also observed later 502 connection refusals, 504 upstream timeouts, and parquet requests approaching 19 seconds.
Qualified the route diagnosis: synchronous parquet and MinIO operations inside async handlers are clear structural defects, but completion-only logs cannot prove which route initiated a specific restart without event-loop-lag and phase instrumentation.
Correlated a colleague retry sequence on 2026-09-09: nginx returned 504 for POST /s3/presign/upload-dir at 11:50:28 UTC; both edge containers restarted at 11:50:37 and 11:50:49; their new instances then logged API upstream connection refusals; health recovered around 11:51:05; the retry succeeded through edge at 11:51:43.594 with HTTP 200.
get_minio_client() constructs a new MinIO client without a fixed region for each request. The first presign performs a synchronous GetBucketLocation; MinIO defaults to five-minute connect/read timeouts and retries. This is a plausible initiator for an upload-dir stall but is not proven from completion-only logs.
Lessons Learned: Pitfalls
kubectl top is unavailable to the current user because metrics API access is forbidden. Cgroup counters inside the API container provided current memory, OOM, and throttling evidence instead.
A connection refused from the Service ClusterIP does not by itself distinguish API process restart from endpoint withdrawal. Later evidence confirmed liveness-triggered termination as well as readiness failures.
Edge /health is upstream-coupled rather than nginx-local, so API failures can unnecessarily kill healthy edge processes and remove all LoadBalancer endpoints.
Nginx buffer-to-temp-file warnings are frequent but are not 502 errors; status-line parsing is needed to distinguish them.
Lessons Learned: Improvements
First decouple edge liveness from API health so backend stalls do not kill the public proxy layer.
Run at least two API replicas with topology spread after moving startup migrations into one controlled operation and reviewing concurrent-write constraints.
Review synchronous work in bulk /resources/read, /query/parquet, and S3 control-plane routes; these correlated with probe delays.
Use edge-local liveness. Choose readiness semantics deliberately, and revisit the one-second probe timeout.
Offload synchronous S3 and fully materialized parquet work with bounded concurrency, timeouts, and request/result limits rather than an unbounded generic thread-pool fix.
Add event-loop lag, request-start, active-request, response-size, and per-phase timings because completion-only logs cannot attribute killed requests.
Add alerts for API readiness failures and edge 502 rates; the current incident was reported by a colleague rather than monitoring.
