Self-hosted LLM inference (historical)

What happened to the self-hosted Ollama/llama.cpp endpoint and the one durable tuning result from it.

The ollama deployment is gone — removed from infra_k8s on 2026-05-29, along with its Argo CD Application and manifests. The April 2026 plan to cut it over to llama.cpp in place left no trace in the repo, so treat it as never completed. An inference.fra.virdx.dev service exists today, but it is the mono/apps/inference service (histo preprocessing, segmentation), not an OpenAI-compatible LLM endpoint. There is currently no documented self-hosted chat/completions endpoint (verified 2026-08-06).

Two results from that episode are worth keeping:

Concurrency hurts at long context on one GPU. Raising llama.cpp --parallel from 1 to 8 dropped throughput from 158.5 to 39.9 tok/s for a coding-agent-shaped workload (~23k prompt tokens, 128 max tokens) on a single-GPU deployment (measured 2026-04-28). Slots split the KV cache, so more of them is a loss unless requests are short. Size --parallel to the real request shape, not to the number of users.
Serving shape, if this is ever redone. Keep namespace/Service/Ingress and cert stable and swap only the container; let llama-server terminate TLS itself (--ssl-cert-file/--ssl-key-file) instead of adding an nginx sidecar (see the worker_processes gotcha in sops/vxdata-db-restore-and-cluster-gotchas.md); use strategy: Recreate for any GPU-requesting Deployment, since a rolling update deadlocks on the single GPU.
Client side, pi consumes any OpenAI-compatible endpoint via a provider entry in ~/.pi/agent/models.json (openai-completions API plus the supportsDeveloperRole / supportsReasoningEffort / thinkingFormat compat flags).