---
title: "Self-hosted LLM inference (historical)"
description: "What happened to the self-hosted Ollama/llama.cpp endpoint and the one durable tuning result from it."
image: "https://docs.virdx.dev/img/virdx-social-card.png"
---

> Documentation Index
> Fetch the complete documentation index at: https://docs.virdx.dev/llms.txt
> Use this file to discover all available pages before exploring further.

# Self-hosted LLM inference (historical)

**The `ollama` deployment is gone** — removed from `infra_k8s` on 2026-05-29,
along with its Argo CD Application and manifests. The April 2026 plan to cut it
over to `llama.cpp` in place left no trace in the repo, so treat it as never
completed. An `inference.fra.virdx.dev` service exists today, but it is the
`mono/apps/inference` service (histo preprocessing, segmentation), not an
OpenAI-compatible LLM endpoint. There is currently no documented self-hosted
chat/completions endpoint (verified 2026-08-06).

Two results from that episode are worth keeping:

- **Concurrency hurts at long context on one GPU.** Raising llama.cpp
  `--parallel` from 1 to 8 dropped throughput from 158.5 to 39.9 tok/s for a
  coding-agent-shaped workload (~23k prompt tokens, 128 max tokens) on a
  single-GPU deployment (measured 2026-04-28). Slots split the KV cache, so
  more of them is a loss unless requests are short. Size `--parallel` to the
  real request shape, not to the number of users.
- **Serving shape, if this is ever redone.** Keep namespace/Service/Ingress and
  cert stable and swap only the container; let `llama-server` terminate TLS
  itself (`--ssl-cert-file`/`--ssl-key-file`) instead of adding an nginx
  sidecar (see the `worker_processes` gotcha in
  [sops/vxdata-db-restore-and-cluster-gotchas.md](/knowledge/wiki/workstreams/infrastructure/sops/vxdata-db-restore-and-cluster-gotchas));
  use `strategy: Recreate` for any GPU-requesting Deployment, since a rolling
  update deadlocks on the single GPU.
- **Client side**, `pi` consumes any OpenAI-compatible endpoint via a provider
  entry in `~/.pi/agent/models.json` (`openai-completions` API plus the
  `supportsDeveloperRole` / `supportsReasoningEffort` / `thinkingFormat`
  compat flags).

Source: https://docs.virdx.dev/knowledge/wiki/workstreams/infrastructure/self-hosted-inference/index.mdx