Cronnie — scheduled unattended agent sessions

How Cronnie runs pi as Argo CronWorkflows against the MLflow gateway, how sessions are traced and shown in the dashboard, and the cluster traps hit while building it.

Code: mono/apps/cronnie (README has design + how to add a job). Runs in namespace autoresearch as Argo CronWorkflows; no server, no Argo Events. Dashboard page /cronnie.

Contract

Runtime tags every gateway call with X-MLflow-Gateway-Caller: cronnie/<workflow uid>. MLflow stores it as trace metadata mlflow.gateway.caller; the dashboard searches POST /api/3.0/mlflow/traces/search with filter: metadata.`mlflow.gateway.caller` = 'cronnie/<uid>' across the experiments listed by GET /api/3.0/mlflow/gateway/endpoints/list. Argo’s metadata.uid is the only join key.
Workflows carry app.kubernetes.io/name=cronnie and cronnie.virdx.dev/job=<job>. Cron-created workflows need workflows.argoproj.io/creator-preferred-username: autoresearch in workflowMetadata: the cron controller runs as the argo SA and is exempt from the autoresearch identity-injection rule, so without it runs are unattributed.
Pi’s openai-completions API always streams; Bedrock via bearer token refuses streaming. The runtime’s loopback adapter does one non-streaming call and replays it as SSE (same trick as apps/autoresearch-box).

Traps (all hit live, 2026-09-18)

MLflow gateway < 3.13 drops assistant tool_calls when converting to Bedrock Converse: every second turn after a tool call 400s with “toolResult blocks exceed toolUse blocks”. Fixed upstream in 3.13.0 (#23223) / 3.16.1 (#24309). Cluster upgraded to v3.16.1-full (infra_k8s#230); DB migration ran via the chart’s init container, traces/endpoints intact.
Argo needs an explicit command for images in the private registry: the controller cannot read image config (secrets "zot-registry-credentials" is forbidden).
Output artifacts in autoresearch fail: the executor cannot verify s3.fra.virdx.dev (IPA CA, x509: unknown authority) and Argo skips artifacts on mounted volumes anyway. Cronnie keeps no artifacts; the transcript lives in MLflow + pod log. Fix would be a caSecret on the namespace artifact repository.
zot rejects Docker-media-type manifests from docker push (manifest invalid, HTTP 415) and is only reachable via kubectl port-forward -n zot svc/zot. Push with docker buildx build --output type=image,oci-mediatypes=true,push=true,registry.insecure=true through a docker-container builder with network=host.
OTLP encodes JSON null as a kvlist entry without value; decoders must tolerate it.
Local dashboard against live Argo without Keycloak: DASHBOARD_MOCK=true plus DASHBOARD_DEV_TOKEN_COMMAND="kubectl oidc-login get-token ... | jq -r .status.token".