Skip to content

2026-08-31 ClearML OSS machine credential research

Researched noninteractive API credential provisioning in ClearML Server OSS and its Helm charts.

Work Done

  • Reviewed the local infra_k8s/current/clearml/, ClearML Argo CD application, and run-image workflow credential injection. The deployment pins Helm chart 7.14.7 and server 2.1.0-664; its clearml-credentials-{production,staging} Secrets are consumed but provisioned out of band.
  • Searched the web through DuckDuckGo after the configured Serper tool was unavailable, then verified results against official ClearML documentation, blog posts, server source, Helm chart source, and SDK source.
  • Confirmed that OSS supports declarative machine principals through secure.credentials.<name> with configured roles and access/secret keys.
  • Confirmed that Docker Compose exposes this mechanism through CLEARML_AGENT_ACCESS_KEY and CLEARML_AGENT_SECRET_KEY for the built-in admin services_agent identity.
  • Identified the Kubernetes pattern: inject CLEARML__secure__credentials__... values into apiserver.extraEnvs from a Kubernetes Secret, then reuse that Secret for agents through CLEARML_API_*.
  • Distinguished Enterprise-only ClearML Service Accounts from OSS config-managed principals.
  • Reviewed fixed users, the public self-only auth.create_credentials API, internal admin APIs, secret-return behavior, and Helm agent credential consumption.
  • Sent the parent a cited report with source permalinks and deployment recommendations.

Lessons Learned: Pitfalls

  • Official user docs focus on UI-created credentials and can obscure the server-configured secure.credentials mechanism.
  • “Service account” is overloaded: a Kubernetes ServiceAccount, an Enterprise ClearML Service Account, and an OSS config-managed API identity are different things.
  • The OSS auth.create_user and auth.get_token_for_user endpoints exist but are marked internal. They should not be treated as the preferred stable bootstrap contract.
  • Web search was unavailable because the Serper connection was not configured. Current official repositories provided stronger primary evidence anyway.

Lessons Learned: Improvements

  • Add an infrastructure note if ClearML is adopted: provision one least-privilege user-role machine principal from a Kubernetes Secret through apiserver.extraEnvs; reserve the built-in admin services_agent identity for genuine bootstrap/admin work.
  • Record that config-managed identities are protected from API credential creation/revocation. Rotate them through configuration and a rollout.
  • Keep the server-side provisioning step separate from the agent chart. The agent chart consumes existing credentials but does not create them.
Navigation

Type to search…

↑↓ navigate↵ selectEsc close