inference-api pod crash-looped after mono#403 because the factory app's hand-copied vxdata-sdk conda deps were not updated; hotfix mono#405 plus CI/CD gap analysis.
Diagnosed inference-api (namespace inference, image factory-api:latest) in
CrashLoopBackOff with ModuleNotFoundError: No module named 'fsspec' raised from
vxdata/sdk/namespaces/storage.py.
Root cause: mono#403 (seekable remote object reader) made the SDK import fsspec
and aiohttp and updated the SDK pyproject.toml + conda recipe, but the factory
image installs the SDK as an editable path dependency (Dockerfile.api copies
packages/vxdata-sdk/src). The SDK’s [project].dependencies is empty, so its conda
runtime deps only reach consumers through the hand-copied list in
apps/factory/pyproject.toml[tool.pixi.dependencies]. That list was not updated.
aiohttp was present transitively via python-kubernetes; fsspec was absent from
the api env (present in inference via histo, which is why only the API crashed).
Hotfix: draft PR https://github.com/virdx/mono/pull/405. First revision added fsspec
and aiohttp to the mirrored list; on the user’s direction it was reworked so factory
consumes the releasedvxdata-sdk conda package (>=4.2.0,<5, virdx channel) instead
of the path dep. The mirrored dep list and the COPY packages/vxdata-* lines in both
Dockerfiles are gone. Also adds an Nx test target running import factory.api in the
api env, so pr_gate.yml’s nx affected -t test executes it. Validated via
bunx nx run factory:test, dev typecheck/lint, check-boundaries.
Consequence: factory now tracks SDK releases. Only 4.2.0 is published (4.3.0 from #403 is
not); an SDK change factory needs must go through vxdata-sdk-release.yml and a pin bump.
Blast radius check of the four path consumers of vxdata-sdk: vxdata-api and
dashboard-api never import vxdata.sdk; vxdata-jobs does, but its image installs
the histo env which carries fsspec transitively. Its default/dev envs lack it
(local breakage only). Follow-up, not in the hotfix.
CI/CD gap analysis (user explicitly wants CI checks, not moving SDK deps to PyPI):
pr_gate.yml already runs nx affected format-check/lint/typecheck/test/build/container
for Python projects with pixi + Artifactory OIDC. nx-pixi-plugin infers the
vxdata-sdk -> factory edge, so factory was affected on #403, but factory has no
test/build/container targets and ty runs with unresolved-import = "ignore",
so nothing ever executed an import. deployments/build.sh builds main with tag
latest, Keel rolls it; there is no smoke test between merge and prod.
Lessons Learned: Pitfalls
AGENTS.md in mono states “Python projects are intentionally skipped in CI until pixi
and Artifactory auth are wired in”. This is stale: pr_gate.yml sets up pixi with
JFrog OIDC and runs Python targets via nx affected. Fix AGENTS.md.
conda-packaging-and-releases.md SOP was the natural place to look for the
“path dependency does not carry conda deps” contract, but the contract is only
documented as a code comment in apps/factory/pyproject.toml and
apps/dashboard-api/pixi.toml. (Could not read the SOP itself this session due to
tool permission denials, so it may already cover it – verify.)
rtk output filtering garbled multi-pattern grep -n results on large lockfiles
(reported “N matches in N files” for a single file). Use one pattern per call or
rtk proxy grep.
Lessons Learned: Improvements
Add to the infrastructure SOPs (probably conda-packaging-and-releases.md or a new
“path dependencies in images” note): every app that installs an in-repo package as
an editable path dep must mirror that package’s [tool.pixi.dependencies]; list
the current consumers (factory, vxdata-jobs, vxdata-api, dashboard-api) and
which deployable env each image installs (api, histo, …). When adding a conda
dep to vxdata-sdk, touch all of them in the same PR.
Document the factory deploy path in one place: apps/factory/deployments/build.sh
-> Tekton buildkit-image on main -> zot.fra.virdx.dev:5000/factory-api:latest
-> Keel. Nothing in the knowledge base mentions factory-api or inference-api.
Proposed CI checks, in order of value/cost:
(DONE in mono#405) Import smoke test per deployable env as an Nx test target.
Extend scripts/check-pixi-locks.js (already walks path-dep consumers) to also
assert each consumer’s [tool.pixi.dependencies] is a superset of the
dependency’s [tool.pixi.dependencies] names.
Add an import smoke line to Dockerfile.api after pixi install (the
vxdata-jobs Dockerfile already does this for histo), so Tekton fails instead of
the pod.
Stop deploying :latest off main via Keel without a gate; tag per commit.
