---
title: "2026-09-17 Factory API fsspec hotfix"
description: "inference-api pod crash-looped after mono#403 because the factory app's hand-copied vxdata-sdk conda deps were not updated; hotfix mono#405 plus CI/CD gap analysis."
image: "https://docs.virdx.dev/img/virdx-social-card.png"
---

> Documentation Index
> Fetch the complete documentation index at: https://docs.virdx.dev/llms.txt
> Use this file to discover all available pages before exploring further.

# 2026-09-17 Factory API fsspec hotfix

## Work Done

- Diagnosed `inference-api` (namespace `inference`, image `factory-api:latest`) in
  `CrashLoopBackOff` with `ModuleNotFoundError: No module named 'fsspec'` raised from
  `vxdata/sdk/namespaces/storage.py`.
- Root cause: mono#403 (seekable remote object reader) made the SDK import `fsspec`
  and `aiohttp` and updated the SDK `pyproject.toml` + conda recipe, but the factory
  image installs the SDK as an editable _path_ dependency (`Dockerfile.api` copies
  `packages/vxdata-sdk/src`). The SDK's `[project].dependencies` is empty, so its conda
  runtime deps only reach consumers through the hand-copied list in
  `apps/factory/pyproject.toml` `[tool.pixi.dependencies]`. That list was not updated.
  `aiohttp` was present transitively via `python-kubernetes`; `fsspec` was absent from
  the `api` env (present in `inference` via `histo`, which is why only the API crashed).
- Hotfix: draft PR https://github.com/virdx/mono/pull/405. First revision added `fsspec`
  and `aiohttp` to the mirrored list; on the user's direction it was reworked so factory
  consumes the _released_ `vxdata-sdk` conda package (`>=4.2.0,<5`, virdx channel) instead
  of the path dep. The mirrored dep list and the `COPY packages/vxdata-*` lines in both
  Dockerfiles are gone. Also adds an Nx `test` target running `import factory.api` in the
  `api` env, so `pr_gate.yml`'s `nx affected -t test` executes it. Validated via
  `bunx nx run factory:test`, dev typecheck/lint, check-boundaries.
- Consequence: factory now tracks SDK releases. Only 4.2.0 is published (4.3.0 from #403 is
  not); an SDK change factory needs must go through `vxdata-sdk-release.yml` and a pin bump.
- Blast radius check of the four path consumers of vxdata-sdk: `vxdata-api` and
  `dashboard-api` never import `vxdata.sdk`; `vxdata-jobs` does, but its image installs
  the `histo` env which carries `fsspec` transitively. Its `default`/`dev` envs lack it
  (local breakage only). Follow-up, not in the hotfix.
- CI/CD gap analysis (user explicitly wants CI checks, not moving SDK deps to PyPI):
  `pr_gate.yml` already runs `nx affected` format-check/lint/typecheck/test/build/container
  for Python projects with pixi + Artifactory OIDC. `nx-pixi-plugin` infers the
  vxdata-sdk -> factory edge, so factory _was_ affected on #403, but factory has no
  `test`/`build`/`container` targets and `ty` runs with `unresolved-import = "ignore"`,
  so nothing ever executed an import. `deployments/build.sh` builds `main` with tag
  `latest`, Keel rolls it; there is no smoke test between merge and prod.

## Lessons Learned: Pitfalls

- AGENTS.md in mono states "Python projects are intentionally skipped in CI until pixi
  and Artifactory auth are wired in". This is stale: `pr_gate.yml` sets up pixi with
  JFrog OIDC and runs Python targets via `nx affected`. Fix AGENTS.md.
- `conda-packaging-and-releases.md` SOP was the natural place to look for the
  "path dependency does not carry conda deps" contract, but the contract is only
  documented as a code comment in `apps/factory/pyproject.toml` and
  `apps/dashboard-api/pixi.toml`. (Could not read the SOP itself this session due to
  tool permission denials, so it may already cover it -- verify.)
- `rtk` output filtering garbled multi-pattern `grep -n` results on large lockfiles
  (reported "N matches in N files" for a single file). Use one pattern per call or
  `rtk proxy grep`.

## Lessons Learned: Improvements

- Add to the infrastructure SOPs (probably `conda-packaging-and-releases.md` or a new
  "path dependencies in images" note): every app that installs an in-repo package as
  an editable path dep must mirror that package's `[tool.pixi.dependencies]`; list
  the current consumers (`factory`, `vxdata-jobs`, `vxdata-api`, `dashboard-api`) and
  which deployable env each image installs (`api`, `histo`, ...). When adding a conda
  dep to `vxdata-sdk`, touch all of them in the same PR.
- Document the factory deploy path in one place: `apps/factory/deployments/build.sh`
  -> Tekton `buildkit-image` on `main` -> `zot.fra.virdx.dev:5000/factory-api:latest`
  -> Keel. Nothing in the knowledge base mentions `factory-api` or `inference-api`.
- Proposed CI checks, in order of value/cost:
  1. (DONE in mono#405) Import smoke test per deployable env as an Nx `test` target.
  2. Extend `scripts/check-pixi-locks.js` (already walks path-dep consumers) to also
     assert each consumer's `[tool.pixi.dependencies]` is a superset of the
     dependency's `[tool.pixi.dependencies]` names.
  3. Add an import smoke line to `Dockerfile.api` after `pixi install` (the
     vxdata-jobs Dockerfile already does this for `histo`), so Tekton fails instead of
     the pod.
  4. Stop deploying `:latest` off `main` via Keel without a gate; tag per commit.

Source: https://docs.virdx.dev/knowledge/inbox/2026-09-17-infrastructure-factory-api-fsspec-hotfix/index.mdx