Packaging and Releases

Our mono repository is designed to hold lots of different packages and apps, that may depend on one another. Modules in packages may have non-circular dependencies onto other packages in the monorepo. Software in apps may depend on packages from mono/packages, but must never import from other apps. Usually, apps are not distributed as importable packages and are rather deployed as service applications or similar.

Problem Setting

A common pattern (and one of the huge advantages!) in monorepos is that we can make changes across package boundaries. During development, we typically want to work using the live state of code in the repo, meaning that local installs need to be editable. In particular, we do not want to have to publish pre-release packages to our Artifactory to be able to use them in different parts of the codebase. During distribution however, we need to package and release our code in such a way that versioning and dependencies are properly managed.

Solution - nx and pixi

We use nx tool for the management of our monorepo. It takes care of detecting dependency graphs, defining tasks such as build steps, and also provides features such as caching of steps.

pixi is our package and dependency manager of choice. It has recently gained features around the building of packages. We had, until now, used rattler-build for that process. Pixi now integrates with rattler-build under the hood.

This allows us to define separate build and runtime dependencies in the exact same manifest. nx can now much easier manage versions and dependencies, and local installs bring the installed package’s dependencies with them.

Declaring local dependency chains

At the example of the vxdata-schemas and vxdata-sdk packages:

vxdata-sdk depends on schemas this way:

# packages/vxdata-schemas/pyproject.toml --------------------------------------
[project]
# ...
version = "4.5.6"

[tool.pixi.package.build]  # tells pixi to build this package 
backend = { name = "pixi-build-python", version = "0.8.*", channels = ["conda-forge"] }

[tool.pixi.package.host-dependencies]
hatchling = "*"  # needed for building 

[tool.pixi.package.run-dependencies]
pydantic = ">=2.11.7,<3"  # misc conda-forge dependencies

[tool.pixi.dependencies]
vxdata-schemas = { path = "." }  # installs vxdata-schemas locally

# packages/vxdata-sdk/pyproject.toml ------------------------------------------
# [project]
# ...
version = "1.2.3"
[tool.pixi.package.build]  # tells pixi to build this package 
backend = { name = "pixi-build-python", version = "0.8.*", channels = ["conda-forge"] }

[tool.pixi.package.host-dependencies]  # dependencies needed during build
hatchling = "*"
vxdata-schemas = { path = "../vxdata-schemas" }

[tool.pixi.package.run-dependencies]  # dependencies used in local installs and during package publishing!
vxdata-schemas = ">=4.5.6,<5.0a0"  # the important bit: the range the published package requires; locally, schemas still comes from the path above. nx rewrites it on release
pydantic = ">=2.11.7,<3"

[tool.pixi.dependencies]
vxdata-sdk = { path = "../vxdata-sdk" }  # not `path = "."`: pixi bug when a run dependency names a workspace package

When you now run pixi install in the vxdata-sdk workspace, you will notice that there’s a build process of vxdata-schemas happening. This shows you that the actual dependencies of vxdata-schemas are being installed now as well! This was not the case with the previous setup and local editable installs.

Versioning and release process

In general, you should never touch packages’ version numbers yourself, nor should you have to change version ranges of local dependencies yourself. All of this is handled by nx and the release process.

Edit your code across the repo, make sure tests and CI passes. Get the PR into main. If you now want to actually release new versions of Python packages, run the Python Release PR GitHub Action. This will use nx and the commit history on main to determine what versions to bump how. It will open a release PR, which previews the changes. Merge this PR, and automatic actions will build and publish the packages to our Artifactory. Do not merge stale release PRs (the release-pr-guard check flags them).