version: ==2024-08-24== It's pretty rare to be working on a single-package project. Given the breadth of workflows different individuals and teams want to adopt, there isn't always a single *best* approach for all projects of all sizes. It might be useful to outline some common and composable development workflows that are useful across various scales of projects and teams so identify what works well and where some functionality might be lacking. # The elephants in the room ## Overloaded terminology Let's get this one out of the way first. Most of the terms we have at our disposal here eg "workspace", "project", "package", "module" etc are heavily overloaded. They might mean something very specific to to one person and something very different to another. - `workspace` when the term is unqualified, means an editor / ide workspace (multiple directories with multiple projects under a single root). `cargo workspace` or `uv workspace` when referring to that more specific concept (ie multiple projects that share a lockfile and dependecies). ## Repository structure I'd very much like for this not to get caught up in details of code repository structure like mono-repo vs per-project repo. I may or may not have multiple projects in the same repo. Maybe I've cloned multiple repos into the same directory and opened that directory in an editor or I might have anything in between e.g. git-submodules, git-subtree, etc. I think tooling should ideally be modular and plug-able on this front. # Desired functionality In no particular order and without dictating how this should work, here are some dev lifecycle features I find myself often wanting: ## Shared dependency locks across multiple projects Multiple projects (e.g. pyproject.toml) that must always be rev'ed together and share a single set of dependency locks. I'm pretty much thinking of cargo workspaces and what uv workspaces is currently defined as. This is also similar to how maven BOMs allow pinning versions and variables across multiple projects ## "Multi-project" workpaces At a high-level, I want some tooling support when multiple projects (eg in single editor workspace) don't share a single lock file. ### Commands that query project info I don't know that I need the full flexibility of Buck or Bazel skylark queries but there are a handful of commands that are useful in multi-project workspaces: - list all projects (e.g. to xargs into a CI run invocation or dependabot version bump) - list all projects that have a source dependency on me (e.g. I make a change to some lib and want to run tests on all projects that have a source dependency on me). ### Shared tool configuration across multiple projects Whether or not projects share dependencies via a uv-style workspace or via uv-style path dependencies, it's often very convenient to share other project settings e.g. ruff code style settings, pytest settings (e.g. `-Werror`) etc. There might be different ways to achieve this. - ruff's ability to have an explicit import from a common file into a project's `pyproject.toml`. This could maybe be streamlined with an `include parent` where you still have to opt in to inheriting workspace-wide tool configs but you don't have to explicitly type the include path. - if a project was created from a template system like cookie cutter, you can sync to keep a project up to date. - some other tool that's able to check that multiple projects are using the same settings. ### Progressive roll out of configs There needs to be a graceful way to progressively roll out a change to linting or type checking configs. E.g. If by default I want all my projects to share the same ruff rules and I want to roll out a new rule that requires a bunch of changes, I want to be able to gradually opt projects and ensure all new projects adhere to the new setting. ## Consistent lifecycle commands by default It's useful to have at least a core set of commands that are standard in any project so you know how to get up and running e.g. `install`, `test`, `fmt`, `check`, `run`. The ability to define arbitrary entrypoints e.g. `package.json` `scripts` section, while useful, means many projects have a different approach to things and requires documentation. One of the things I think tools like `blaze`, `bazel`, `buck`, have done really well is separate the concept of a target, which might be a lib or a runnable from the actions you want to do on it (build, test, check, run, publish). Cargo also does this to a certain extent. This doesn't need to preclude the more ad-hoc and flexible shell script type of tools or task runners like `Poe The Poet` or `Just` but it does mean that there's a sane default starting point for most projects. ### Local Dev is the same as CI CI tends to work best when it does little magic. Specifically, I want to be able to run the same `check` and `test` commands locally and in CI. ## Build-time dependencies (toolchains) It's common to have dependencies that are only required for dev purposes (e.g. mkdocs, ruff, pyright,...) - Ideally these aren't included in the project's venv (even when running tests) - If I have multiple projects that need the same toolchain version, I'd like to share the same artifacts ## Multi-langauge Python shines as a glue language. I'd like to be able to gracefully interoperate with different languages – this doesn't necessarily mean being a build system for these languages but I want to be able to use the output of a project in one language as a dependency to a python project in a way that feels like source dependencies across python projects. A few important languages come to mind: - Rust / maturin (today, if I use, say, poetry for pure python projects, I have to use a different workflow for a python/rust project) - C + python native bindings - Cython - Mojo # Impedance mismatches Where there's still some rough edges - Editor workspace vs uv workspace - e.g. if I have multiple projects (possibly multiple uv workspaces) how to I more gracefully let vscode, zed, etc discover all those projects and venvs. - LSPs and discovering venvs - lsp start args usually want a venv path. How can we bootstrap the process e.g. something similar to this psuedo shell code `list-projects | xargs which-python`.