Optimizing Production Images with Poetry Dependency Groups

The Bloated Production Image Problem

A common friction point in Python deployment is the "dev-dependency leak." You need pytest, black, and mypy to ensure code quality during development, but including them in a production Docker image increases the attack surface, slows down build times, and inflates the image size. The challenge is maintaining a single source of truth for all dependencies while ensuring only the runtime essentials reach the server.

The solution is leveraging Dependency Groups in Poetry. By moving away from a monolithic dependency list, you can categorize packages into logical groups and use targeted installation flags to prune your production environment.

Defining Groups in pyproject.toml

In older versions of Poetry, developers used a single dev-dependencies section. Modern Poetry (v1.2+) uses named groups, which allows for more granularity (e.g., separating test, docs, and lint groups). These are defined under [tool.poetry.group.<name>.dependencies].

The main dependency group (the default [tool.poetry.dependencies]) contains everything required to run the application. Any other group is considered optional by default during specific installation commands, though poetry install will install all groups unless told otherwise.

Practical Implementation: Separating Runtime from Tooling

Consider a project that requires FastAPI for the API and httpx for testing. Instead of mixing them, structure your pyproject.toml as follows:

[tool.poetry.dependencies]
python = "^3.11"
fastapi = "^0.100.0"
uvicorn = "^0.22.0"

[tool.poetry.group.test.dependencies]
pytest = "^7.4.0"
httpx = "^0.24.0"

[tool.poetry.group.lint.dependencies]
black = "^23.7.0"
flake8 = "^6.0.0"

Controlling the Install

Depending on where you are running the code, you will use different flags with the poetry install command. These commands should be run within the project root where the pyproject.toml and poetry.lock files reside.

• Local Development: Run poetry install. This installs the main dependencies and all defined groups.
• CI Pipeline (Testing): Run poetry install --only main,test. This ignores the lint group to save time.
• Production Dockerfile: Run poetry install --only main. This ensures no testing or linting tools enter the production image.

The Role of the Lock File in Reproducibility

While groups organize what gets installed, the poetry.lock file controls which version gets installed. When you run poetry install --only main, Poetry doesn't just look at the main group; it references the lock file to ensure the exact versions used in development are mirrored in production.

Engineering Decision: For applications, always commit poetry.lock to version control. This prevents "it works on my machine" bugs caused by a sub-dependency releasing a breaking change between your local test and your production deploy.

Limitations and Risks

The primary risk when using groups is dependency overlap. If a package is listed in both the main group and a test group with conflicting version constraints, the Poetry resolver will throw an error during the locking phase. You must ensure that your group constraints are compatible with your main runtime constraints.

Additionally, avoid mixing pip install with poetry install in the same environment. Pip is unaware of Poetry's group logic and lock file; manually installing a package via pip can desynchronize your environment from the poetry.lock file, leading to unpredictable behavior in CI/CD.

Verification Checklist

To verify your group configuration is working as intended, follow these steps:

1. Run poetry install --only main.
2. Execute poetry run pip list (or activate the virtualenv and run pip list).
3. Confirm that packages from your test or lint groups (e.g., pytest) are absent from the list.
4. Run poetry install --with test and verify those packages now appear.