Support hermetic runs of pip-compile

[brandonchinn178]

What's the problem this feature will solve?

The recent broken wheel release broke all invocations of pip-compile. More context here: bazel-contrib/rules_python#2763

Describe the solution you'd like

I would like a mechanism to pin the version of setuptools/wheel that pip-compile installs when bootstrapping the environment. This would allow Bazel's rules_python to pin the versions and ensure a hermetic build.

Alternative Solutions

Additional context

[webknjaz]

The only way to do this (assuming pip provisions the build env) is to set PIP_CONSTRAINT.

P.S. Wheel should never be in the list of build deps. In the past, setuptools would auto-inject this dep when building wheels and now (since the last year's PyCon US sprints), wheel building integration moved inside of setuptools itself. The wheel package is not needed anymore, unless you do additional wheel manipulations on top of normal packaging which most people don't.

[webknjaz]

Related discussions regarding pinning build deps and the use of PIP_CONSTRAINT as a workaround:

• Pin build dependencies pypa/pip#13113
• Feature: Specify constraints file for the isolated build environment pypa/pip#13300
• PEP 518 build requirements cannot be overriden by user pypa/pip#4582 (comment)
• Respect top-level "constraints" file when populating isolated build environments pypa/pip#8439 (comment)
• Allow setting build env constraints pypa/build#292
• fix: Don't override PIP_CONSTRAINT if set by the user pypa/cibuildwheel#1675

[brandonchinn178]

@webknjaz Thanks for getting back!

Wheel should never be in the list of build deps. In the past, setuptools would auto-inject this dep when building wheels and now (since the last year's PyCon US sprints), wheel building integration moved inside of setuptools itself. The wheel package is not needed anymore, unless you do additional wheel manipulations on top of normal packaging which most people don't.

Is this true? I have

[build-system]
build-backend = "setuptools.build_meta"
requires = ["setuptools == 78.1.0"]

And I still see wheel being installed

Updating requirements.txt
Using pip-tools configuration defaults found in 'pyproject.toml'.
Creating isolated environment: venv+pip...
Installing packages in isolated environment:
- setuptools == 78.1.0
Getting build dependencies for wheel...
running egg_info
writing foo.egg-info/PKG-INFO
writing dependency_links to foo.egg-info/dependency_links.txt
writing top-level names to foo.egg-info/top_level.txt
reading manifest file 'foo.egg-info/SOURCES.txt'
writing manifest file 'foo.egg-info/SOURCES.txt'
Installing packages in isolated environment:
- wheel

[webknjaz]

@brandonchinn178 I'd need a more comple example to understand what's happening.

Using pip or pypa/build wouldn't do that. And I don't think pip-tools would either. Your snippet is incomplete, however, so it's rather difficult to judge what's happening in there...

[brandonchinn178]

aha. Okay I think I found it. So running pip-compile directly does use the builtin wheel library, but running the Bazel action that invokes pip-compile downloads it fresh. It looks like rules_python pins setuptools 70.0.0:
https://github.com/bazel-contrib/rules_python/blob/84351d4ec14e474bc196c0b8cd70e04fcc9a25ca/python/private/pypi/deps.bzl#L79

And it looks like 70.1.0 was the version that started vendoring wheel. So it looks like it's on rules_python to bump their pin of setuptools

[webknjaz]

That's correct. I think that this incompatibility will only be hit when really old setuptools is combined with the new wheel.. So when you're pinning through PIP_CONSTRAINT, make sure you pin all the things in the env and not just a part of them, since they are coupled. Usually, when the pin bumps go through the CI, it would break forcing one to troubleshoot the issue and then, bump related things together.