pip-compile fails on GPyOpt

[yurivict]

When the req.in contains only GPyOpt this command fails:

$ pip-compile req.in
    error: subprocess-exited-with-error
    
    × Preparing metadata (pyproject.toml) did not run successfully.
    │ exit code: 1
    ╰─> [53 lines of output]
        + meson setup /tmp/pip-resolve-lj0r1wut/scipy_538fc0b03c634b3c8169487991b309d2 /tmp/pip-resolve-lj0r1wut/scipy_538fc0b03c634b3c8169487991b309d2/.mesonpy-2v1b0pq6 -Dbuildtype=release -Db_ndebug=if-release -Db_vscrt=md --native-file=/tmp/pip-resolve-lj0r1wut/scipy_538fc0b03c634b3c8169487991b309d2/.mesonpy-2v1b0pq6/meson-python-native-file.ini
        The Meson build system
        Version: 1.7.0
        Source dir: /tmp/pip-resolve-lj0r1wut/scipy_538fc0b03c634b3c8169487991b309d2
        Build dir: /tmp/pip-resolve-lj0r1wut/scipy_538fc0b03c634b3c8169487991b309d2/.mesonpy-2v1b0pq6
        Build type: native build
        Project name: scipy
        Project version: 1.15.2
        C compiler for the host machine: ccache cc (clang 19.1.7 "FreeBSD clang version 19.1.7 (https://github.com/llvm/llvm-project.git llvmorg-19.1.7-0-gcd708029e0b2)")
        C linker for the host machine: cc ld.lld 19.1.7
        C++ compiler for the host machine: ccache c++ (clang 19.1.7 "FreeBSD clang version 19.1.7 (https://github.com/llvm/llvm-project.git llvmorg-19.1.7-0-gcd708029e0b2)")
        C++ linker for the host machine: c++ ld.lld 19.1.7
        Cython compiler for the host machine: cython (cython 3.0.12)
        Host machine cpu family: x86_64
        Host machine cpu: x86_64
        Program python found: YES (/usr/local/bin/python3.11)
        Found pkg-config: YES (/usr/local/bin/pkg-config) 2.3.0
        Run-time dependency python found: YES 3.11
        Program cython found: YES (/tmp/pip-build-env-csnlgp7w/overlay/bin/cython)
        Compiler for C supports arguments -Wno-unused-but-set-variable: YES
        Compiler for C supports arguments -Wno-unused-function: YES
        Compiler for C supports arguments -Wno-conversion: YES
        Compiler for C supports arguments -Wno-misleading-indentation: YES
        Library m found: YES
        
        ../meson.build:80:0: ERROR: Unknown compiler(s): [['gfortran'], ['flang-new'], ['flang'], ['nvfortran'], ['pgfortran'], ['ifort'], ['ifx'], ['g95']]
        The following exception(s) were encountered:
        Running `gfortran --help` gave "[Errno 2] No such file or directory: 'gfortran'"
        Running `gfortran --version` gave "[Errno 2] No such file or directory: 'gfortran'"
        Running `gfortran -V` gave "[Errno 2] No such file or directory: 'gfortran'"
        Running `flang-new --help` gave "[Errno 2] No such file or directory: 'flang-new'"
        Running `flang-new --version` gave "[Errno 2] No such file or directory: 'flang-new'"
        Running `flang-new -V` gave "[Errno 2] No such file or directory: 'flang-new'"
        Running `flang --help` gave "[Errno 2] No such file or directory: 'flang'"
        Running `flang --version` gave "[Errno 2] No such file or directory: 'flang'"
        Running `flang -V` gave "[Errno 2] No such file or directory: 'flang'"
        Running `nvfortran --help` gave "[Errno 2] No such file or directory: 'nvfortran'"
        Running `nvfortran --version` gave "[Errno 2] No such file or directory: 'nvfortran'"
        Running `nvfortran -V` gave "[Errno 2] No such file or directory: 'nvfortran'"
        Running `pgfortran --help` gave "[Errno 2] No such file or directory: 'pgfortran'"
        Running `pgfortran --version` gave "[Errno 2] No such file or directory: 'pgfortran'"
        Running `pgfortran -V` gave "[Errno 2] No such file or directory: 'pgfortran'"
        Running `ifort --help` gave "[Errno 2] No such file or directory: 'ifort'"
        Running `ifort --version` gave "[Errno 2] No such file or directory: 'ifort'"
        Running `ifort -V` gave "[Errno 2] No such file or directory: 'ifort'"
        Running `ifx --help` gave "[Errno 2] No such file or directory: 'ifx'"
        Running `ifx --version` gave "[Errno 2] No such file or directory: 'ifx'"
        Running `ifx -V` gave "[Errno 2] No such file or directory: 'ifx'"
        Running `g95 --help` gave "[Errno 2] No such file or directory: 'g95'"
        Running `g95 --version` gave "[Errno 2] No such file or directory: 'g95'"
        Running `g95 -V` gave "[Errno 2] No such file or directory: 'g95'"
        
        A full log can be found at /tmp/pip-resolve-lj0r1wut/scipy_538fc0b03c634b3c8169487991b309d2/.mesonpy-2v1b0pq6/meson-logs/meson-log.txt
        [end of output]
    
    note: This error originates from a subprocess, and is likely not a problem with pip.
Traceback (most recent call last):
  File "/usr/local/lib/python3.11/site-packages/pip/_internal/operations/build/metadata.py", line 35, in generate_metadata
    distinfo_dir = backend.prepare_metadata_for_build_wheel(metadata_dir)
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/pip/_internal/utils/misc.py", line 772, in prepare_metadata_for_build_wheel
    return super().prepare_metadata_for_build_wheel(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/pip/_vendor/pyproject_hooks/_impl.py", line 186, in prepare_metadata_for_build_wheel
    return self._call_hook('prepare_metadata_for_build_wheel', {
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/pip/_vendor/pyproject_hooks/_impl.py", line 311, in _call_hook
    self._subprocess_runner(
  File "/usr/local/lib/python3.11/site-packages/pip/_internal/utils/subprocess.py", line 252, in runner
    call_subprocess(
  File "/usr/local/lib/python3.11/site-packages/pip/_internal/utils/subprocess.py", line 224, in call_subprocess
    raise error
pip._internal.exceptions.InstallationSubprocessError: Preparing metadata (pyproject.toml) exited with 1

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/bin/pip-compile", line 8, in <module>
    sys.exit(cli())
             ^^^^^
  File "/usr/local/lib/python3.11/site-packages/click/core.py", line 1157, in __call__
    return self.main(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/click/core.py", line 1078, in main
    rv = self.invoke(ctx)
         ^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/click/core.py", line 1434, in invoke
    return ctx.invoke(self.callback, **ctx.params)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/click/core.py", line 783, in invoke
    return __callback(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/click/decorators.py", line 33, in new_func
    return f(get_current_context(), *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/piptools/scripts/compile.py", line 470, in cli
    results = resolver.resolve(max_rounds=max_rounds)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/piptools/resolver.py", line 604, in resolve
    is_resolved = self._do_resolve(
                  ^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/piptools/resolver.py", line 636, in _do_resolve
    resolver.resolve(
  File "/usr/local/lib/python3.11/site-packages/pip/_internal/resolution/resolvelib/resolver.py", line 95, in resolve
    result = self._result = resolver.resolve(
                            ^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/pip/_vendor/resolvelib/resolvers.py", line 546, in resolve
    state = resolution.resolve(requirements, max_rounds=max_rounds)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/pip/_vendor/resolvelib/resolvers.py", line 427, in resolve
    failure_causes = self._attempt_to_pin_criterion(name)
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/pip/_vendor/resolvelib/resolvers.py", line 239, in _attempt_to_pin_criterion
    criteria = self._get_updated_criteria(candidate)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/pip/_vendor/resolvelib/resolvers.py", line 230, in _get_updated_criteria
    self._add_to_criteria(criteria, requirement, parent=candidate)
  File "/usr/local/lib/python3.11/site-packages/pip/_vendor/resolvelib/resolvers.py", line 173, in _add_to_criteria
    if not criterion.candidates:
  File "/usr/local/lib/python3.11/site-packages/pip/_vendor/resolvelib/structs.py", line 156, in __bool__
    return bool(self._sequence)
           ^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/pip/_internal/resolution/resolvelib/found_candidates.py", line 155, in __bool__
    return any(self)
           ^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/pip/_internal/resolution/resolvelib/found_candidates.py", line 143, in <genexpr>
    return (c for c in iterator if id(c) not in self._incompatible_ids)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/pip/_internal/resolution/resolvelib/found_candidates.py", line 47, in _iter_built
    candidate = func()
                ^^^^^^
  File "/usr/local/lib/python3.11/site-packages/pip/_internal/resolution/resolvelib/factory.py", line 182, in _make_candidate_from_link
    base: Optional[BaseCandidate] = self._make_base_candidate_from_link(
                                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/pip/_internal/resolution/resolvelib/factory.py", line 228, in _make_base_candidate_from_link
    self._link_candidate_cache[link] = LinkCandidate(
                                       ^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/pip/_internal/resolution/resolvelib/candidates.py", line 293, in __init__
    super().__init__(
  File "/usr/local/lib/python3.11/site-packages/pip/_internal/resolution/resolvelib/candidates.py", line 156, in __init__
    self.dist = self._prepare()
                ^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/pip/_internal/resolution/resolvelib/candidates.py", line 225, in _prepare
    dist = self._prepare_distribution()
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/pip/_internal/resolution/resolvelib/candidates.py", line 304, in _prepare_distribution
    return preparer.prepare_linked_requirement(self._ireq, parallel_builds=True)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/pip/_internal/operations/prepare.py", line 525, in prepare_linked_requirement
    return self._prepare_linked_requirement(req, parallel_builds)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/pip/_internal/operations/prepare.py", line 640, in _prepare_linked_requirement
    dist = _get_prepared_distribution(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/pip/_internal/operations/prepare.py", line 71, in _get_prepared_distribution
    abstract_dist.prepare_distribution_metadata(
  File "/usr/local/lib/python3.11/site-packages/pip/_internal/distributions/sdist.py", line 67, in prepare_distribution_metadata
    self.req.prepare_metadata()
  File "/usr/local/lib/python3.11/site-packages/pip/_internal/req/req_install.py", line 577, in prepare_metadata
    self.metadata_directory = generate_metadata(
                              ^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/pip/_internal/operations/build/metadata.py", line 37, in generate_metadata
    raise MetadataGenerationFailed(package_details=details) from error
pip._internal.exceptions.MetadataGenerationFailed: metadata generation failed

Environment Versions

1. OS Type: FreeBSD 14.2
2. Python version: 3.11
3. pip version: 23.3.2
4. pip-tools version:pip-compile, version 7.4.1

The same happens on Linux on a completely unrelated system with Python-3.11

[sirosen]

Thanks for including the full trace in your report! Are you able to pip install GPyOpt on this same system (in a virtualenv) without issue?

The error looks like a failure to build scipy from source. That would happen because you aren't able to find a valid wheel matching the requirements of your current system, and a wheel build is being invoked to allow metadata extraction.

---

Sorry for the slow reply. I appreciate your patience, and if this issue isn't relevant to you anymore, please let us know and we can close it.

[yurivict]

pip-compile should fail with a relevant human-readable error message then.

The above log looks like an ungraceful failure.

[sirosen]

pip-compile should give you the best possible error message within reason.

In this case, there are three layers of underlying tooling which have to report this failure. (pip, the build backend, and SciPy's build configuration)
Reporting back the error we were given by these tools is currently the best option.

Unless you or someone else has an idea about a specific improvement we can make, I'm afraid I don't see how we can do better right now.

[webknjaz]

https://github.com/jazzband/pip-tools/blob/e4ed0c1/piptools/scripts/compile.py#L471 is in the trace. The handling only covers two exceptions, there may be more. And also, it just stringifies them. I think it may be reasonable to add a few log lines explaining what this means. Also, those resolvelib exceptions do have a certain amount of context that pip displays but we don't. I integrated resolvelib into ansible-galaxy years ago and still keep improving the error output.

[sirosen]

Ah, I was too focused on the first part of the trace, where the useful content is. Yes, we should catch the error, print it (as we do in the other cases), and cleanly exit! Thanks for the pointer.

I think we should capture MetadataGenerationFailed, and see what useful information we can print to decorate the error as well. I'll work on this after we get a release out with the fixes we already have in flight.