python butler.py bootstrap -> ModuleNotFoundError: No module named 'distutils.sysconfig'

[ArthurSonzogni]

Reproducer

#!/bin/bash
git clone https://github.com/google/clusterfuzz
cd clusterfuzz
git pull

# Make sure gcloud is installed
if ! command -v gcloud &> /dev/null; then
    echo "gcloud could not be found. Please install Google Cloud SDK."
    exit 1
fi

# Make sure python 3.11 is installed
alias python=python3.11
if ! command -v python &> /dev/null; then
    echo "Python 3.11 could not be found. Please install Python 3.11."
    exit 1
fi

# Make sure go is installed
if ! command -v go &> /dev/null; then
    echo "Go could not be found. Please install Go."
    exit 1
fi

# Print the name of the current platform
echo "Current platform: $(uname -s)-$(uname -m)"

# Install dependencies
local/install_deps.bash

Output

+ python butler.py bootstrap
Running: python3.11 -m pipenv requirements  (cwd="src")
| Return code is non-zero (1).
Running: python3.11 -m pipenv lock -r --no-header  (cwd="src")
| Return code is non-zero (1).
Traceback (most recent call last):
  File "/usr/local/google/home/arthursonzogni/clusterfuzz/butler.py", line 433, in <module>
    sys.exit(main())
             ^^^^^^
  File "/usr/local/google/home/arthursonzogni/clusterfuzz/butler.py", line 419, in main
    return command.execute(args)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/google/home/arthursonzogni/clusterfuzz/src/local/butler/bootstrap.py", line 25, in execute
    common.install_dependencies()
  File "/usr/local/google/home/arthursonzogni/clusterfuzz/src/local/butler/common.py", line 340, in install_dependencies
    _pipfile_to_requirements('src', 'src/requirements.txt')
  File "/usr/local/google/home/arthursonzogni/clusterfuzz/src/local/butler/common.py", line 275, in _pipfile_to_requirements
    raise RuntimeError('Failed to generate requirements from Pipfile.')
RuntimeError: Failed to generate requirements from Pipfile.

If you print the stderr, you get:

| Traceback (most recent call last):
|   File "<frozen runpy>", line 189, in _run_module_as_main
|   File "<frozen runpy>", line 148, in _get_module_details
|   File "<frozen runpy>", line 112, in _get_module_details
|   File "/usr/local/google/home/arthursonzogni/.local/share/virtualenvs/clusterfuzz-9c9_bw-f/lib/python3.11/site-packages/pipenv/__init__.py", line 59, in <module>
|     from .cli import cli
|   File "/usr/local/google/home/arthursonzogni/.local/share/virtualenvs/clusterfuzz-9c9_bw-f/lib/python3.11/site-packages/pipenv/cli/__init__.py", line 1, in <module>
|     from .command import cli  # noqa
|     ^^^^^^^^^^^^^^^^^^^^^^^^
|   File "/usr/local/google/home/arthursonzogni/.local/share/virtualenvs/clusterfuzz-9c9_bw-f/lib/python3.11/site-packages/pipenv/cli/command.py", line 7, in <module>
|     from pipenv.cli.options import (
|   File "/usr/local/google/home/arthursonzogni/.local/share/virtualenvs/clusterfuzz-9c9_bw-f/lib/python3.11/site-packages/pipenv/cli/options.py", line 3, in <module>
|     from pipenv.project import Project
|   File "/usr/local/google/home/arthursonzogni/.local/share/virtualenvs/clusterfuzz-9c9_bw-f/lib/python3.11/site-packages/pipenv/project.py", line 47, in <module>
|     from pipenv.vendor.requirementslib.models.utils import get_default_pyproject_backend
|   File "/usr/local/google/home/arthursonzogni/.local/share/virtualenvs/clusterfuzz-9c9_bw-f/lib/python3.11/site-packages/pipenv/vendor/requirementslib/__init__.py", line 9, in <module>
|     from .models.lockfile import Lockfile
|   File "/usr/local/google/home/arthursonzogni/.local/share/virtualenvs/clusterfuzz-9c9_bw-f/lib/python3.11/site-packages/pipenv/vendor/requirementslib/models/lockfile.py", line 16, in <module>
|     from .requirements import Requirement
|   File "/usr/local/google/home/arthursonzogni/.local/share/virtualenvs/clusterfuzz-9c9_bw-f/lib/python3.11/site-packages/pipenv/vendor/requirementslib/models/requirements.py", line 10, in <module>
|     from distutils.sysconfig import get_python_lib
| ModuleNotFoundError: No module named 'distutils.sysconfig'
| Return code is non-zero (1).
Running: python3.11 -m pipenv lock -r --no-header  (cwd="src")
| Traceback (most recent call last):
|   File "<frozen runpy>", line 189, in _run_module_as_main
|   File "<frozen runpy>", line 148, in _get_module_details
|   File "<frozen runpy>", line 112, in _get_module_details
|   File "/usr/local/google/home/arthursonzogni/.local/share/virtualenvs/clusterfuzz-9c9_bw-f/lib/python3.11/site-packages/pipenv/__init__.py", line 59, in <module>
|     from .cli import cli
|   File "/usr/local/google/home/arthursonzogni/.local/share/virtualenvs/clusterfuzz-9c9_bw-f/lib/python3.11/site-packages/pipenv/cli/__init__.py", line 1, in <module>
|     from .command import cli  # noqa
|     ^^^^^^^^^^^^^^^^^^^^^^^^
|   File "/usr/local/google/home/arthursonzogni/.local/share/virtualenvs/clusterfuzz-9c9_bw-f/lib/python3.11/site-packages/pipenv/cli/command.py", line 7, in <module>
|     from pipenv.cli.options import (
|   File "/usr/local/google/home/arthursonzogni/.local/share/virtualenvs/clusterfuzz-9c9_bw-f/lib/python3.11/site-packages/pipenv/cli/options.py", line 3, in <module>
|     from pipenv.project import Project
|   File "/usr/local/google/home/arthursonzogni/.local/share/virtualenvs/clusterfuzz-9c9_bw-f/lib/python3.11/site-packages/pipenv/project.py", line 47, in <module>
|     from pipenv.vendor.requirementslib.models.utils import get_default_pyproject_backend
|   File "/usr/local/google/home/arthursonzogni/.local/share/virtualenvs/clusterfuzz-9c9_bw-f/lib/python3.11/site-packages/pipenv/vendor/requirementslib/__init__.py", line 9, in <module>
|     from .models.lockfile import Lockfile
|   File "/usr/local/google/home/arthursonzogni/.local/share/virtualenvs/clusterfuzz-9c9_bw-f/lib/python3.11/site-packages/pipenv/vendor/requirementslib/models/lockfile.py", line 16, in <module>
|     from .requirements import Requirement
|   File "/usr/local/google/home/arthursonzogni/.local/share/virtualenvs/clusterfuzz-9c9_bw-f/lib/python3.11/site-packages/pipenv/vendor/requirementslib/models/requirements.py", line 10, in <module>
|     from distutils.sysconfig import get_python_lib
| ModuleNotFoundError: No module named 'distutils.sysconfig'
| Return code is non-zero (1).

Potential solution

gemini-cli suggest setting:

pipenv = "==2025.0.4"

in Pipfile

Reasoning:

The traceback confirms the error originates from an outdated, vendored dependency in pipenv itself. The Pipfile pins pipenv to version 2022.8.5, which is incompatible with
  Python 3.11.9. I need to upgrade pipenv to a more recent version. I'll use web search to find the latest version on PyPI.