Instructions for installing pydiso from source not working

[santisoler]

As pointed out by @johnweis0480 in today's SimPEG meeting, the instructions on how to install pydiso from sources that are currently in the README.md are not working properly.

Reproducible example:

conda create --name pydiso "python=3.11.*" pip numpy scipy mkl-devel
conda activate pydiso
pip install .

Processing /home/santi/git/pydiso
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
  Installing backend dependencies ... done
  Preparing metadata (pyproject.toml) ... error
  error: subprocess-exited-with-error

  × Preparing metadata (pyproject.toml) did not run successfully.
  │ exit code: 1
  ╰─> [29 lines of output]
      + meson setup /home/santi/git/pydiso /home/santi/git/pydiso/.mesonpy-8jqt8l3r -Dbuildtype=release -Db_ndebug=if-release -Db_vscrt=md --native-file=/home/santi/git/pydiso/.mesonpy-8jqt8l3r/meson-python-native-file.ini
      The Meson build system
      Version: 1.4.1
      Source dir: /home/santi/git/pydiso
      Build dir: /home/santi/git/pydiso/.mesonpy-8jqt8l3r
      Build type: native build
      Project name: pydiso
      Project version: 0.0.5
      C compiler for the host machine: cc (gcc 12.2.0 "cc (Debian 12.2.0-14) 12.2.0")
      C linker for the host machine: cc ld.bfd 2.40
      Cython compiler for the host machine: cython (cython 3.0.10)
      Host machine cpu family: x86_64
      Host machine cpu: x86_64
      Program python found: YES (/home/santi/.mambaforge/envs/pydiso/bin/python3.11)
      Found pkg-config: YES (/usr/bin/pkg-config) 1.8.1
      Run-time dependency python found: YES 3.11
      Program cython found: YES (/tmp/pip-build-env-vg_i50ii/overlay/bin/cython)
      Compiler for C supports arguments -Wno-unused-but-set-variable: YES
      Compiler for C supports arguments -Wno-unused-function: YES
      Compiler for C supports arguments -Wno-conversion: YES
      Compiler for C supports arguments -Wno-misleading-indentation: YES
      Library m found: YES
      Found CMake: /usr/bin/cmake (3.25.1)
      WARNING: CMake Toolchain: Failed to determine CMake compilers state
      Run-time dependency mkl-sdl found: NO (tried pkgconfig and cmake)

      ../pydiso/meson.build:61:14: ERROR: Dependency "mkl-sdl" not found, tried pkgconfig and cmake

      A full log can be found at /home/santi/git/pydiso/.mesonpy-8jqt8l3r/meson-logs/meson-log.txt
      [end of output]

  note: This error originates from a subprocess, and is likely not a problem with pip.
error: metadata-generation-failed

× Encountered error while generating package metadata.
╰─> See above for output.

note: This is an issue with the package mentioned above, not pip.
hint: See above for details.

List of packages in environment

# packages in environment at /home/santi/.mambaforge/envs/pydiso:
#
# Name                    Version                   Build  Channel
_libgcc_mutex             0.1                 conda_forge    conda-forge
_openmp_mutex             4.5                  2_kmp_llvm    conda-forge
bzip2                     1.0.8                hd590300_5    conda-forge
ca-certificates           2024.6.2             hbcca054_0    conda-forge
icu                       73.2                 h59595ed_0    conda-forge
ld_impl_linux-64          2.40                 hf3520f5_3    conda-forge
libblas                   3.9.0           22_linux64_openblas    conda-forge
libcblas                  3.9.0           22_linux64_openblas    conda-forge
libexpat                  2.6.2                h59595ed_0    conda-forge
libffi                    3.4.2                h7f98852_5    conda-forge
libgcc-ng                 13.2.0               h77fa898_8    conda-forge
libgfortran-ng            13.2.0               h69a702a_8    conda-forge
libgfortran5              13.2.0               h3d2ce59_8    conda-forge
libhwloc                  2.10.0          default_h5622ce7_1001    conda-forge
libiconv                  1.17                 hd590300_2    conda-forge
liblapack                 3.9.0           22_linux64_openblas    conda-forge
libnsl                    2.0.1                hd590300_0    conda-forge
libopenblas               0.3.27          pthreads_h413a1c8_0    conda-forge
libsqlite                 3.46.0               hde9e2c9_0    conda-forge
libstdcxx-ng              13.2.0               hc0a3c3a_8    conda-forge
libuuid                   2.38.1               h0b41bf4_0    conda-forge
libxcrypt                 4.4.36               hd590300_1    conda-forge
libxml2                   2.12.7               hc051c1a_1    conda-forge
libzlib                   1.3.1                h4ab18f5_1    conda-forge
llvm-openmp               18.1.7               ha31de31_0    conda-forge
mkl                       2024.1.0           ha957f24_693    conda-forge
mkl-devel                 2024.1.0           ha770c72_693    conda-forge
mkl-include               2024.1.0           ha957f24_693    conda-forge
ncurses                   6.5                  h59595ed_0    conda-forge
numpy                     1.26.4          py311h64a7726_0    conda-forge
openssl                   3.3.1                h4ab18f5_0    conda-forge
pip                       24.0               pyhd8ed1ab_0    conda-forge
python                    3.11.9          hb806964_0_cpython    conda-forge
python_abi                3.11                    4_cp311    conda-forge
readline                  8.2                  h8228510_1    conda-forge
scipy                     1.13.1          py311h517d4fd_0    conda-forge
setuptools                70.0.0             pyhd8ed1ab_0    conda-forge
tbb                       2021.12.0            h297d8ca_1    conda-forge
tk                        8.6.13          noxft_h4845f30_101    conda-forge
tzdata                    2024a                h0c530f3_0    conda-forge
wheel                     0.43.0             pyhd8ed1ab_1    conda-forge
xz                        5.2.6                h166bdaf_0    conda-forge
zstd                      1.5.6                ha6fb4c9_0    conda-forge

Possible solution

By installing the packages that are being installed in Actions to build the sources, I was able to build and install pydiso. These are:

cython mkl mkl-devel pkg-config meson-python meson ninja setuptools_scm

I think we should update the instructions in the README.md adding packages that need to be installed to build pydiso. And also maybe update the list of requirements for the build in pyproject.toml:

pydiso/pyproject.toml

Lines 4 to 30 in 79c4103

	requires = [
	"meson-python>=0.14.0",
	"Cython>=0.29.35", # when updating version, also update check in meson.build
	"setuptools_scm[toml]>=6.2",
	# This package automatically provides all of the numpy pinning for different python
	# versions and runtime requirements.
	"oldest-supported-numpy",
	# The following is taken from scipy's pyproject.toml file to handle
	# building against the proper numpy API
	# When numpy 2.0.0rc1 comes out, we should update this to build against 2.0,
	# and then runtime depend on the range 1.22.X to <2.3. No need to switch to
	# 1.25.2 in the meantime (1.25.x is the first version which exports older C
	# API versions by default).
	# default numpy requirements
	# "numpy==1.22.4; python_version<='3.10' and platform_python_implementation != 'PyPy'",
	# "numpy==1.23.2; python_version=='3.11' and platform_python_implementation != 'PyPy'",
	# For Python versions which aren't yet officially supported, we specify an
	# unpinned NumPy which allows source distributions to be used and allows
	# wheels to be used as soon as they become available.
	# "numpy>=1.26.0b1; python_version>='3.12'",
	# "numpy; python_version>='3.8' and platform_python_implementation=='PyPy'",
	]

[jcapriot]

You also need a valid pkg-config or cmake as the error message points out, unfortunately these are not things that you can have as a requirement through python.

[jcapriot]

Also, as the readme states, you might need to set PKG_CONFIG_PATH appropriately (which is not something I saw in your original code line). Also note that you’ll likely get a different error on a windows or osx machine as compared to this, as Linux usually has a system pkg_config to rely on.

[santisoler]

You also need a valid pkg-config or cmake as the error message points out, unfortunately these are not things that you can have as a requirement through python.

I imagined. But since the installation instructions are using conda for installing mkl-devel, we can instruct ppl to also install pkg-config through conda as well. With that we don't even need to set the PKG_CONFIG_PATH variable.

[jcapriot]

Installing pkg-config through conda sets PKG_CONFIG_PATH to the correct directory for you, which is why it worked after you did that.

There are other ways of installing MKL though, besides just conda, in which case you would need to set PKG_CONFIG_PATH yourself.

Because it is a built package that links to external c code, installing from source is not as straightforward as other libraries.

[sgkang]

Thank you so much for this discussion!
Can we modify instructions to include?:

conda install conda-forge::cmake pkg-config mkl-devel

This worked well for my mac (both python 3.10 and 3.11)

[jcapriot]

Just curious for what required cmake for you? I've always had success with just pkg-config and mkl-devel.

[sgkang]

Testing on our cluster, but got following errors:

Traceback (most recent call last):
File "/home/groups/rknight/python-codes/pydiso/tests/test_pydiso.py", line 3, in
from pydiso.mkl_solver import (
File "", line 1027, in _find_and_load
File "", line 1002, in _find_and_load_unlocked
File "", line 945, in _find_spec
File "/home/users/sgkang09/miniconda3/envs/em_test/lib/python3.10/site-packages/_pydiso_editable_loader.py", line 310, in find_spec
tree = self._rebuild()
File "/home/users/sgkang09/miniconda3/envs/em_test/lib/python3.10/site-packages/_pydiso_editable_loader.py", line 343, in _rebuild
subprocess.run(self._build_cmd, cwd=self._build_path, env=env, stdout=subprocess.DEVNULL)
File "/home/users/sgkang09/miniconda3/envs/em_test/lib/python3.10/subprocess.py", line 503, in run
with Popen(*popenargs, **kwargs) as process:
File "/home/users/sgkang09/miniconda3/envs/em_test/lib/python3.10/subprocess.py", line 971, in init
self._execute_child(args, executable, preexec_fn, close_fds,
File "/home/users/sgkang09/miniconda3/envs/em_test/lib/python3.10/subprocess.py", line 1863, in _execute_child
raise child_exception_type(errno_num, err_msg, err_filename)
FileNotFoundError: [Errno 2] No such file or directory: '/tmp/pip-build-env-8nzol7yy/normal/bin/ninja'

Any ideas? @jcapriot @santisoler

[sgkang]

Was able to make it work on our cluster both python 3.10 and 3.11 [I think it is linux machine with ubuntu].

Few things that I had to do:

• conda install conda-forge::ninja cython
• pip install numpy=1.26 [there was an issue with numpy 2.0]

Associated error that I got is provided below:

ImportError: numpy.core.multiarray failed to import (auto-generated because you didn't call 'numpy.import_array()' after cimporting numpy; use 'numpy._import_array' to disable if you ar
e certain you don't need it).

[sgkang]

Just curious for what required cmake for you? I've always had success with just pkg-config and mkl-devel.

My mac did not have cmake, so had to install. I could have install through brew.

[jcapriot]

What command were you using to install it?

[jcapriot]

Just to be clear, the package uses a meson-python build backend. When you do pip install . (not an inplace or editable build). the isolated pip build environment will grab the dependencies for meson-python (which will install meson and ninja) cython, numpy and setuptools_scm. Outside of the build environment it still needs to be able to find where the mkl related headers (to compile with) and libraries are (to link against). The meson build system uses the dependency function to find this information. That function in turn uses several different methods, of which pkg-config and cmake are both methods (see https://mesonbuild.com/Dependencies.html#dependency-detection-method).

If you do an inplace build without isolation then you must already have meson-python, ninja, cython, numpy, and setuptools_scm in your environment (along with the necessary mkl information).