Official binding for Python.

[trivialfis]

Hi, hwloc maintainers.

We are working on HPC support for Python applications like Dask and XGBoost, and would like to utilize the excellent hwloc library for controlling NUMA configurations. We would like to have a library for programmatic access instead of relying on the command line interface.

The Python interface listed on the official site appears to be somewhat outdated, with the last commit dated Aug 14, 2019. We have also searched online but couldn't find any existing Python bindings that are actively maintained. Python is a fast-moving language with a changing tool chain, so I assume existing bindings are no longer maintained or only thinly maintained. Feel free to correct me if I'm wrong.

Before we start another Python package, I would like to ask whether hwloc is interested in adopting an official Python interface, given the popularity of Python nowadays. I can contribute to the initial implementation based on the maintainers' guidance and work towards a higher-level interface if needed. My initial focus will be on developing a basic wrapper and helping integrate CUDA-related features, along with basic setup for a Python package.

[bgoglin]

Hello. In theory yes. In practice the main issue is that somebody needs to maintain it. I am not a Python user. But I could do some easy maintenance is there are some good regression tests to detect breaks.
Also, iirc, one big issue with previous bindings is that the API is huge.
Brice

[trivialfis]

Thank you for the reply, Brice. I can help with maintaining the package itself, but not the CI infrastructure or related items. I assume there are specialized setups, as Jenkins is used.

I will update this thread once we can work on the package.

[bgoglin]

I can take care of the jenkins integration. I just need some scripts to run.

[trivialfis]

@bgoglin In order to have third-party tools like breathe to parse the C doc, we need to turn on the XML gen:

hwloc/doc/doxygen.cfg

Line 2140 in c124d19

GENERATE_XML = NO

Would it be a problem if we set it to YES by default? This should help #727 as well.

[bgoglin]

I don't think it would be a problem. But we need to understand when you'll need the XML output. If building hwloc from GIT, doxygen would build both HTML/PDF as usual and XML. If building hwloc from a tarball/release, the doxygen HTML/PDF is prebuilt. If you need the XML there, we'll likely need to distribute the XML in the tarballs too. Hopefully it's not too big.

[trivialfis]

But we need to understand when you'll need the XML output.

The Python package that I'm working on will use the git repo.

Let me create a PR to change it to using an environment variable instead.

[trivialfis]

I made the output optional in this PR: #740 (opt-in)

[trivialfis]

@bgoglin Could you please help clarify whether this is a misunderstanding from my side or it's a mistake in the code:

The following hwloc__add_info implementation always returns either 1 or -1:

hwloc/hwloc/topology.c

Line 490 in e83dfb5

return 1;

But the public document suggests that it should return 0 on success:

hwloc/include/hwloc.h

Line 1295 in e83dfb5

* \return \c 0 on success, \c -1 on error.

I have been running into failure with this function. I can start a new thread for discussion if needed.

[bgoglin]

@bgoglin Could you please help clarify whether this is a misunderstanding from my side or it's a mistake in the code:
The following hwloc__add_info implementation always returns either 1 or -1:
[...]
But the public document suggests that it should return 0 on success:

Indeed, this is an internal change in 3.0 that wasn't properly handled in the external API, I'll fix it, thanks.

[trivialfis]

Thank you for the quick fix!