Tools for managing python packages

[TeresiaOlsson]

The template from @MJGaughran that was shown at the maintainer's meeting is here:

https://github.com/DiamondLightSource/python-copier-template

I think perhaps the first things to discuss are:

• Which build tool
• Linting and which standard to use

I think mbtrack2 has a nice setup so perhaps @GamelinAl and @gubaidulinvadim can explain how you do it?

[MJGaughran]

Sorry, I was planning to make this post just now!

I think the only thing to note with that Copier template is that it includes the bash-config feature, as container builds are a lot faster than with alternatives. The initializeCommand will automatically create a folder in your home drive.

I'm not tied to us using this particular template, but I am strongly in favour of a Copier template that we can use across our multiple Python packages.

I was thinking that perhaps all core PyAML code could follow either the DLS template, a fork or a similar Copier template. We can also provide a more general-use, trimmed down version that particular facilities can adopt where e.g. dev container setup is non-trivial or the people working with it are not as familiar with programming. E.g. we don't enable strict type checking and coverage, etc.

I'd be interested to hear people's opinions. If you get stuck, feel free to ask me questions.

[MJGaughran]

The Copier template is relatively conservative, but not outdated. We are considering e.g. moving to uv for package management, which I know several people are already using.

With a fork, we would have time to review any major changes as they come.

[GamelinAl]

I am not an expert on this at all, so probably what was done in Diamond is much better but we can share the experience we had with mbtrack2.

In mbtrack2, we are using poetry (https://python-poetry.org/) for dependency management and packaging (building wheels for pip). I had a good experience with poetry and maybe it can be considered for pyAML. When the code is pure python, it's really straightforward. When it's mixed in with other languages, I don't know so much. But recently @gubaidulinvadim added some fortran code which needed compilation and he was able to build it with meson (https://mesonbuild.com/) and make it work with poetry.

We are using isort (https://pycqa.github.io/isort/) and yapf (https://github.com/google/yapf) as formatters. For me this part is not so important, we just need to have some formatters and stick to it.

A very important point is the CI/CD with the automatic building of docker containters to run tests and so on. Github is different from Gitlab on that point so I don't think I can help much there (but Patrick Madela from SOLEIL should be able to if needed).

But from what I see, all the features listed above are included in the template from Diamond (and much more).
I strongly suggest to start from some template like that (even if we change some part of it to suit our needs), we will gain a lot of time.

[MJGaughran]

It would be interesting to see how the Copier template performs for an experienced Python developer who is used to different tools.

If you have the time to try it out, I would be grateful.

[gubaidulinvadim]

I could try it out this week. The main point, I think, is that your template covers many things at once, and it's a template for all projects. The mbtrack2 setup is not a template; it's just a project that is set up somewhat properly. We do not use templates for projects at SOLEIL (but we should!). If anything, it is I and Alexis who should use Copier template to simplify our workflows when a new package is developed. The main thing that I do not like in the copier template (without trying it) is that it does not use poetry or uv for package management. In my experience it really simplifies things.

[MJGaughran]

You can actually try the uv version on the move-to-uv branch (from the PR). You just need to pass -r move-to-uv to the copy or update commands.

I think there were several external reasons for not moving to uv previously. One I saw mentioned is this, although that is not yet resolved.

Regardless, with the Copier template it's very easy to switch a bunch of repositories over. I think it'll be soon, but I'll double check.

[GamelinAl]

Nice to know, then if we start from this template we can use the version using uv, as it's a new project the issue @MJGaughran pointed out should not matter to us.