Investigate using Py_LIMITED_API to reduce release size

[jlaine]

Currently our release size is around 1GB due to the large number of wheels we produce. This has started being a problem since we have hit the default 10 GB limit on PyPI, which is currently holding up release 11.0.0, see pypi/support#3324.

Cython 3 has added support for Python's limited API - which means a single wheel can cover multiple Python versions. This would greatly reduce the size of our releases and possibly speed up CI build times too.

[WyattBlue]

Cython does not support Py_LIMITED_API. See still open issue

[jlaine]

The changelog says otherwise:

https://cython.readthedocs.io/en/latest/src/changes.html#initial-support-for-limited-api

[da-woods]

To clarify - it's very preliminary in Cython 3. Some very trivial test examples will compile and run in with Py_LIMITED_API but most won't.

Cython 3.1 should be better - it'll still be fairly experimental but there's a decent chance things will work (although no promises). The main limitation is memoryviews which aren't possible to support on Python <3.11. No info on likely release dates of Cython 3.1 but I'd hold off on spending much time on this until then.

There may also be some limitations in build tooling that you've have to work around (e.g. setuptools doesn't give compiled files the right version tags)

[jlaine]

@da-woods thanks for investigating regarding the current state of Cython.

The tags issue is trivial to work around, i have for instance done so for the "aioquic" package.

[WyattBlue]

See: https://github.com/cython/cython/pull/6297/files