PEP 489 Compatibility

[davidhewitt]

Inspired by #2269 (comment), this is an issue to track all of the places which we know need to change for full PEP 489 compatibility (draft implementation in #2245).

The latest documentation about supporting isolated extensions can be found at https://docs.python.org/3.14/howto/isolating-extensions.html

• Rust statics - #[pyclass] type objects, most implementations of PyTypeInfo, intern!

[davidhewitt]

The above is currently incomplete - please post / edit things as you think of them

[messense]

Linking some related PEPs here:

• PEP 489 – Multi-phase extension module initialization
• PEP 573 – Module State Access from C Extension Methods
• PEP 630 – Isolating Extension Modules
• PEP 684 - A Per-Interpreter GIL
• PEP 687 – Isolating modules in the standard library

[davidhewitt]

Beginning to think that this is something that we may not want to do for now - see #2346 (comment)

[davidhewitt]

@bazaah following up from #5525 (comment)

Let's use this issue to track follow-up work.

Defining a way to set per-module state as part of the #[pymodule] macro will allow users to get a long way towards isolating their own modules. There is potentially a lot of work we need to do inside various function macro machinery to make that state accessible.

One other alternative to removing statics is to have interpreter-safe datastructures (e.g. store multiple values in some weakly-referenced form, indexed by interpreter ID).

[bazaah]

Yeah, my experimenting so far has lead me to the realization that exposing generics sanely while interacting with PyModule_GetState and #[pymodule] is pretty difficult to expose via an (e.g) PyModuleState<T>, as we have to erase and then "remember" the T later, inside the closure callbacks. I'm currently exploring a typemap approach instead, which is worse ergonomically, but shields pyo3 from having to care about the user's state T at all.

I haven't yet really started to look at what statics will need to be updated / moved. In general I think it will be pretty difficult to handle m_traverse stuff without storing stuff directly in the module state, but we probably just don't have to do that immediately either.

[bazaah]

@davidhewitt -- I'm trying to get my first go at this into a reviewable shape, and have a question around what exactly Python::attach / detach mean. These are referring to the "thread state" (ala python/cpython#127989), yes? More specifically, once a module (and it's PyModule_GetState state) is initialized, that module instance cannot move across threads, right? I have tried reading the cpython docs, but I'm still a bit confused 😆