feat: implement multi-phase initialization (PEP-489) #5525

bazaah · 2025-10-16T15:30:07Z

Description

This is a rebased version of #5142.

Within, we switch to using PEP-489 multi-phase initialization for pyo3 modules, over the previous single-phase initialization method.

This work lays the foundation for several major upsides for pyo3 (and consumers therein):

Improved control over pymodule initialization via rusty wrappers around the Py_mod_exec callback
Per module state, allowing consumers to avoid the use of rust statics when managing pymodule state
Support for subinterpreters

To be clear, this PR does not implement these, but it is a necessary base step towards them.

Of note, we also switch to gil_used = false by default in this PR, which may require consumers to explicitly opt in, if their pymodule is not thread safe. I expect this to be a relatively rare occurrence given rust's stance on thread safety.

davidhewitt

Thanks very much for picking this up, it's been a constant want to move forward with this but only so many things I can proceed on at once!

Have placed a full build label on the PR, so if you push any changes it'll trigger a full run. Last I recall, GraalPy testing was not working properly so that'll be what we need to understand before we can merge.

davidhewitt · 2025-10-17T10:04:12Z

src/impl_/pymodule.rs

+        // TODO: remove this once we default to `gil_used = false` (i.e. assume free-threading
+        // support in extensions). This is purely a convenience so that users only need to
+        // annotate the top-level module.


I would be keen to switch the default to gil_used = false in PyO3 0.28, which would allow us to simplify here.

I'll leave as is until I understand the graalpy failure, but it should be easy to drop the gil_used stuff out before final review

I've removed the gil_used_once stuff in ee6ebc3, and switched to gil_used = false by default in 84a512c.

I think these need a note in... somewhere. The migration guide maybe?

pyo3-ffi/src/modsupport.rs

bazaah · 2025-10-17T17:49:53Z

Interesting errors, I think this is because of

hint: Wheels are required for ... because building from source is disabled for all packages (i.e., with --no-build)

&

hint: You require GraalPy 3.11 (graalpy311), but we only found wheels for gevent (v23.9.0) with the following Python implementation tags: cp38, cp39, cp310, cp311, cp312

And I doubt there are prebuilts for the esoteric triples happening here. I'll look into this locally

davidhewitt · 2025-10-17T19:14:08Z

I think those warnings are red herrings, we should be skipping tests for any packages which don't support graalpy.

bazaah · 2025-10-20T15:04:15Z

Found the issue, eventually (thanks strace):

Function not implemented in GraalPy: PyModule_FromDefAndSpec2
Native stacktrace:
/home/<REDACTED>/.local/share/uv/python/graalpy-3.11.0-linux-x86_64-gnu/lib/graalpy24.2/libpython-native.so(+0x6e128) [0x7f7188390128]
/home/<REDACTED>/.local/share/uv/python/graalpy-3.11.0-linux-x86_64-gnu/lib/graalpy24.2/libpython-native.so(PyModule_FromDefAndSpec2+0xd) [0x7f718839223d]
/home/<REDACTED>/pyo3/examples/maturin-starter/.nox/python/lib/python3.11/site-packages/maturin_starter/maturin_starter.graalpy242-311-native-x86_64-linux.so(+0x656a9) [0x7f717159d6a9]
/home/<REDACTED>/pyo3/examples/maturin-starter/.nox/python/lib/python3.11/site-packages/maturin_starter/maturin_starter.graalpy242-311-native-x86_64-linux.so(+0x5ef78) [0x7f7171596f78]
/home/<REDACTED>/pyo3/examples/maturin-starter/.nox/python/lib/python3.11/site-packages/maturin_starter/maturin_starter.graalpy242-311-native-x86_64-linux.so(+0x4f433) [0x7f7171587433]
/home/<REDACTED>/pyo3/examples/maturin-starter/.nox/python/lib/python3.11/site-packages/maturin_starter/maturin_starter.graalpy242-311-native-x86_64-linux.so(+0x7b37c) [0x7f71715b337c]
/home/<REDACTED>/pyo3/examples/maturin-starter/.nox/python/lib/python3.11/site-packages/maturin_starter/maturin_starter.graalpy242-311-native-x86_64-linux.so(+0x7c3c6) [0x7f71715b43c6]
/home/<REDACTED>/pyo3/examples/maturin-starter/.nox/python/lib/python3.11/site-packages/maturin_starter/maturin_starter.graalpy242-311-native-x86_64-linux.so(+0x7d41b) [0x7f71715b541b]
/home/<REDACTED>/pyo3/examples/maturin-starter/.nox/python/lib/python3.11/site-packages/maturin_starter/maturin_starter.graalpy242-311-native-x86_64-linux.so(+0x7ad98) [0x7f71715b2d98]
/home/<REDACTED>/pyo3/examples/maturin-starter/.nox/python/lib/python3.11/site-packages/maturin_starter/maturin_starter.graalpy242-311-native-x86_64-linux.so(+0x72bf7) [0x7f71715aabf7]
/home/<REDACTED>/pyo3/examples/maturin-starter/.nox/python/lib/python3.11/site-packages/maturin_starter/maturin_starter.graalpy242-311-native-x86_64-linux.so(+0x4f2d4) [0x7f71715872d4]
/home/<REDACTED>/pyo3/examples/maturin-starter/.nox/python/lib/python3.11/site-packages/maturin_starter/maturin_starter.graalpy242-311-native-x86_64-linux.so(+0x4eb97) [0x7f7171586b97]
/home/<REDACTED>/pyo3/examples/maturin-starter/.nox/python/lib/python3.11/site-packages/maturin_starter/maturin_starter.graalpy242-311-native-x86_64-linux.so(+0x70592) [0x7f71715a8592]
/home/<REDACTED>/pyo3/examples/maturin-starter/.nox/python/lib/python3.11/site-packages/maturin_starter/maturin_starter.graalpy242-311-native-x86_64-linux.so(+0x4ea85) [0x7f7171586a85]
/home/<REDACTED>/pyo3/examples/maturin-starter/.nox/python/lib/python3.11/site-packages/maturin_starter/maturin_starter.graalpy242-311-native-x86_64-linux.so(+0x4687a) [0x7f717157e87a]
/home/<REDACTED>/pyo3/examples/maturin-starter/.nox/python/lib/python3.11/site-packages/maturin_starter/maturin_starter.graalpy242-311-native-x86_64-linux.so(+0x409d6) [0x7f71715789d6]

Not sure how we want to handle this @davidhewitt?

bazaah · 2025-10-20T15:14:47Z

Looks like graalpy supports these from v25.0.0 onwards: oracle/graalpython@9faded9

davidhewitt · 2025-10-20T15:58:23Z

Aha great find! In that case I think we should drop support for GraalPy 3.11, it seems justified if it allows us to get a big fix in.

That can be done as a separate PR and then we can rebase here.

bazaah · 2025-10-20T16:06:12Z

Okay, can you give me a quick rundown of what that entails? crate feature/cfgs, workflow bump?

ngoldbaum · 2025-10-20T16:10:53Z

#5116 is probably a good example to look at

bazaah · 2025-10-20T18:08:44Z

I get a clean test suite locally with graalpy-3.12.0-linux-x86_64-gnu, so once #5542 lands I'll cleanup this PR & switch to gil_used = false

bazaah · 2025-10-21T11:59:35Z

Not sure how to deal with the codecov failure -- it seems it mostly thinks PyModuleSlotsBuilder::with_gil_used isn't covered, but it's just wrong as far as I can tell?

Icxolu · 2025-10-22T20:10:16Z

src/impl_/pymodule.rs

+            // Because the array is c-style, the empty elements should be zeroed.
+            // `std::mem::zeroed()` requires msrv 1.75 for const
+            values: [ZEROED_SLOT; N],


we recently bumped msrv to 1.83

Switch to mem::zeroed in 01b44ed, though I think the explicitness of ZEROED_SLOT is better, personally

This code was for user convenience when gil_used defaulted to true, but since we're changing that to false, it is no longer needed.

davidhewitt

Thanks very much for helping push this over the line! I am happy with this code as-is, and I'll follow up by writing migration notes for users both related to gil_used and for the change to multi-phase init.

PyO3 0.28 will probably be shipping around December, so perhaps with some more iteration we can figure out what's needed for users to (unsafely) opt-in to subinterpreter support in time.

(We have a lot of globals which are not subinterpreter safe, so it might be that we need to iterate gradually and accept it'll be sometime next year when the support lands.)

bazaah · 2025-10-27T09:19:29Z

Thanks David!

I'll push it for as long as I have time. I think next up is looking into how to support a public interface to module state, creating an inventory of the aforementioned globals, and then a plan for how we'll handle them. I expect module state will play a part in that plan.

davidhewitt added the CI-build-full label Oct 17, 2025

davidhewitt reviewed Oct 17, 2025

View reviewed changes

bazaah force-pushed the feature/multi-phase-init branch from c84e6e5 to f056e32 Compare October 17, 2025 12:37

bazaah mentioned this pull request Oct 17, 2025

fix PyPyModule_ExecDef, PyPyModule_FromDefAndSpec2 definitions #5529

Merged

bazaah force-pushed the feature/multi-phase-init branch from f056e32 to edb4193 Compare October 17, 2025 17:32

davidhewitt added the CI-no-fail-fast If one job fails, allow the rest to keep testing label Oct 17, 2025

bazaah force-pushed the feature/multi-phase-init branch from edb4193 to d4e90fc Compare October 18, 2025 07:14

davidhewitt mentioned this pull request Oct 19, 2025

convert module initialization to use multi-phase-init #5142

Closed

bazaah force-pushed the feature/multi-phase-init branch from 74b35bf to b4e88e9 Compare October 20, 2025 11:29

bazaah mentioned this pull request Oct 20, 2025

bump supported graalpy to v25.0 #5542

Merged

bazaah force-pushed the feature/multi-phase-init branch from b7bb908 to fc9dcee Compare October 21, 2025 08:42

bazaah marked this pull request as ready for review October 21, 2025 12:00

bazaah force-pushed the feature/multi-phase-init branch from 84a512c to cf095d8 Compare October 22, 2025 09:20

Icxolu reviewed Oct 22, 2025

View reviewed changes

davidhewitt and others added 7 commits October 23, 2025 11:07

convert module initialization to use multi-phase-init

cdf210d

attempt to fix CI issues

7ec94aa

alternative spec implementation, fix oob read

21abced

fix MSRV

3cdadf7

need to exec multi-phase-init module

d1adbfd

tests/ui: rere invalid_result_conversion

311b972

newsfragments: add PR fragment

549019a

bazaah added 4 commits October 23, 2025 11:07

src/impl_: drop gil_used_once special casing

36a94c6

This code was for user convenience when gil_used defaulted to true, but since we're changing that to false, it is no longer needed.

pyo3-macros-backend: switch default gil_used to false

512bf59

tests/ui: rere invalid_property_args

8702979

src/impl_: use mem::zeroed over ZEROED_SLOT

01b44ed

bazaah force-pushed the feature/multi-phase-init branch from cf095d8 to 01b44ed Compare October 23, 2025 11:17

davidhewitt approved these changes Oct 26, 2025

View reviewed changes

davidhewitt added this pull request to the merge queue Oct 26, 2025

Merged via the queue into PyO3:main with commit 5589b17 Oct 26, 2025
82 of 84 checks passed

davidhewitt mentioned this pull request Oct 26, 2025

default to gil_used = false #5564

Merged

bazaah deleted the feature/multi-phase-init branch October 27, 2025 09:14

This was referenced Nov 1, 2025

PEP 489 Compatibility #2274

Open

docs: note of multi-phase init change, better slots handling #5587

Open

feat: implement multi-phase initialization (PEP-489) #5525

feat: implement multi-phase initialization (PEP-489) #5525

Conversation

bazaah commented Oct 16, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Uh oh!

davidhewitt left a comment

Choose a reason for hiding this comment

Uh oh!

davidhewitt Oct 17, 2025

Choose a reason for hiding this comment

Uh oh!

bazaah Oct 17, 2025

Choose a reason for hiding this comment

Uh oh!

bazaah Oct 21, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

bazaah commented Oct 17, 2025

Uh oh!

davidhewitt commented Oct 17, 2025

Uh oh!

bazaah commented Oct 20, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

bazaah commented Oct 20, 2025

Uh oh!

davidhewitt commented Oct 20, 2025

Uh oh!

bazaah commented Oct 20, 2025

Uh oh!

ngoldbaum commented Oct 20, 2025

Uh oh!

bazaah commented Oct 20, 2025

Uh oh!

bazaah commented Oct 21, 2025

Uh oh!

Icxolu Oct 22, 2025

Choose a reason for hiding this comment

Uh oh!

bazaah Oct 23, 2025

Choose a reason for hiding this comment

Uh oh!

davidhewitt left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

bazaah commented Oct 27, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

bazaah commented Oct 16, 2025 •

edited

Loading

bazaah commented Oct 20, 2025 •

edited

Loading