diff --git a/guide/src/free-threading.md b/guide/src/free-threading.md index 92d20eaffa4..ac38a236723 100644 --- a/guide/src/free-threading.md +++ b/guide/src/free-threading.md @@ -19,29 +19,36 @@ This document provides advice for porting Rust code using PyO3 to run under free ## Supporting free-threaded Python with PyO3 -Many simple uses of PyO3, like exposing bindings for a "pure" Rust function with no side-effects or defining an immutable Python class, will likely work "out of the box" on the free-threaded build. -All that will be necessary is to annotate Python modules declared by rust code in your project to declare that they support free-threaded Python, for example by declaring the module with `#[pymodule(gil_used = false)]`. +Since PyO3 0.28, PyO3 defaults to assuming Python modules created with it are thread-safe. +This will be the case except for Rust code which has used `unsafe` to assume thread-safety incorrectly. +An example of this is `unsafe` code which was written with the historical assumption that Python was single-threaded due to the GIL, and so the `Python<'py>` token used by PyO3 could be used to guarantee thread-safety. +A module can opt-out of supporting free-threaded Python until it has audited its `unsafe` code for correctness by declaring the module with `#[pymodule(gil_used = true)]` (see below). -More complicated `#[pyclass]` types may need to deal with thread-safety directly; there is [a dedicated section of the guide](./class/thread-safety.md) to discuss this. +Complicated `#[pyclass]` types may need to deal with thread-safety directly; there is [a dedicated section of the guide](./class/thread-safety.md) to discuss this. At a low-level, annotating a module sets the `Py_MOD_GIL` slot on modules defined by an extension to `Py_MOD_GIL_NOT_USED`, which allows the interpreter to see at runtime that the author of the extension thinks the extension is thread-safe. -You should only do this if you know that your extension is thread-safe. -Because of Rust's guarantees, this is already true for many extensions, however see below for more discussion about how to evaluate the thread safety of existing Rust extensions and how to think about the PyO3 API using a Python runtime with no GIL. -If you do not explicitly mark that modules are thread-safe, the Python interpreter will re-enable the GIL at runtime while importing your module and print a `RuntimeWarning` with a message containing the name of the module causing it to re-enable the GIL. +By opting-out of supporting free-threaded Python, the Python interpreter will re-enable the GIL at runtime while importing your module and print a `RuntimeWarning` with a message containing the name of the module causing it to re-enable the GIL. You can force the GIL to remain disabled by setting the `PYTHON_GIL=0` as an environment variable or passing `-Xgil=0` when starting Python (`0` means the GIL is turned off). If you are sure that all data structures exposed in a `PyModule` are thread-safe, then pass `gil_used = false` as a parameter to the `pymodule` procedural macro declaring the module or call `PyModule::gil_used` on a `PyModule` instance. For example: -```rust,no_run -use pyo3::prelude::*; +### Example opting-in + +(Note: for PyO3 versions 0.23 through 0.27, the default was `gil_used = true` and so the opposite was needed; modules needed to opt-in to free-threaded Python support with `gil_used = false`.) -/// This module supports free-threaded Python -#[pymodule(gil_used = false)] -fn my_extension(m: &Bound<'_, PyModule>) -> PyResult<()> { - // add members to the module that you know are thread-safe - Ok(()) +```rust,no_run +/// This module relies on the GIL for thread safety +#[pyo3::pymodule(gil_used = true)] +mod my_extension { + use pyo3::prelude::*; + + // this type is not thread-safe + #[pyclass] + struct MyNotThreadSafeType { + // insert not thread-safe code + } } ``` @@ -53,15 +60,12 @@ use pyo3::prelude::*; # #[allow(dead_code)] fn register_child_module(parent_module: &Bound<'_, PyModule>) -> PyResult<()> { let child_module = PyModule::new(parent_module.py(), "child_module")?; - child_module.gil_used(false)?; + child_module.gil_used(true)?; parent_module.add_submodule(&child_module) } ``` -For now you must explicitly opt in to free-threading support by annotating modules defined in your extension. -In a future version of `PyO3`, we plan to make `gil_used = false` the default. - See the [`string-sum`](https://github.com/PyO3/pyo3/tree/main/pyo3-ffi/examples/string-sum) example for how to declare free-threaded support using raw FFI calls for modules using single-phase initialization and the [`sequential`](https://github.com/PyO3/pyo3/tree/main/pyo3-ffi/examples/sequential) example for modules using multi-phase initialization. If you would like to use conditional compilation to trigger different code paths under the free-threaded build, you can use the `Py_GIL_DISABLED` attribute once you have configured your crate to generate the necessary build configuration data. diff --git a/guide/src/migration.md b/guide/src/migration.md index b25524a95aa..6e6d1922ae2 100644 --- a/guide/src/migration.md +++ b/guide/src/migration.md @@ -5,6 +5,13 @@ For a detailed list of all changes, see the [CHANGELOG](changelog.md). ## from 0.27.* to 0.28 +### Default to supporting free-threaded Python + +When PyO3 0.23 added support for free-threaded Python, this was as an opt-in feature for modules by annotating with `#[pymodule(gil_used = false)]`. + +As the support has matured and PyO3's own API has evolved to remove reliance on the GIL, the time is right to switch the default. +Modules now automatically allow use on free-threaded Python, unless they directly state they require the GIL with `#[pymodule(gil_used = true)]`. + ### Deprecation of automatic `FromPyObject` for `#[pyclass]` types which implement `Clone` `#[pyclass]` types which implement `Clone` used to also implement `FromPyObject` automatically. diff --git a/newsfragments/5564.packaging.md b/newsfragments/5564.packaging.md new file mode 100644 index 00000000000..51ee50528b8 --- /dev/null +++ b/newsfragments/5564.packaging.md @@ -0,0 +1 @@ +Support for free-threaded Python is now opt-out rather than opt-in. diff --git a/pyo3-macros-backend/src/module.rs b/pyo3-macros-backend/src/module.rs index 8eef9d57876..df00f319ac0 100644 --- a/pyo3-macros-backend/src/module.rs +++ b/pyo3-macros-backend/src/module.rs @@ -536,9 +536,6 @@ fn module_initialization( impl_::ModuleDef::new(__PYO3_NAME, #doc, &SLOTS) }; - #[doc(hidden)] - // so wrapped submodules can see what gil_used is - pub static __PYO3_GIL_USED: bool = #gil_used; }; if !is_submodule { result.extend(quote! { @@ -547,10 +544,7 @@ fn module_initialization( #[doc(hidden)] #[export_name = #pyinit_symbol] pub unsafe extern "C" fn __pyo3_init() -> *mut #pyo3_path::ffi::PyObject { - _PYO3_DEF.init_multi_phase( - unsafe { #pyo3_path::Python::assume_attached() }, - #gil_used - ) + _PYO3_DEF.init_multi_phase() } }); } diff --git a/pyo3-macros/src/lib.rs b/pyo3-macros/src/lib.rs index 6e4a46ee95e..04fa83b740d 100644 --- a/pyo3-macros/src/lib.rs +++ b/pyo3-macros/src/lib.rs @@ -27,6 +27,7 @@ use syn::{parse_macro_input, Item}; /// | `#[pyo3(submodule)]` | Skips adding a `PyInit_` FFI symbol to the compiled binary. | /// | `#[pyo3(module = "...")]` | Defines the Python `dotted.path` to the parent module for use in introspection. | /// | `#[pyo3(crate = "pyo3")]` | Defines the path to PyO3 to use code generated by the macro. | +/// | `#[pyo3(gil_used = true)]` | Declares the GIL is needed to run this module safely under free-threaded Python. | /// /// For more on creating Python modules see the [module section of the guide][1]. /// diff --git a/pytests/src/awaitable.rs b/pytests/src/awaitable.rs index 8481e2f6528..b8ba1ca3ca2 100644 --- a/pytests/src/awaitable.rs +++ b/pytests/src/awaitable.rs @@ -78,7 +78,7 @@ impl FutureAwaitable { } } -#[pymodule(gil_used = false)] +#[pymodule] pub fn awaitable(m: &Bound<'_, PyModule>) -> PyResult<()> { m.add_class::()?; m.add_class::()?; diff --git a/pytests/src/buf_and_str.rs b/pytests/src/buf_and_str.rs index 15230a5e153..bbaad40f312 100644 --- a/pytests/src/buf_and_str.rs +++ b/pytests/src/buf_and_str.rs @@ -47,7 +47,7 @@ fn return_memoryview(py: Python<'_>) -> PyResult> { PyMemoryView::from(&bytes) } -#[pymodule(gil_used = false)] +#[pymodule] pub fn buf_and_str(m: &Bound<'_, PyModule>) -> PyResult<()> { m.add_class::()?; m.add_function(wrap_pyfunction!(return_memoryview, m)?)?; diff --git a/pytests/src/comparisons.rs b/pytests/src/comparisons.rs index df28b423c3c..70edb52dede 100644 --- a/pytests/src/comparisons.rs +++ b/pytests/src/comparisons.rs @@ -147,7 +147,7 @@ impl OrderedDefaultNe { } } -#[pymodule(gil_used = false)] +#[pymodule] pub mod comparisons { #[pymodule_export] use super::{ diff --git a/pytests/src/datetime.rs b/pytests/src/datetime.rs index 341a5e0536f..d4919db7cf7 100644 --- a/pytests/src/datetime.rs +++ b/pytests/src/datetime.rs @@ -203,7 +203,7 @@ impl TzClass { } } -#[pymodule(gil_used = false)] +#[pymodule] pub fn datetime(m: &Bound<'_, PyModule>) -> PyResult<()> { m.add_function(wrap_pyfunction!(make_date, m)?)?; m.add_function(wrap_pyfunction!(get_date_tuple, m)?)?; diff --git a/pytests/src/enums.rs b/pytests/src/enums.rs index 02431fdd1e8..8ccca3336c8 100644 --- a/pytests/src/enums.rs +++ b/pytests/src/enums.rs @@ -1,6 +1,6 @@ use pyo3::{pyclass, pyfunction, pymodule}; -#[pymodule(gil_used = false)] +#[pymodule] pub mod enums { #[pymodule_export] use super::{ diff --git a/pytests/src/lib.rs b/pytests/src/lib.rs index 76c07937eed..26a33f81cab 100644 --- a/pytests/src/lib.rs +++ b/pytests/src/lib.rs @@ -18,7 +18,7 @@ pub mod pyfunctions; pub mod sequence; pub mod subclassing; -#[pymodule(gil_used = false)] +#[pymodule] mod pyo3_pytests { use super::*; diff --git a/pytests/src/misc.rs b/pytests/src/misc.rs index fee824a66fa..0e6b8dd5ccc 100644 --- a/pytests/src/misc.rs +++ b/pytests/src/misc.rs @@ -54,7 +54,7 @@ fn get_item_and_run_callback(dict: Bound<'_, PyDict>, callback: Bound<'_, PyAny> Ok(()) } -#[pymodule(gil_used = false)] +#[pymodule] pub fn misc(m: &Bound<'_, PyModule>) -> PyResult<()> { m.add_function(wrap_pyfunction!(issue_219, m)?)?; m.add_function(wrap_pyfunction!(hammer_attaching_in_thread, m)?)?; diff --git a/pytests/src/objstore.rs b/pytests/src/objstore.rs index 89643c3967d..af5cf5776a4 100644 --- a/pytests/src/objstore.rs +++ b/pytests/src/objstore.rs @@ -18,7 +18,7 @@ impl ObjStore { } } -#[pymodule(gil_used = false)] +#[pymodule] pub fn objstore(m: &Bound<'_, PyModule>) -> PyResult<()> { m.add_class::() } diff --git a/pytests/src/othermod.rs b/pytests/src/othermod.rs index 0de912d7d04..36ad4b5e23e 100644 --- a/pytests/src/othermod.rs +++ b/pytests/src/othermod.rs @@ -28,7 +28,7 @@ fn double(x: i32) -> i32 { x * 2 } -#[pymodule(gil_used = false)] +#[pymodule] pub fn othermod(m: &Bound<'_, PyModule>) -> PyResult<()> { m.add_function(wrap_pyfunction!(double, m)?)?; diff --git a/pytests/src/path.rs b/pytests/src/path.rs index b52c038ed34..0675e56d13a 100644 --- a/pytests/src/path.rs +++ b/pytests/src/path.rs @@ -11,7 +11,7 @@ fn take_pathbuf(path: PathBuf) -> PathBuf { path } -#[pymodule(gil_used = false)] +#[pymodule] pub fn path(m: &Bound<'_, PyModule>) -> PyResult<()> { m.add_function(wrap_pyfunction!(make_path, m)?)?; m.add_function(wrap_pyfunction!(take_pathbuf, m)?)?; diff --git a/pytests/src/pyclasses.rs b/pytests/src/pyclasses.rs index ca943833f24..8d12afc5fb1 100644 --- a/pytests/src/pyclasses.rs +++ b/pytests/src/pyclasses.rs @@ -168,7 +168,7 @@ fn map_a_class(cls: AClass) -> AClass { cls } -#[pymodule(gil_used = false)] +#[pymodule] pub mod pyclasses { #[cfg(any(Py_3_10, not(Py_LIMITED_API)))] #[pymodule_export] diff --git a/pytests/src/sequence.rs b/pytests/src/sequence.rs index 175f5fba8aa..f552b4048b8 100644 --- a/pytests/src/sequence.rs +++ b/pytests/src/sequence.rs @@ -16,7 +16,7 @@ fn vec_to_vec_pystring(vec: Vec>) -> Vec vec } -#[pymodule(gil_used = false)] +#[pymodule] pub fn sequence(m: &Bound<'_, PyModule>) -> PyResult<()> { m.add_function(wrap_pyfunction!(vec_to_vec_i32, m)?)?; m.add_function(wrap_pyfunction!(array_to_array_i32, m)?)?; diff --git a/pytests/src/subclassing.rs b/pytests/src/subclassing.rs index 0f00e74c19d..8e451cd9183 100644 --- a/pytests/src/subclassing.rs +++ b/pytests/src/subclassing.rs @@ -17,7 +17,7 @@ impl Subclassable { } } -#[pymodule(gil_used = false)] +#[pymodule] pub fn subclassing(m: &Bound<'_, PyModule>) -> PyResult<()> { m.add_class::()?; Ok(()) diff --git a/src/impl_/pymodule.rs b/src/impl_/pymodule.rs index 1ad5a4f499f..b1bf6fb8878 100644 --- a/src/impl_/pymodule.rs +++ b/src/impl_/pymodule.rs @@ -47,8 +47,6 @@ pub struct ModuleDef { interpreter: AtomicI64, /// Initialized module object, cached to avoid reinitialization. module: PyOnceLock>, - /// Whether or not the module supports running without the GIL - gil_used: bool, } unsafe impl Sync for ModuleDef {} @@ -95,17 +93,16 @@ impl ModuleDef { ))] interpreter: AtomicI64::new(-1), module: PyOnceLock::new(), - gil_used: false, } } - pub fn init_multi_phase(&'static self, _py: Python<'_>, _gil_used: bool) -> *mut ffi::PyObject { + pub fn init_multi_phase(&'static self) -> *mut ffi::PyObject { + // SAFETY: `ffi_def` is correctly initialized in `new()` unsafe { ffi::PyModuleDef_Init(self.ffi_def.get()) } } /// Builds a module object directly. Used for [`#[pymodule]`][crate::pymodule] submodules. - #[cfg_attr(any(Py_LIMITED_API, not(Py_GIL_DISABLED)), allow(unused_variables))] - pub fn make_module(&'static self, py: Python<'_>, _gil_used: bool) -> PyResult> { + pub fn make_module(&'static self, py: Python<'_>) -> PyResult> { // Check the interpreter ID has not changed, since we currently have no way to guarantee // that static data is not reused across interpreters. // @@ -307,10 +304,7 @@ impl PyAddToModule for PyFunctionDef { /// For adding a module to a module. impl PyAddToModule for ModuleDef { fn add_to_module(&'static self, module: &Bound<'_, PyModule>) -> PyResult<()> { - module.add_submodule( - self.make_module(module.py(), self.gil_used)? - .bind(module.py()), - ) + module.add_submodule(self.make_module(module.py())?.bind(module.py())) } } @@ -347,7 +341,7 @@ mod tests { static MODULE_DEF: ModuleDef = ModuleDef::new(c"test_module", c"some doc", &SLOTS); Python::attach(|py| { - let module = MODULE_DEF.make_module(py, false).unwrap().into_bound(py); + let module = MODULE_DEF.make_module(py).unwrap().into_bound(py); assert_eq!( module .getattr("__name__") diff --git a/src/macros.rs b/src/macros.rs index 0af8251603d..b6d501b9361 100644 --- a/src/macros.rs +++ b/src/macros.rs @@ -159,7 +159,7 @@ macro_rules! wrap_pymodule { &|py| { use $module as wrapped_pymodule; wrapped_pymodule::_PYO3_DEF - .make_module(py, wrapped_pymodule::__PYO3_GIL_USED) + .make_module(py) .expect("failed to wrap pymodule") } }; diff --git a/src/types/module.rs b/src/types/module.rs index 82f3f176180..2337ade8194 100644 --- a/src/types/module.rs +++ b/src/types/module.rs @@ -51,11 +51,16 @@ impl PyModule { /// ``` pub fn new<'py>(py: Python<'py>, name: &str) -> PyResult> { let name = PyString::new(py, name); - unsafe { + let module = unsafe { ffi::PyModule_NewObject(name.as_ptr()) - .assume_owned_or_err(py) + .assume_owned_or_err(py)? .cast_into_unchecked() - } + }; + + // By default, PyO3 assumes modules use the GIL for thread safety. + module.gil_used(false)?; + + Ok(module) } /// Imports the Python module with the specified name. @@ -367,34 +372,30 @@ pub trait PyModuleMethods<'py>: crate::sealed::Sealed { /// Declare whether or not this module supports running with the GIL disabled /// - /// If the module does not rely on the GIL for thread safety, you can pass - /// `false` to this function to indicate the module does not rely on the GIL - /// for thread-safety. + /// Since PyO3 0.28, PyO3 defaults to assuming that modules do not require the + /// GIL for thread safety. Call this function with `true` to opt-out of supporting + /// free-threaded Python. /// /// This function sets the [`Py_MOD_GIL` /// slot](https://docs.python.org/3/c-api/module.html#c.Py_mod_gil) on the - /// module object. The default is `Py_MOD_GIL_USED`, so passing `true` to - /// this function is a no-op unless you have already set `Py_MOD_GIL` to - /// `Py_MOD_GIL_NOT_USED` elsewhere. + /// module object. /// /// # Examples /// /// ```rust,no_run /// use pyo3::prelude::*; /// - /// #[pymodule(gil_used = false)] + /// #[pymodule] /// fn my_module(py: Python<'_>, module: &Bound<'_, PyModule>) -> PyResult<()> { /// let submodule = PyModule::new(py, "submodule")?; - /// submodule.gil_used(false)?; + /// submodule.gil_used(true)?; /// module.add_submodule(&submodule)?; /// Ok(()) /// } /// ``` /// - /// The resulting module will not print a `RuntimeWarning` and re-enable the - /// GIL when Python imports it on the free-threaded build, since all module - /// objects defined in the extension have `Py_MOD_GIL` set to - /// `Py_MOD_GIL_NOT_USED`. + /// The resulting module will print a `RuntimeWarning` and re-enable the + /// GIL when Python imports it on the free-threaded build. /// /// This is a no-op on the GIL-enabled build. fn gil_used(&self, gil_used: bool) -> PyResult<()>; diff --git a/tests/test_append_to_inittab.rs b/tests/test_append_to_inittab.rs index 2da4f29efb2..e147967a0c7 100644 --- a/tests/test_append_to_inittab.rs +++ b/tests/test_append_to_inittab.rs @@ -7,13 +7,13 @@ fn foo() -> usize { 123 } -#[pymodule(gil_used = false)] +#[pymodule] fn module_fn_with_functions(m: &Bound<'_, PyModule>) -> PyResult<()> { m.add_function(wrap_pyfunction!(foo, m)?)?; Ok(()) } -#[pymodule(gil_used = false)] +#[pymodule] mod module_mod_with_functions { #[pymodule_export] use super::foo; diff --git a/tests/test_module.rs b/tests/test_module.rs index 211a2f5f21a..55b64845dce 100644 --- a/tests/test_module.rs +++ b/tests/test_module.rs @@ -38,7 +38,7 @@ fn test_module_with_functions() { use pyo3::wrap_pymodule; /// This module is implemented in Rust. - #[pymodule(gil_used = false)] + #[pymodule] mod module_with_functions { use super::*; @@ -125,7 +125,7 @@ fn test_module_with_pyfn() { use pyo3::wrap_pymodule; /// This module is implemented in Rust. - #[pymodule(gil_used = false)] + #[pymodule] fn module_with_pyfn(m: &Bound<'_, PyModule>) -> PyResult<()> { #[pyfn(m)] #[pyo3(name = "no_parameters")] @@ -263,8 +263,6 @@ fn test_module_from_code_bound() { .extract() .expect("The value should be able to be converted to an i32"); - adder_mod.gil_used(false).expect("Disabling the GIL failed"); - assert_eq!(ret_value, 3); }); }