Skip to content

Conversation

@hppritcha
Copy link
Member

@hppritcha hppritcha commented May 27, 2025

Two external MPI libraries are now created: libmpi.so and libmpi_abi.so.
Backend code that was originally in libmpi.la has been extracted into
libopen-mpi.la to be linked into both libraries.

Parts of the Open MPI C interface are now being generated by a python
script (abi.py) from modified source files (named with *.in). This
script generates files for both the ompi ABI and the standard ABI from
the same source file, also including new bigcount interfaces.

To compile standard ABI code, there's a new mpicc_abi compiler wrapper.

A few todos left:

  • simplify python framework ,
  • integrate the self standing mpi.h/abi.h generator into the python framework
  • add a bit of plumbing to assist with cleanup of resources used by the attributes callback function wrappers
  • write up docs to explain ABI support both for users and for developers
  • add man pages
  • open separate PR to deal with the libnbc release_arrays problem

This PR supercedes #12033

@dalcinl
Copy link
Contributor

dalcinl commented Jul 11, 2025

Maybe you should somehow vendor the mpi.h header from https://github.com/mpi-forum/mpi-abi-stubs and use it as the baseline to extract values for handles and constants? That's what MPICH is doing.
Alternatively, the mpi.h header the mpi-abi-stubs repo could be downloaded on the fly by some Python script and the values in that header used to update stuff to be committed in the ompi repo.

In short, I think it would be in everyone's convenience to use https://github.com/mpi-forum/mpi-abi-stubs as the "source of truth" for ABI-related stuff, avoiding manual synchronization of handle/constant values.

@github-actions
Copy link

Hello! The Git Commit Checker CI bot found a few problems with this PR:

f7d94fa: WIP: explain issue with pympistandard for callback...

  • check_signed_off: does not contain a valid Signed-off-by line

4d79937: WIP: fix JSONs

  • check_signed_off: does not contain a valid Signed-off-by line

ec9c45a: WIP: fix typo in pympistd arg

  • check_signed_off: does not contain a valid Signed-off-by line

b047b19: WIP: add JSONs for ABI and API

  • check_signed_off: does not contain a valid Signed-off-by line

c9d0c7a: WIP: bump pympistandard commit for profiling embig...

  • check_signed_off: does not contain a valid Signed-off-by line

a1255ce: WIP: move Aint helper macros under ifndef OMPI_NO_...

  • check_signed_off: does not contain a valid Signed-off-by line

342b072: WIP: add some workarounds for MPI_Fint and MPI_Inf...

  • check_signed_off: does not contain a valid Signed-off-by line

53aeac5: WIP: mangle some more functions

  • check_signed_off: does not contain a valid Signed-off-by line

93c0a39: WIP: avoid double inclusion of abi.h

  • check_signed_off: does not contain a valid Signed-off-by line

afd9eb2: WIP: use pympistandard by editing PYTHONPATH (inst...

  • check_signed_off: does not contain a valid Signed-off-by line

d397bfc: WIP: fix some bugs in mangling names

  • check_signed_off: does not contain a valid Signed-off-by line

da2630f: WIP: fix typo for MPI_internal

  • check_signed_off: does not contain a valid Signed-off-by line

956dded: WIP: add additional types and functions to be mang...

  • check_signed_off: does not contain a valid Signed-off-by line

0b10ce6: WIP: temp fix for Aint problems

  • check_signed_off: does not contain a valid Signed-off-by line

9830327: WIP: add input for abi.h.in

  • check_signed_off: does not contain a valid Signed-off-by line

68c69df: WIP: move abi.h.in

  • check_signed_off: does not contain a valid Signed-off-by line

702cb23: WIP: add in 5.0 apis.json

  • check_signed_off: does not contain a valid Signed-off-by line

6b2784f: WIP: move code out of consts.py

  • check_signed_off: does not contain a valid Signed-off-by line

22d4cb2: WIP: call c_header from Makefile

  • check_signed_off: does not contain a valid Signed-off-by line

b0d0ff2: WIP: mangle names for internal usage

  • check_signed_off: does not contain a valid Signed-off-by line

96a33c3: WIP: generate callback function prototypes

  • check_signed_off: does not contain a valid Signed-off-by line

d2dd7ed: WIP: remove comment function

  • check_signed_off: does not contain a valid Signed-off-by line

45d8789: WIP: print out embiggened versions of functions

  • check_signed_off: does not contain a valid Signed-off-by line

b081aa2: WIP: add MPI and ABI versions

  • check_signed_off: does not contain a valid Signed-off-by line

80ebfe7: WIP: generate API prototypes

  • check_signed_off: does not contain a valid Signed-off-by line

93e6375: WIP: Comment out a Fortran-only category

  • check_signed_off: does not contain a valid Signed-off-by line

db8a2b5: WIP: add comment pointing back to MPI standard

  • check_signed_off: does not contain a valid Signed-off-by line

08dfb42: WIP: use enums for most int values

  • check_signed_off: does not contain a valid Signed-off-by line

aab2023: WIP: create ABI header file from template with cat...

  • check_signed_off: does not contain a valid Signed-off-by line

46fae8e: WIP: generate header with ABI values for #defines

  • check_signed_off: does not contain a valid Signed-off-by line

c15a05d: WIP: remove abi.py

  • check_signed_off: does not contain a valid Signed-off-by line

Please fix these problems and, if necessary, force-push new commits back up to the PR branch. Thanks!

1 similar comment
@github-actions
Copy link

Hello! The Git Commit Checker CI bot found a few problems with this PR:

f7d94fa: WIP: explain issue with pympistandard for callback...

  • check_signed_off: does not contain a valid Signed-off-by line

4d79937: WIP: fix JSONs

  • check_signed_off: does not contain a valid Signed-off-by line

ec9c45a: WIP: fix typo in pympistd arg

  • check_signed_off: does not contain a valid Signed-off-by line

b047b19: WIP: add JSONs for ABI and API

  • check_signed_off: does not contain a valid Signed-off-by line

c9d0c7a: WIP: bump pympistandard commit for profiling embig...

  • check_signed_off: does not contain a valid Signed-off-by line

a1255ce: WIP: move Aint helper macros under ifndef OMPI_NO_...

  • check_signed_off: does not contain a valid Signed-off-by line

342b072: WIP: add some workarounds for MPI_Fint and MPI_Inf...

  • check_signed_off: does not contain a valid Signed-off-by line

53aeac5: WIP: mangle some more functions

  • check_signed_off: does not contain a valid Signed-off-by line

93c0a39: WIP: avoid double inclusion of abi.h

  • check_signed_off: does not contain a valid Signed-off-by line

afd9eb2: WIP: use pympistandard by editing PYTHONPATH (inst...

  • check_signed_off: does not contain a valid Signed-off-by line

d397bfc: WIP: fix some bugs in mangling names

  • check_signed_off: does not contain a valid Signed-off-by line

da2630f: WIP: fix typo for MPI_internal

  • check_signed_off: does not contain a valid Signed-off-by line

956dded: WIP: add additional types and functions to be mang...

  • check_signed_off: does not contain a valid Signed-off-by line

0b10ce6: WIP: temp fix for Aint problems

  • check_signed_off: does not contain a valid Signed-off-by line

9830327: WIP: add input for abi.h.in

  • check_signed_off: does not contain a valid Signed-off-by line

68c69df: WIP: move abi.h.in

  • check_signed_off: does not contain a valid Signed-off-by line

702cb23: WIP: add in 5.0 apis.json

  • check_signed_off: does not contain a valid Signed-off-by line

6b2784f: WIP: move code out of consts.py

  • check_signed_off: does not contain a valid Signed-off-by line

22d4cb2: WIP: call c_header from Makefile

  • check_signed_off: does not contain a valid Signed-off-by line

b0d0ff2: WIP: mangle names for internal usage

  • check_signed_off: does not contain a valid Signed-off-by line

96a33c3: WIP: generate callback function prototypes

  • check_signed_off: does not contain a valid Signed-off-by line

d2dd7ed: WIP: remove comment function

  • check_signed_off: does not contain a valid Signed-off-by line

45d8789: WIP: print out embiggened versions of functions

  • check_signed_off: does not contain a valid Signed-off-by line

b081aa2: WIP: add MPI and ABI versions

  • check_signed_off: does not contain a valid Signed-off-by line

80ebfe7: WIP: generate API prototypes

  • check_signed_off: does not contain a valid Signed-off-by line

93e6375: WIP: Comment out a Fortran-only category

  • check_signed_off: does not contain a valid Signed-off-by line

db8a2b5: WIP: add comment pointing back to MPI standard

  • check_signed_off: does not contain a valid Signed-off-by line

08dfb42: WIP: use enums for most int values

  • check_signed_off: does not contain a valid Signed-off-by line

aab2023: WIP: create ABI header file from template with cat...

  • check_signed_off: does not contain a valid Signed-off-by line

46fae8e: WIP: generate header with ABI values for #defines

  • check_signed_off: does not contain a valid Signed-off-by line

c15a05d: WIP: remove abi.py

  • check_signed_off: does not contain a valid Signed-off-by line

Please fix these problems and, if necessary, force-push new commits back up to the PR branch. Thanks!

@github-actions
Copy link

Hello! The Git Commit Checker CI bot found a few problems with this PR:

e86186b: WIP: add JSONs for ABI and API

  • check_signed_off: does not contain a valid Signed-off-by line

24417ec: WIP: bump pympistandard commit for profiling embig...

  • check_signed_off: does not contain a valid Signed-off-by line

718b1d0: WIP: move Aint helper macros under ifndef OMPI_NO_...

  • check_signed_off: does not contain a valid Signed-off-by line

3431c8d: WIP: add some workarounds for MPI_Fint and MPI_Inf...

  • check_signed_off: does not contain a valid Signed-off-by line

566fdaa: WIP: mangle some more functions

  • check_signed_off: does not contain a valid Signed-off-by line

8396bef: WIP: avoid double inclusion of abi.h

  • check_signed_off: does not contain a valid Signed-off-by line

39c20b7: WIP: use pympistandard by editing PYTHONPATH (inst...

  • check_signed_off: does not contain a valid Signed-off-by line

d1aece4: WIP: fix some bugs in mangling names

  • check_signed_off: does not contain a valid Signed-off-by line

c7c1809: WIP: fix typo for MPI_internal

  • check_signed_off: does not contain a valid Signed-off-by line

2bbf9eb: WIP: add additional types and functions to be mang...

  • check_signed_off: does not contain a valid Signed-off-by line

1db8082: WIP: temp fix for Aint problems

  • check_signed_off: does not contain a valid Signed-off-by line

870e925: WIP: add input for abi.h.in

  • check_signed_off: does not contain a valid Signed-off-by line

a56da85: WIP: move abi.h.in

  • check_signed_off: does not contain a valid Signed-off-by line

4c2aee1: WIP: add in 5.0 apis.json

  • check_signed_off: does not contain a valid Signed-off-by line

3d85943: WIP: move code out of consts.py

  • check_signed_off: does not contain a valid Signed-off-by line

4e85726: WIP: call c_header from Makefile

  • check_signed_off: does not contain a valid Signed-off-by line

8d8f554: WIP: mangle names for internal usage

  • check_signed_off: does not contain a valid Signed-off-by line

5f29a48: WIP: generate callback function prototypes

  • check_signed_off: does not contain a valid Signed-off-by line

c5d5f30: WIP: remove comment function

  • check_signed_off: does not contain a valid Signed-off-by line

1613149: WIP: print out embiggened versions of functions

  • check_signed_off: does not contain a valid Signed-off-by line

3f229b3: WIP: add MPI and ABI versions

  • check_signed_off: does not contain a valid Signed-off-by line

584aeb7: WIP: generate API prototypes

  • check_signed_off: does not contain a valid Signed-off-by line

7c73091: WIP: Comment out a Fortran-only category

  • check_signed_off: does not contain a valid Signed-off-by line

72d2bff: WIP: add comment pointing back to MPI standard

  • check_signed_off: does not contain a valid Signed-off-by line

c303345: WIP: use enums for most int values

  • check_signed_off: does not contain a valid Signed-off-by line

7d79681: WIP: create ABI header file from template with cat...

  • check_signed_off: does not contain a valid Signed-off-by line

99e9992: WIP: generate header with ABI values for #defines

  • check_signed_off: does not contain a valid Signed-off-by line

Please fix these problems and, if necessary, force-push new commits back up to the PR branch. Thanks!

@jsquyres
Copy link
Member

I wonder if we should make the bot not complain about unsigned commits on draft PRs. That would reduce some of the noise on PR's like this.

hppritcha added a commit to hppritcha/ompi that referenced this pull request Aug 25, 2025
Turns out that in commit 6bd36a7 we had a function that is not part of the MPI standard.
This showed while working on ABI support - which requires us to pay attention to the truth
rather than make stuff up.

This commit removes our made up MPI_Session_set_info method.
Turns out who ever was doing the fortran bindings knew this wasn't a method in the standard
so there's no need to change the fortran bindings.  Same thing applies to the man pages.

Related to open-mpi#13280

Signed-off-by: Howard Pritchard <howardp@lanl.gov>
@dalcinl
Copy link
Contributor

dalcinl commented Aug 27, 2025

@hppritcha There is some issue with out-of-source builds, i.e

git clone ... ompi-main
...
mkdir -p ompi-BUILD/main
cd ompi-BUILD/main
../../ompi-main/configure ...
make install
Making install in mpi/c
make[2]: Entering directory '/home/dalcinl/Devel/REPOS/ompi-BUILD/main/ompi/mpi/c'
mkdir -p standard_abi
  GENERATE abi.h
  GENERATE standard_abi/mpi.h
Traceback (most recent call last):
  File "/home/dalcinl/Devel/REPOS/ompi-BUILD/main/ompi/mpi/c/../../../../../ompi-main/ompi/mpi/bindings/c_header.py", line 262, in <module>
    with open(OUTPUT, 'tw') as header_out:
         ~~~~^^^^^^^^^^^^^^
FileNotFoundError: [Errno 2] No such file or directory: '../../../../../ompi-main/ompi/mpi/c/standard_abi/mpi.h'
make[2]: *** [Makefile:23335: standard_abi/mpi.h] Error 1
make[2]: *** Waiting for unfinished jobs....
make[2]: Leaving directory '/home/dalcinl/Devel/REPOS/ompi-BUILD/main/ompi/mpi/c'
make[1]: *** [Makefile:2785: install-recursive] Error 1
make[1]: Leaving directory '/home/dalcinl/Devel/REPOS/ompi-BUILD/main/ompi'
make: *** [Makefile:1526: install-recursive] Error 1

@hppritcha
Copy link
Member Author

interesting distcheck didn't check this.

@hppritcha
Copy link
Member Author

jenkins ci runs make distcheck

@dalcinl
Copy link
Contributor

dalcinl commented Aug 28, 2025

The ABI mpi.h header is missing some MPI_T_XXX types. Also, the Status f08/c converters are declared, and they should not.

I did the following manual edits to the installed mpi.h header:

diff -up ./mpi.h.orig ./mpi.h
--- ./mpi.h.orig	2025-08-28 18:55:49.842968779 +0300
+++ ./mpi.h	2025-08-28 19:03:07.192957305 +0300
@@ -490,6 +490,13 @@ enum {
 /* C preprocessor constants and Fortran parameters */
 /* $CATEGORY:C_PREPROCESSOR_CONSTANTS_FORTRAN_PARAMETERS$ */
 
+typedef struct MPI_T_enum_t* MPI_T_enum;
+typedef struct MPI_T_cvar_handle_t* MPI_T_cvar_handle;
+typedef struct MPI_T_pvar_handle_t* MPI_T_pvar_handle;
+typedef struct MPI_T_pvar_session_t* MPI_T_pvar_session;
+typedef struct MPI_T_event_registration_t* MPI_T_event_registration;
+typedef struct MPI_T_event_instance_t* MPI_T_event_instance;
+
 /* Handles used in the MPI tool information interface */
 #define MPI_T_ENUM_NULL                       ((MPI_T_enum) 0)
 #define MPI_T_CVAR_HANDLE_NULL                ((MPI_T_cvar_handle) 0)
@@ -558,20 +565,20 @@ enum {
 };
 
 /* Source event ordering guarantees in the MPI tool information interface */
-enum {
+typedef enum MPI_T_source_order {
     MPI_T_SOURCE_ORDERED                      = 1,
     MPI_T_SOURCE_UNORDERED                    = 2,
-};
+} MPI_T_source_order;
 
 /*
  * Callback safety requirement levels used in the MPI tool information interface
  */
-enum {
+typedef enum MPI_T_cb_safety {
     MPI_T_CB_REQUIRE_NONE                     = 0x00,
     MPI_T_CB_REQUIRE_MPI_RESTRICTED           = 0x03,
     MPI_T_CB_REQUIRE_THREAD_SAFE              = 0x0F,
     MPI_T_CB_REQUIRE_ASYNC_SIGNAL_SAFE        = 0x3F,
-};
+} MPI_T_cb_safety;
 
 
 /* Callback functions */
@@ -1107,13 +1114,13 @@ int MPI_Ssend_init(const void* buf, int
 int MPI_Ssend_init_c(const void* buf, MPI_Count count, MPI_Datatype datatype, int dest, int tag, MPI_Comm comm, MPI_Request* request);
 int MPI_Start(MPI_Request* request);
 int MPI_Startall(int count, MPI_Request array_of_requests[]);
-/* int MPI_Status_c2f(const MPI_Status* c_status, MPI_Fint* f_status);
- */int MPI_Status_c2f08(const MPI_Status* c_status, MPI_F08_status* f08_status);
-int MPI_Status_f082c(const MPI_F08_status* f08_status, MPI_Status* c_status);
-/* int MPI_Status_f082f(const MPI_F08_status* f08_status, MPI_Fint* f_status);
- *//* int MPI_Status_f2c(const MPI_Fint* f_status, MPI_Status* c_status);
- *//* int MPI_Status_f2f08(const MPI_Fint* f_status, MPI_F08_status* f08_status);
- */int MPI_Status_get_error(const MPI_Status* status, int* err);
+// /* int MPI_Status_c2f(const MPI_Status* c_status, MPI_Fint* f_status);
+//  */int MPI_Status_c2f08(const MPI_Status* c_status, MPI_F08_status* f08_status);
+// int MPI_Status_f082c(const MPI_F08_status* f08_status, MPI_Status* c_status);
+// /* int MPI_Status_f082f(const MPI_F08_status* f08_status, MPI_Fint* f_status);
+//  *//* int MPI_Status_f2c(const MPI_Fint* f_status, MPI_Status* c_status);
+//  *//* int MPI_Status_f2f08(const MPI_Fint* f_status, MPI_F08_status* f08_status);
+//  */int MPI_Status_get_error(const MPI_Status* status, int* err);
 int MPI_Status_get_source(const MPI_Status* status, int* source);
 int MPI_Status_get_tag(const MPI_Status* status, int* tag);
 int MPI_Status_set_cancelled(MPI_Status* status, int flag);
@@ -1799,13 +1806,13 @@ int PMPI_Ssend_init(const void* buf, int
 int PMPI_Ssend_init_c(const void* buf, MPI_Count count, MPI_Datatype datatype, int dest, int tag, MPI_Comm comm, MPI_Request* request);
 int PMPI_Start(MPI_Request* request);
 int PMPI_Startall(int count, MPI_Request array_of_requests[]);
-/* int PMPI_Status_c2f(const MPI_Status* c_status, MPI_Fint* f_status);
- */int PMPI_Status_c2f08(const MPI_Status* c_status, MPI_F08_status* f08_status);
-int PMPI_Status_f082c(const MPI_F08_status* f08_status, MPI_Status* c_status);
-/* int PMPI_Status_f082f(const MPI_F08_status* f08_status, MPI_Fint* f_status);
- *//* int PMPI_Status_f2c(const MPI_Fint* f_status, MPI_Status* c_status);
- *//* int PMPI_Status_f2f08(const MPI_Fint* f_status, MPI_F08_status* f08_status);
- */int PMPI_Status_get_error(const MPI_Status* status, int* err);
+// /* int PMPI_Status_c2f(const MPI_Status* c_status, MPI_Fint* f_status);
+//  */int PMPI_Status_c2f08(const MPI_Status* c_status, MPI_F08_status* f08_status);
+// int PMPI_Status_f082c(const MPI_F08_status* f08_status, MPI_Status* c_status);
+// /* int PMPI_Status_f082f(const MPI_F08_status* f08_status, MPI_Fint* f_status);
+//  *//* int PMPI_Status_f2c(const MPI_Fint* f_status, MPI_Status* c_status);
+//  *//* int PMPI_Status_f2f08(const MPI_Fint* f_status, MPI_F08_status* f08_status);
+//  */int PMPI_Status_get_error(const MPI_Status* status, int* err);
 int PMPI_Status_get_source(const MPI_Status* status, int* source);
 int PMPI_Status_get_tag(const MPI_Status* status, int* tag);
 int PMPI_Status_set_cancelled(MPI_Status* status, int flag);

I'm able to compile (using mpicc_abi) the pure-C demo/helloworld.c file from mpi4py sources using gcc-15.
Note however that the executable fails to link:

$ mpicc_abi helloworld.c
/usr/bin/ld: /home/devel/mpi/openmpi/main/lib/libmpi_abi.so: undefined reference to `ompi_mpi_info_null'
/usr/bin/ld: /home/devel/mpi/openmpi/main/lib/libmpi_abi.so: undefined reference to `ompi_info_get'
/usr/bin/ld: /home/devel/mpi/openmpi/main/lib/libmpi_abi.so: undefined reference to `ompi_mpi_info_env'
/usr/bin/ld: /home/devel/mpi/openmpi/main/lib/libmpi_abi.so: undefined reference to `ompi_isendrecv'
/usr/bin/ld: /home/devel/mpi/openmpi/main/lib/libmpi_abi.so: undefined reference to `ompi_info_get_nthkey'
/usr/bin/ld: /home/devel/mpi/openmpi/main/lib/libmpi_abi.so: undefined reference to `ompi_info_memkind_copy_or_set'
/usr/bin/ld: /home/devel/mpi/openmpi/main/lib/libmpi_abi.so: undefined reference to `ompi_info_set'
/usr/bin/ld: /home/devel/mpi/openmpi/main/lib/libmpi_abi.so: undefined reference to `ompi_info_delete'
/usr/bin/ld: /home/devel/mpi/openmpi/main/lib/libmpi_abi.so: undefined reference to `ompi_sendrecv'
/usr/bin/ld: /home/devel/mpi/openmpi/main/lib/libmpi_abi.so: undefined reference to `ompi_info_get_bool'
/usr/bin/ld: /home/devel/mpi/openmpi/main/lib/libmpi_abi.so: undefined reference to `ompi_info_free'
/usr/bin/ld: /home/devel/mpi/openmpi/main/lib/libmpi_abi.so: undefined reference to `ompi_info_memkind_cb'
/usr/bin/ld: /home/devel/mpi/openmpi/main/lib/libmpi_abi.so: undefined reference to `ompi_info_memkind_process'
/usr/bin/ld: /home/devel/mpi/openmpi/main/lib/libmpi_abi.so: undefined reference to `ompi_info_dup'
/usr/bin/ld: /home/devel/mpi/openmpi/main/lib/libmpi_abi.so: undefined reference to `ompi_mpiinfo_init'
/usr/bin/ld: /home/devel/mpi/openmpi/main/lib/libmpi_abi.so: undefined reference to `ompi_info_get_nkeys'
/usr/bin/ld: /home/devel/mpi/openmpi/main/lib/libmpi_abi.so: undefined reference to `ompi_info_allocate'
/usr/bin/ld: /home/devel/mpi/openmpi/main/lib/libmpi_abi.so: undefined reference to `ompi_mpiinfo_init_env'
/usr/bin/ld: /home/devel/mpi/openmpi/main/lib/libmpi_abi.so: undefined reference to `ompi_info_get_valuelen'
collect2: error: ld returned 1 exit status

@dalcinl
Copy link
Contributor

dalcinl commented Sep 2, 2025

@hppritcha Actually, take a look at mpi-forum/mpi-abi-stubs#63

@dalcinl
Copy link
Contributor

dalcinl commented Sep 3, 2025

The installed mpi.h header is still broken, I had to fix it as per the patch in my previous comment.
Afterwards, my basic helloworld program compiles and links, but it fails to run:

$ mpicc_abi helloworld.c
$ mpiexec -n 1 ./a.out 
[optiplex:00000] *** An error occurred in MPI_Init_thread
[optiplex:00000] *** reported by process [3633774593,0]
[optiplex:00000] *** on a NULL communicator
[optiplex:00000] *** MPI_ERR_ARG: invalid argument of some other kind
[optiplex:00000] *** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
[optiplex:00000] ***    and MPI will try to terminate your MPI job as well)
--------------------------------------------------------------------------
prterun has exited due to process rank 0 with PID 0 on node optiplex calling
"abort". This may have caused other processes in the application to be
terminated by signals sent by prterun (as reported here).
--------------------------------------------------------------------------

@dalcinl
Copy link
Contributor

dalcinl commented Sep 3, 2025

@hppritcha The following definitions in the generated mpi.h file are wrong, they should be like ((MPI_Comm_copy_attr_function*) 0), that is, add the missing * to get a function POINTER type.
https://github.com/mpi-forum/mpi-abi-stubs/blob/main/mpi.h#L470

/* Predefined functions */
#define MPI_COMM_NULL_COPY_FN                 ((MPI_Comm_copy_attr_function) 0)
#define MPI_COMM_DUP_FN                       ((MPI_Comm_copy_attr_function) 1)
#define MPI_COMM_NULL_DELETE_FN               ((MPI_Comm_delete_attr_function) 0)
#define MPI_WIN_NULL_COPY_FN                  ((MPI_Win_copy_attr_function) 0)
#define MPI_WIN_DUP_FN                        ((MPI_Win_copy_attr_function) 1)
#define MPI_WIN_NULL_DELETE_FN                ((MPI_Win_delete_attr_function) 0)
#define MPI_TYPE_NULL_COPY_FN                 ((MPI_Type_copy_attr_function) 0)
#define MPI_TYPE_DUP_FN                       ((MPI_Type_copy_attr_function) 1)
#define MPI_TYPE_NULL_DELETE_FN               ((MPI_Type_delete_attr_function) 0)
#define MPI_CONVERSION_FN_NULL                ((MPI_Datarep_conversion_function) 0)
#define MPI_CONVERSION_FN_NULL_C              ((MPI_Datarep_conversion_function_c) 0)

/* Deprecated predefined functions */
#define MPI_NULL_COPY_FN                      ((MPI_Copy_function) 0)
#define MPI_DUP_FN                            ((MPI_Copy_function) 1)
#define MPI_NULL_DELETE_FN                    ((MPI_Delete_function) 0)

@dalcinl
Copy link
Contributor

dalcinl commented Sep 3, 2025

PS: You may have a similar problems in some MPI_T_XXX routines having arguments of type MPI_T_xxx_function, I believe you should add the * .


__opal_attribute_always_inline__ static inline int ompi_convert_abi_ts_level_intern_ts_level(int ts_level)
{
if (MPI_THREAD_SINGLE_ABI_INTERNAL == ts_level) {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Wouldn't a switch statement be better? It may ultimately be a matter of taste (optimizing compilers may generate similar binary code). Just a mild observation.

@dalcinl
Copy link
Contributor

dalcinl commented Sep 3, 2025

@hppritcha What's your plan for stuff introduced in MPI 4.1 and MPI 5.0? The generated mpi.h header says MPI_VERSION=5, however the MPI library libmpi_abi.so.0 seems to miss a bunch of stuff from MPI 4.1 and 5.0.

This is a list of what I managed to detect as missing from mpi4py's configuration machinery for missing MPI stuff.
Note there are a few that are quite old, like MPI_Aint_add/diff, MPI_Wtick, and MPI_Pcontrol.

MPI_Aint_add
MPI_Aint_diff
MPI_Type_get_value_index
MPI_Type_get_envelope
MPI_Type_get_contents
MPI_Type_get_envelope_c
MPI_Type_get_contents_c
MPI_Buffer_flush
MPI_Buffer_iflush
MPI_Comm_attach_buffer
MPI_Comm_detach_buffer
MPI_Comm_flush_buffer
MPI_Comm_iflush_buffer
MPI_Session_attach_buffer
MPI_Session_detach_buffer
MPI_Session_flush_buffer
MPI_Session_iflush_buffer
MPI_Comm_errhandler_fn
MPI_Comm_attach_buffer_c
MPI_Comm_detach_buffer_c
MPI_Session_attach_buffer_c
MPI_Session_detach_buffer_c
MPI_Remove_error_class
MPI_Remove_error_code
MPI_Remove_error_string
MPI_Abi_get_fortran_info
MPI_Get_hw_resource_info
MPI_Wtick
MPI_Pcontrol
MPI_Comm_toint
MPI_Errhandler_toint
MPI_File_toint
MPI_Group_toint
MPI_Info_toint
MPI_Message_toint
MPI_Op_toint
MPI_Request_toint
MPI_Session_toint
MPI_Type_toint
MPI_Win_toint
MPI_Comm_fromint
MPI_Errhandler_fromint
MPI_File_fromint
MPI_Group_fromint
MPI_Info_fromint
MPI_Message_fromint
MPI_Op_fromint
MPI_Request_fromint
MPI_Session_fromint
MPI_Type_fromint
MPI_Win_fromint

@dalcinl
Copy link
Contributor

dalcinl commented Sep 3, 2025

Current status regarding mpi4py:

  • I'm able to build mpi4py using export MPICC=mpicc_abi.
  • However, due to missing symbols, I'm not able to import the Python module.
  • I can switch via LD_LIBRARY_PATH to MPICH's libmpi_abi.so, but there are a few issues related to datatype handles and a few failing tests involving MPI callbacks (errhandlers and attributes). I need to investigate further.

@hppritcha
Copy link
Member Author

Thanks for checking this stuff out in its early state @dalcinl. some of the above functions have been implemented but are sitting in various states in PRs. some of these should be defined - like MPI_Wtick and the aint related functions so i'll see what's going on. As for the fromint/toint that seems like an excellent project for a AI LLM. I'll see what can be done there.

We still need to add some plumbing in the ompi internals for callbacks. that will come in as part of this PR.

The NERSC folks would really like ABI working for Doudna so to the extent there's a "plan" it would be nice to get this working sooner than later.

@dalcinl
Copy link
Contributor

dalcinl commented Nov 7, 2025

@hppritcha I'll report back once I have a chance to run tests again. In the meantime, please reconsider applying the following trivial patch.

diff --git a/ompi/mpi/bindings/ompi_bindings/c_type.py b/ompi/mpi/bindings/ompi_bindings/c_type.py
index e3414993cb..56ce3368fe 100644
--- a/ompi/mpi/bindings/ompi_bindings/c_type.py
+++ b/ompi/mpi/bindings/ompi_bindings/c_type.py
@@ -681,7 +681,7 @@ class TypeDatatypeArrayAsyncStandard(TypeDatatypeArrayStandard):
         code.append(f'if((MPI_SUCCESS == ret_value) && (MPI_REQUEST_NULL != {request_tmp_name}) && (!REQUEST_COMPLETE({request_tmp_name})))' + '{')
         code.append(f'ompi_coll_base_nbc_request_t* nb_request = (ompi_coll_base_nbc_request_t*){request_tmp_name};')
         code.append('assert(nb_request->data.release_arrays[idx] == NULL);')
-        code.append(f'nb_request->data.release_arrays[idx++] = (void *){self.tmpname};')
+        code.append(f'if (NULL != {self.tmpname}) nb_request->data.release_arrays[idx++] = (void *){self.tmpname};')
         code.append('nb_request->data.release_arrays[idx] = NULL;')
         code.append('} else {')
         code.append(f'if (NULL != {self.tmpname}) free({self.tmpname});')

@hppritcha
Copy link
Member Author

the arrays were't being cleaned up anyway in the existing code. some checks for predfined datatypes resulted in not running the callback that was freeing these arrays.

again for the data free code for alltoallw non-blocking variants

Signed-off-by: Howard Pritchard <howardp@lanl.gov>
@dalcinl
Copy link
Contributor

dalcinl commented Nov 9, 2025

@hppritcha another trivial fix ompi-info-array.patch

@dalcinl
Copy link
Contributor

dalcinl commented Nov 9, 2025

@hppritcha Things are working fine. I still have issues with Fortran datatypes when configuring with --disable-mpi-fortran. Under such configuration, all Fortran datatypes are missing, including those considered "non-optional" in the MPI standard.

IMHO, the distinction between "optional" and "non-optional" Fortran datatypes is quite irrelevant after the deprecation of mpif.h. I would rather define all Fortran datatypes unconditionally. If Fortran is not available, they could still be supported by default picking a default mapping C -> Fortran, e.g MPI_INT -> MPI_INTEGER. I should also recall that MPI 5.0 is supposed to allow a third-party Fortran binding to specify the Fortran ABI. IMHO, an MPI implementation should use a configure/build-time Fortran compiler to a) build their Fortran binding implementation, and b) provide a default ABI for Fortran datatypes. If Fortran is disabled to configure time, Fortran datatypes should still work at runtime provided that a third party specifies the Fortran ABI via MPI_Abi_set_fortran_info().

Long story short, this is my proposal: Open MPI should stop special-casing size-specific Fortran datatypes as optional, and treat all Fortran datatypes similarly, that is:

#define MPI_INTEGER OMPI_PREDEFINED_GLOBAL(MPI_Datatype, ompi_mpi_integer)
#define MPI_INTEGER1 OMPI_PREDEFINED_GLOBAL(MPI_Datatype, ompi_mpi_integer1)
#define MPI_INTEGER2 OMPI_PREDEFINED_GLOBAL(MPI_Datatype, ompi_mpi_integer2)
#define MPI_INTEGER4 OMPI_PREDEFINED_GLOBAL(MPI_Datatype, ompi_mpi_integer4)
#define MPI_INTEGER8 OMPI_PREDEFINED_GLOBAL(MPI_Datatype, ompi_mpi_integer8)
#define MPI_INTEGER16 OMPI_PREDEFINED_GLOBAL(MPI_Datatype, ompi_mpi_integer16)

and so on.

If a Fortran compiler is not configured-in, the runtime behavior of these Fortran datatypes has yet to be defined. The MPI standard mandates MPI_Type_size to return MPI_UNDEFINED, but there has been a lot controversy about it. Again, remember that the intention of MPI-5 is to allow a third-party Fortran binding to set the Fortran ABI.

@dalcinl
Copy link
Contributor

dalcinl commented Nov 9, 2025

@hppritcha This is not correct, MPI_Info is supposed to work before init.

*** The MPI_Info_toint() function was called before MPI_INIT was invoked.
*** This is disallowed by the MPI standard.
*** Your MPI job will now abort.

Other than that, everything seems to be OK. I've added Open MPI to my ABI CI on GitHub Actions.
This is a run of mpi4py tests with Open MPI. mpi4py itself is built with the MPI ABI stubs, and these same binaries are used to run with MPICH's MPI std ABI implementation.
https://github.com/mpi4py/mpi4py-testing/actions/runs/19205824355/job/54900848120

@hppritcha
Copy link
Member Author

Hmmm. I don't mind tweaking MPI_Info_toint to be callable without an MPI_Init or MPI_Session_init but where is it stated in the standard 5.1+ doc (i build at 40e9f52f) that this is the case? Such functions should be listed in 11.1.

@hppritcha
Copy link
Member Author

Thanks for the thoughts with respect to fortran datatypes. But definitely something for a different PR - perhaps as a series of changes to the fortran bindings support leading to fortran ABI support. We don't plan to include fortran ABI support in this PR - its already big enough.

@dalcinl
Copy link
Contributor

dalcinl commented Nov 10, 2025

Hmmm. I don't mind tweaking MPI_Info_toint to be callable without an MPI_Init or MPI_Session_init

Please do. Both MPI_Info_toint/MPI_Info_fromint should be available, exactly like MPI_Info_f2c/c2f are, and all of the other MPI_Info APIs. IOW, if you are able to create/manipulate/destroy Info objects, then you should also be allowed to convert handle representations.

but where is it stated in the standard 5.1+ doc (i build at 40e9f52f) that this is the case? Such functions should be listed in 11.1.

IMHO, it is obvious omission since the original MPI ABI proposal and these two APIs should be added to table 11.1 as an errata.

EDIT: mpi-forum/mpi-issues#1095

PS: There are even more things that are probably missing from that table, for example MPI_Session_create_errhandler.

Signed-off-by: Howard Pritchard <howardp@lanl.gov>
prior to MPI_Init or MPI_Session_init

Signed-off-by: Howard Pritchard <howardp@lanl.gov>
@dalcinl
Copy link
Contributor

dalcinl commented Nov 10, 2025

@hppritcha Success! The mpi4py build is ABI-compatible with MPICH's and OMPI's libmpi_abi.so.0:
https://github.com/mpi4py/mpi4py-testing/actions/runs/19205824355/job/55001607207

As mpi4py tests almost every API from MPI 5.0, I think we are in very good shape. Nonetheless, I'm planning on testing a few extra things, mostly related to callbacks, to make sure everything is fine.

Additionally, I plan to review a few things here and there. I still don't like STATUS_OUT having to copy the status on input, many times the input status for STATUS_OUT is simply uninitialized memory (I had to add a fix for that in mpi4py to silence valgrind). Maybe some day we could get the MPI Forum to relax the restrictions of not writing on status->MPI_ERROR and leave it up to the implementation.

PS: I'm about to leave the full mpi4py test suite running overnight under valgrind and report back the outcome tomorrow.

@hppritcha
Copy link
Member Author

excellent. most of the callbacks are going to cause problems (except hopefully attr copy/del). I'm adding support for errhandlers this week.

Signed-off-by: Howard Pritchard <howardp@lanl.gov>
Signed-off-by: Howard Pritchard <howardp@lanl.gov>
Signed-off-by: Howard Pritchard <howardp@lanl.gov>
Signed-off-by: Howard Pritchard <howardp@lanl.gov>
add a blurb about how current and age and revision
should work for this library.

Signed-off-by: Howard Pritchard <howardp@lanl.gov>
Signed-off-by: Howard Pritchard <howardp@lanl.gov>
using the bindings_extra_state arg to ompi_attr_create_keyval

Signed-off-by: Howard Pritchard <howardp@lanl.gov>
Signed-off-by: Howard Pritchard <howardp@lanl.gov>
Signed-off-by: Howard Pritchard <howardp@lanl.gov>
Signed-off-by: Howard Pritchard <howardp@lanl.gov>
and how to use it

Signed-off-by: Howard Pritchard <howardp@lanl.gov>
Comment on lines +9 to +11
Starting with MPI 5.0, the MPI standard specifies an ABI for the c and
Fortran MPI interfaces. In this release, Open MPI supports the c
part of the MPI ABI.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is not accurate. The Fortran interfaces are not part of the ABI.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do you mean the MPI_Abi_fortran* are not part of the ABI or that MPI fortran entry points in general are not part of the ABI? Section 20.4.2 The MPI ABI Fortran Modules and Shared Library states that there is a Fortran ABI.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

MPI fortran entry points in general are not part of the ABI?

This one.

Section 20.4.2 The MPI ABI Fortran Modules and Shared Library states that there is a Fortran ABI.

Maybe a leftover from @jeffhammond ?

@hppritcha What I'm trying to point out is that currently (and probably forever) it is not possible to define an interoperable ABI for Fortran. User cannot expect to compiler their Fortran code with MPICH's Fortran modules and libraries, and at runtime switch and run against Open MPI's implementation of Fortran stuff.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

somehow i figured the fortran in the mpi_f08 module would be generic enough that it would be interoperable. I guess i need to tweak my slides for the OMPI BOF.

Okay in this light I'll scrub the fortran wording in the README.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm really worried that the lack of a Fortran ABI will defeat the core interoperability purpose of the ABI to the point that we the new great stuff will not adopted widely.

There is only one possible solution for Fortran: Someone (Jeff) or the community (MPICH+OpenMPI) develop Fortran bindings that use the MPI C API exclusively, and then MPI implementaitons can vendor in their releases mostly unmodified. However, such Fortran bindings would be much easier to implement if MPI fixed a few shortcomings here and there to better support callbacks/trampolines. Such enhancements would benefit other language bindings, not only Fortran.

Comment on lines +18 to +20
The MPI 5.0 standard specifies the file name of the MPI ABI compliant
library - libmpi_abi. The major version of the library is 1 and minor
version is 0.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is confusing. How is Open MPI going to define the SONAME? I'm assume that is going to be libmpi_abi.so.0. As per the libtool rules, this is the first version ever of the library, so the .0 prefix.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What's the soname used by mpich? If they use a different soname that defeats the whole point of having a consistent ABI

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Right now it is libmpi_abi.so.0, but I do not know if that is because the ABI in MPICH is in a "development" stage, I hope not.

Again, by the Linux/libtool conventions, the SONAME should be libmpi_abi.so.0, and hopefully stay that way forever, unless the MPI Forum takes the unwise route of removing/changing interfaces.

I elaborated in the matter long ago, but I'm not sure people really understood my point.
mpi-forum/mpi-abi-stubs#28

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yes I was going to check that (the MPICH libmpich_abi soname) as I had the same concern. Right now we are using a c/r/a that ends up generating a libmpi_abi with an soname of libmpi_abi.so.0.

I wrote a blurb in the VERSION file (where we set these c,r,a for each of our Open MPI so's):

# The current and age values of the libmpi_abi shared library are defined by the
# specification of the ABI for the version of the MPI standard supported
# by this Open MPI release.  One would hope that these always move in
# sync.  We assume we have flexibility with the revision number as
# bugs are fixed, etc.

I guess it should say are implicitly defined. I was thinking that as we have to patch our ABI related code we (Open MPI project) would increment the revision number.

I was thinking that if we (MPI Forum) add interfaces to the ABI we (Open MPI project) would crank up the current and age such that the C-A remains 0. And as @dalcinl notes we (MPI forum) would hopefully not do something that would case us to have to increment C and reset A to zero!

I'll check out mpi-forum/mpi-abi-stubs#28 - it looks relevant.

In any case I'll tweak the comment in the VERSION file.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I guess it should say are implicitly defined.

Sounds good.

I'll check out mpi-forum/mpi-abi-stubs#28 - it looks relevant.

My initial proposal there is outdated, I originally proposed to start with current=1 age=0 to be in sync with MPI_ABI_VERSION. But that's not the usual recommendation, the very first release of a shared library should have current=0 age=0. So let it be that way. It will hopefully never change, and we will all enjoy libmpi_abi.so.0 for ever after.

Comment on lines +29 to +31
This release does not support the Fortran ABI so there is no ``mpifort_abi``
compiler wrapper. This mixed c/Fortran MPI apps cannot make use of the
MPI ABI library with this release.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe rephrase this part as per my previous comment about no Fortran ABI?

In principle a single mpi.h include file can be used, for example
the one provided at https://github.com/mpi-forum/mpi-abi-stubs.

Chapter 21 specifies the shared library name (soname) of the ABI MPI
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
Chapter 21 specifies the shared library name (soname) of the ABI MPI
Chapter 20 specifies the shared library name (soname) of the ABI MPI

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

okay but in the current latex its moved to 21. oh i think that's because there's a process fault tolerance chapter now . Maybe better not to have explicit chapter number references.

in the VERSION file as this is the file updated
by the OMPI release managers.

Signed-off-by: Howard Pritchard <howardp@lanl.gov>
@rhc54
Copy link
Contributor

rhc54 commented Nov 18, 2025

(MPI forum) would hopefully not do something that would case us to have to increment C and reset A to zero

Hasn't that happened a number of times in recent years with removal of APIs? C++ bindings, MPI-1 stuff, etc. 🤷‍♂️ Not sure why you guys consider it so horrible to have different ABI versions - it's pretty common for projects specifying an ABI. You shift between implementations using libraries that support the same ABI version. No big deal. Forum just has to specify what is in each ABI version.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

10 participants