[BUG] Examples using older MONAI Bundles not compatible with MONAI v1.5.1 (which reveals errors in older MONAI bundles)

[MMelQin]

Describe the bug

TL,DR:
After a bit more investigation, it has been determined that the older MONAI segmentation models/bundles trained with flipped DICOM CT images from the Medical Decathlon dataset incorrectly re-orient image to "RAS", and this issue finally shows after the orientation transform is improved in monai v1.5.1 to recognize correctly the DICOM image orientation. Details are in the comment below.

A compatibility issue has been observed after monai v1.5.1 has been released and the said version is installed in the Python environment for running the example apps.

MD App SDK itself does not has a direct package dependency on monai, however monai is required for a majority of example apps. The update v1.5.1 was released on 09/22 to enhance the support of the latest NVIDIA GPUs and to support torch 2.7 and 2.8 as as such latest CUDA runtime 12.

With high hope that this monai update will allow App SDK to be compatible with the latest Holoscan SDK which requires CUDA runtime >=12.6, test revealed that example applications execute without errors, but the results, e.g. segmentation and the STL generated from it, are not correct.

Steps/Code to reproduce bug

• Clone this repo and CD to the repo root
• Pick an example app, e.g. examples/apps/ai_spleen_seg_app
• Create and activate venv
  python3 -m venv .testenv && source .testenv/bin/activate
• Install monai-deploy-app-sdk (not using the dev mode)
  pip install monai-deploy-app-sdk
• Install dependencies in the requirements.txt as is
  pip install -r examples/apps/ai_spleen_seg_app/requirements.txt
• Note the monai version, pip list | grep monai
• Stage the model and input DICOM files, and set env vars, e.g.

export HOLOSCAN_INPUT_PATH="inputs/spleen_ct_tcia"
export HOLOSCAN_MODEL_PATH="models/spleen_ct"
export HOLOSCAN_OUTPUT_PATH="output_spleen"

• Run the app
  python /home/mqin/src/md-app-sdk/examples/apps/ai_spleen_seg_app
• Once the app completes successfully, look for the spleen.stl file in the stl folder
  ls $HOLOSCAN_OUTPUT_PATH
• Render the stl and see the broken surface.

Expected behavior
The latest monai v1.5.1 also introduced some changes in image and meta tensor transforms, the App SDK's built-in opertors will need to be reviewed, updated, and tested to be working with the latest monai. Given the breaking changes, the App SDK will also need to pin monai>=1.5.1, but for now, <=1.5.0.

Environment details (please complete the following information)

• OS/Platform: Ubuntu 22.04 LTS x86_64
• Python Version: 3.10
• Method of MONAI Deploy App SDK install: pip
• SDK Version: 3.5.1

Additional context
Workaround is for the example apps to pin monai<=1.5.0

[ericspod]

Hi @MMelQin I've added this to our project for MONAI 1.6, however we can potentially hotfix sooner. If we can identify what is changed with 1.5.1 and determine if these are bugs or compatibility breaks we can address them.

[MMelQin]

Hi @MMelQin I've added this to our project for MONAI 1.6, however we can potentially hotfix sooner. If we can identify what is changed with 1.5.1 and determine if these are bugs or compatibility breaks we can address them.

Thanks @ericspod . I think it is most likely due to the change in requiring setting the labels of Orientation transform, which I did not catch before to test and update the App SDK code. Raising the issue here is more informational before I update the code and release a new SDK version.

@ericspod I have to say again, years after first complaining to the model trainers, that the Medical Decathlon dataset in NIfTI mis-represented the patient anatomical orientation, in that what should have been LPS was marked as RAS, causing the early MONAI models (e.g. Spleen Seg and Pancreas Seg etc) being trained on flipped image and would fail when DICOM series is correctly loaded and the space is set to LPS.

UPDATE
I have since spent more time on this issue, and think I got to the bottom of it

• Filed Issue 766 requesting to stop using the flipped image as input in the inference config. This use in fact caused confusion before and some workaround in App SDK (see below). The rest of the inference config is in fact good and works with properly converted NIfTI from production DICOM series from TCIA.
• I have discovered that a long while ago, App SDK implemented DICOM series to volume image operator and had to calculate/transform the affine in RAS+, to fit the then MONAI code, largely because that setting space to LPS did not work. However, I forgot to flip space from "LPS" to "RAS" in a separate in-mem ImageReader class (as it had no effect anyway). Not a issue before monai 1.5.1, but now the space metadata is parsed, this missed change of course would cause unnecessary image rotation/flipping and as a result the bad segmentation. I've move the space metadata setting to be in the same class of array affine calculation to ensure settings are consistent.

Thanks.
@Nic-Ma @daguangxu @rahul-imaging @mocsharp @dongyang0122 @bluna301 @vikashg