Codecov Results Discrepancy

[KumoLiu]

Describe the bug
The coverage data is inaccurate, as all tests have been executed but the results are not being properly included in the coverage report.

Randomly select one test log:

[2025-04-07T13:56:11.741Z] Starting test: test_task_init (tests.handlers.test_handler_clearml_image.TestHandlerClearMLImageHandler)...
[2025-04-07T13:56:11.741Z] .Finished test: test_task_init (tests.handlers.test_handler_clearml_image.TestHandlerClearMLImageHandler) (0.0435s)

....
[2025-04-07T14:51:03.366Z] ----------------------------------------------------------------------
[2025-04-07T14:51:03.366Z] Ran 16251 tests in 3882.484s
[2025-04-07T14:51:03.366Z] 
[2025-04-07T14:51:03.366Z] OK (skipped=388)

Result reported:
https://app.codecov.io/gh/Project-MONAI/MONAI/blob/dev/tests%2Fhandlers%2Ftest_handler_clearml_image.py

[garciadias]

Hi @KumoLiu, @ericspod.

To understand what is happening, I have identified which workflows have code that interacts with codecov.

Different parameters to runtests.sh

Here you have a table of the parameters used to run the ./runtests.sh file:

Run tags	cron						pythonapp-gpu		setupapp
	cron-gpu		cron-pt-image		cron-docker		GPU-quick-py3		coverage-py3		test-py3x
Step	1	2	1	2	1	2	1	2	1	2	1	2
build	x	x	x	x	x	x	x		x	x	x	x
coverage	x	x	x	x	x	x			x	x
disttests	x		x		x		x		x
net		x		x		x				x
min												x
quick								x			x	x
pytype					x
unittests	x		x		x			x	x		x
BUILD_MONA	1	1	1	1	1	1	1	None	1	1	0	1
TORCH_CUDA_ARCH_LIST	None	None	None	None	None	None	"7.0"	None	None	None	None	None

Generating coverage report

All jobs run coverage xml-- ignore-errors after the second step only, which makes me more sceptical about the overwriting theory. The only overwriting that could occur is when sending sequential coverages to codecov with different values. However, in that case, I assume both would appear in codecov, and we would see some spikes in the plot.

Pushing the report to codecov

All jobs use the same 'vanilla' command to send the data to codecov, except for GPU-quick-py3.

The 'vanilla' command is:

  - name: Upload coverage
    uses: codecov/codecov-action@v5
    with:
      fail_ci_if_error: false
      files: ./coverage.xml

While in GPU-quick-py3 we have:

• workflow: pythonapp-gpu.yml
  • job: GPU-quick-py3

            BUILD_MONAI=1 TORCH_CUDA_ARCH_LIST="7.0" ./runtests.sh --build --disttests
            ./runtests.sh --quick --unittests
            if [ ${{ matrix.environment }} = "PT230+CUDA124" ]; then
              # test the clang-format tool downloading once
              coverage run -m tests.clang_format_utils
            fi
            coverage xml --ignore-errors
            if pgrep python; then pkill python; fi
          shell: bash
        - name: Upload coverage
          if: ${{ github.head_ref != 'dev' && github.event.pull_request.merged != x }}
          uses: codecov/codecov-action@v5
          with:
            files: ./coverage.xml

The difference is the inclusion of this line:
if: ${{ github.head_ref != 'dev' && github.event.pull_request.merged != x }}

I think this means that only merges to dev result in a coverage report being pushed.
If these are the only coverage reports we are seeing in codecove, the running of the tests with the parameters build, disttest, quick, and unittests could be the cause of the low coverage, since the net is never running.

Running locally

Trying to run all tests locally, I get 85.94% coverage with 79 errors and 466 tests skipped.
Which makes me think that the actual coverage should be over 90%.

Conclusion

I am still not sure what is causing the issue, but if you can look at Blossom and see which workflow is actually sending the coverage report, I can dig into the runtests.sh with those exact parameters.