feat: add automatic postprocessing of integratedTestsLogs. #3509

CusiniM · 2025-01-08T09:06:02Z

This PR adds a slightly modified version of the processIntegratedTestsFailures script as a step of the CI integratedTests job. I have slightly modified the script and moved it to the geosats python package (pythonPackages PR GEOS-DEV/geosPythonPackages#53) since I feel that's where it belongs.

The results of the processing scripts are written to a txt file that it's packaged along with the logs.

CusiniM · 2025-01-10T22:54:12Z

This is an example of the processedTestsLogs.txt in case numerical diffs are found:

INFO: Total number of log files processed: 1537

WARNING: Found unfiltered diff in: /Users/cusini1/Downloads/integratedTests/TestResults/test_data/wavePropagation/elas3D_DAS_smoke_08/1497.python3_elas3D_DAS_smoke_08_2_restartcheck_.log
INFO: Details of diffs:   ********************************************************************************

  Error: /Problem/Solvers/elasticSolver/dasSignalNp1AtReceivers

  	Arrays of types float32 and float32 have 402 values of which 200 fail both the relative and absolute tests.

  		Max absolute difference is at index (np.int64(200), np.int64(1)): value = 0.2, base_value = 0.0

  		Max relative difference is at index (np.int64(200), np.int64(1)): value = 0.2, base_value = 0.0

  	Statistics of the q values greater than 1.0 defined by absolute tolerance: N = 200

  		max = 2000.0001, mean = 500.0, std = 646.787

  ********************************************************************************

  Error: /Problem/Solvers/elasticSolver/dasSignalNp1AtReceivers

  	Arrays of types float32 and float32 have 402 values of which 200 fail both the relative and absolute tests.

  		Max absolute difference is at index (np.int64(200), np.int64(1)): value = 0.2, base_value = 0.0

  		Max relative difference is at index (np.int64(200), np.int64(1)): value = 0.2, base_value = 0.0

  	Statistics of the q values greater than 1.0 defined by absolute tolerance: N = 200

  		max = 2000.0001, mean = 500.0, std = 646.787

  ********************************************************************************

  Error: /Problem/Solvers/elasticSolver/dasSignalNp1AtReceivers

  	Arrays of types float32 and float32 have 402 values of which 200 fail both the relative and absolute tests.

  		Max absolute difference is at index (np.int64(200), np.int64(1)): value = 0.2, base_value = 0.0

  		Max relative difference is at index (np.int64(200), np.int64(1)): value = 0.2, base_value = 0.0

  	Statistics of the q values greater than 1.0 defined by absolute tolerance: N = 200

  		max = 2000.0001, mean = 500.0, std = 646.787

  ********************************************************************************

  Error: /Problem/Solvers/elasticSolver/dasSignalNp1AtReceivers

  	Arrays of types float32 and float32 have 402 values of which 200 fail both the relative and absolute tests.

  		Max absolute difference is at index (np.int64(200), np.int64(1)): value = 0.2, base_value = 0.0

  		Max relative difference is at index (np.int64(200), np.int64(1)): value = 0.2, base_value = 0.0

  	Statistics of the q values greater than 1.0 defined by absolute tolerance: N = 200

  		max = 2000.0001, mean = 500.0, std = 646.787

  ********************************************************************************

  Error: /Problem/Solvers/elasticSolver/dasSignalNp1AtReceivers

  	Arrays of types float32 and float32 have 402 values of which 200 fail both the relative and absolute tests.

  		Max absolute difference is at index (np.int64(200), np.int64(1)): value = 0.2, base_value = 0.0

  		Max relative difference is at index (np.int64(200), np.int64(1)): value = 0.2, base_value = 0.0

  	Statistics of the q values greater than 1.0 defined by absolute tolerance: N = 200

  		max = 2000.0001, mean = 500.0, std = 646.787

  ********************************************************************************

  Error: /Problem/Solvers/elasticSolver/dasSignalNp1AtReceivers

  	Arrays of types float32 and float32 have 402 values of which 200 fail both the relative and absolute tests.

  		Max absolute difference is at index (np.int64(200), np.int64(1)): value = 0.2, base_value = 0.0

  		Max relative difference is at index (np.int64(200), np.int64(1)): value = 0.2, base_value = 0.0

  	Statistics of the q values greater than 1.0 defined by absolute tolerance: N = 200

  		max = 2000.0001, mean = 500.0, std = 646.787

  ********************************************************************************

  Error: /Problem/Solvers/elasticSolver/dasSignalNp1AtReceivers

  	Arrays of types float32 and float32 have 402 values of which 200 fail both the relative and absolute tests.

  		Max absolute difference is at index (np.int64(200), np.int64(1)): value = 0.2, base_value = 0.0

  		Max relative difference is at index (np.int64(200), np.int64(1)): value = 0.2, base_value = 0.0

  	Statistics of the q values greater than 1.0 defined by absolute tolerance: N = 200

  		max = 2000.0001, mean = 500.0, std = 646.787

CusiniM · 2025-01-10T23:12:16Z

And this is the ouput in case all diffs are non numerical:

INFO: Total number of log files processed: 1397

INFO: No unfiltered differences were found.

INFO: The following file(s) had at least one error block that was excluded:
0030.python3_co2_hybrid_1d_02_2_restartcheck_.log
0068.python3_deadoil_3ph_corey_1d_fractured_03_2_restartcheck_.log
0078.python3_deadoil_3ph_baker_1d_03_2_restartcheck_.log
0034.python3_co2_hybrid_1d_03_2_restartcheck_.log
0122.python3_co2_flux_3d_27_2_restartcheck_.log
0074.python3_deadoil_3ph_baker_1d_02_2_restartcheck_.log
0038.python3_2ph_cap_1d_ihu_01_2_restartcheck_.log
0006.python3_4comp_2ph_1d_02_2_restartcheck_.log
0010.python3_4comp_2ph_1d_03_2_restartcheck_.log
0090.python3_deadoil_3ph_staircase_hybrid_3d_01_2_restartcheck_.log
0094.python3_deadoil_3ph_staircase_hybrid_3d_08_2_restartcheck_.log
0018.python3_4comp_2ph_cap_1d_02_2_restartcheck_.log
0062.python3_deadoil_3ph_stone2_1d_03_2_restartcheck_.log
0046.python3_deadoil_3ph_corey_1d_01_2_restartcheck_.log
0022.python3_4comp_2ph_cap_1d_03_2_restartcheck_.log
0166.python3_grav_seg_c1ppu_hyst_01_2_restartcheck_.log
0170.python3_grav_seg_c1ppu_hyst_02_2_restartcheck_.log
0026.python3_co2_hybrid_1d_01_2_restartcheck_.log
0070.python3_deadoil_3ph_baker_1d_01_2_restartcheck_.log
0066.python3_deadoil_3ph_corey_1d_fractured_01_2_restartcheck_.log
0134.python3_deadOilSpe10Layers84_85_smoke_2d_01_2_restartcheck_.log
0138.python3_deadOilSpe10Layers84_85_smoke_2d_04_2_restartcheck_.log
0150.python3_isothermalLeakyWell_smoke_3d_01_2_restartcheck_.log
0154.python3_isothermalLeakyWell_smoke_3d_04_2_restartcheck_.log
0162.python3_thermalLeakyWell_smoke_3d_04_2_restartcheck_.log
0158.python3_thermalLeakyWell_smoke_3d_01_2_restartcheck_.log
0142.python3_buckleyLeverett_smoke_01_2_restartcheck_.log
0146.python3_buckleyLeverett_smoke_02_2_restartcheck_.log
0130.python3_co2_flux_dirichlet_06_2_restartcheck_.log
0126.python3_co2_flux_dirichlet_01_2_restartcheck_.log
0002.python3_4comp_2ph_1d_01_2_restartcheck_.log
0042.python3_2ph_cap_1d_ihu_03_2_restartcheck_.log
0114.python3_co2_flux_3d_01_2_restartcheck_.log
0118.python3_co2_flux_3d_08_2_restartcheck_.log
0054.python3_deadoil_3ph_corey_1d_03_2_restartcheck_.log
0014.python3_4comp_2ph_cap_1d_01_2_restartcheck_.log
0058.python3_deadoil_3ph_stone2_1d_01_2_restartcheck_.log
0086.python3_deadoil_3ph_staircase_3d_08_2_restartcheck_.log
0082.python3_deadoil_3ph_staircase_3d_01_2_restartcheck_.log
0110.python3_deadoil_2ph_staircase_gravity_segregation_3d_08_2_restartcheck_.log
0106.python3_deadoil_2ph_staircase_gravity_segregation_3d_01_2_restartcheck_.log
0050.python3_deadoil_3ph_corey_1d_02_2_restartcheck_.log
1285.python3_ThermoPoroElastic_staircase_co2_smoke_01_2_restartcheck_.log
1289.python3_ThermoPoroElastic_staircase_co2_smoke_04_2_restartcheck_.log
0502.python3_deadOil_fractureMatrixFlow_edfm_inclinedFrac_smoke_01_2_restartcheck_.log
0500.python3_deadOil_fractureMatrixFlow_edfm_horizontalFrac_smoke_04_2_restartcheck_.log
0498.python3_deadOil_fractureMatrixFlow_edfm_horizontalFrac_smoke_02_2_restartcheck_.log
0504.python3_deadOil_fractureMatrixFlow_edfm_inclinedFrac_smoke_04_2_restartcheck_.log
0496.python3_deadOil_fractureMatrixFlow_edfm_horizontalFrac_smoke_01_2_restartcheck_.log
1245.python3_co2_thermal_2d_01_2_restartcheck_.log
1249.python3_co2_thermal_2d_04_2_restartcheck_.log
0594.python3_PoroElastic_deadoil_3ph_baker_2d_fim_01_2_restartcheck_.log
0600.python3_PoroElastic_deadoil_3ph_baker_2d_sequential_04_2_restartcheck_.log
0686.python3_PoroElastic_hybridHexPrism_co2_sequential_3d_04_2_restartcheck_.log
0662.python3_PoroElastic_staircase_co2_3d_fim_04_2_restartcheck_.log
0666.python3_PoroElastic_staircase_co2_3d_sequential_01_2_restartcheck_.log
0678.python3_PoroElastic_hybridHexPrism_co2_fim_3d_04_2_restartcheck_.log
0598.python3_PoroElastic_deadoil_3ph_baker_2d_sequential_01_2_restartcheck_.log
0596.python3_PoroElastic_deadoil_3ph_baker_2d_fim_04_2_restartcheck_.log
0658.python3_PoroElastic_staircase_co2_3d_fim_01_2_restartcheck_.log
0682.python3_PoroElastic_hybridHexPrism_co2_sequential_3d_01_2_restartcheck_.log
0674.python3_PoroElastic_hybridHexPrism_co2_fim_3d_01_2_restartcheck_.log
0670.python3_PoroElastic_staircase_co2_3d_sequential_04_2_restartcheck_.log
0222.python3_staircase_co2_wells_3d_01_2_restartcheck_.log
0226.python3_staircase_co2_wells_3d_08_2_restartcheck_.log
0190.python3_dead_oil_wells_2d_01_2_restartcheck_.log
0238.python3_black_oil_wells_saturated_3d_stone2_04_2_restartcheck_.log
0218.python3_black_oil_wells_unsaturated_3d_04_2_restartcheck_.log
0210.python3_black_oil_wells_saturated_3d_04_2_restartcheck_.log
0186.python3_compositional_multiphase_wells_2d_04_2_restartcheck_.log
0198.python3_dead_oil_wells_hybrid_2d_01_2_restartcheck_.log
0178.python3_compositional_multiphase_wells_1d_02_2_restartcheck_.log
0214.python3_black_oil_wells_unsaturated_3d_01_2_restartcheck_.log
0250.python3_deadOilEgg_smoke_3d_01_2_restartcheck_.log
0246.python3_class09_pb3_smoke_3d_08_2_restartcheck_.log
0242.python3_class09_pb3_smoke_3d_01_2_restartcheck_.log
0234.python3_black_oil_wells_saturated_3d_stone2_01_2_restartcheck_.log
0194.python3_dead_oil_wells_2d_04_2_restartcheck_.log
0232.python3_staircase_co2_wells_hybrid_3d_08_2_restartcheck_.log
0230.python3_staircase_co2_wells_hybrid_3d_01_2_restartcheck_.log
0202.python3_dead_oil_wells_hybrid_2d_04_2_restartcheck_.log
0182.python3_compositional_multiphase_wells_2d_01_2_restartcheck_.log
0206.python3_black_oil_wells_saturated_3d_01_2_restartcheck_.log
0174.python3_compositional_multiphase_wells_1d_01_2_restartcheck_.log

…om:GEOS-DEV/GEOS into feat/cusini/autoMaticProcessingOfTestsLogs

codecov · 2025-01-12T14:32:49Z

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 56.79%. Comparing base (907fb3f) to head (8178c36).
Report is 72 commits behind head on develop.

Additional details and impacted files

@@           Coverage Diff            @@
##           develop    #3509   +/-   ##
========================================
  Coverage    56.79%   56.79%           
========================================
  Files         1154     1154           
  Lines       100074   100074           
========================================
  Hits         56836    56836           
  Misses       43238    43238

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

rrsettgast · 2025-01-13T17:33:34Z

@CusiniM "excluded" == "filtered"?

CusiniM · 2025-01-13T17:35:02Z

@CusiniM "excluded" == "filtered"?

yes.

rrsettgast · 2025-01-13T17:37:00Z

@CusiniM "excluded" == "filtered"?

yes.

Maybe:
INFO: The following file(s) had at least one error block that was excluded:
should be changed to:
INFO: The following file(s) had at least one error block that was filtered:

just to avoid confusion? And then adding the nature of the filter? Like showing the filter string that led to the filtering of the diff?

feat: add automatic postprocessing of integratedTestsLogs.

01f23ad

CusiniM self-assigned this Jan 8, 2025

CusiniM added the ci: run integrated tests Allows to run the integrated tests in GEOS CI label Jan 8, 2025

change pythonPackages branch.

52c8c49

CusiniM requested review from cssherman and paveltomin January 8, 2025 10:52

CusiniM mentioned this pull request Jan 10, 2025

feat: Add script to process IntegratedTests failures to geos_ats. GEOS-DEV/geosPythonPackages#53

Merged

TEST: add new wrapper.

8b4b044

rrsettgast approved these changes Jan 10, 2025

View reviewed changes

CusiniM marked this pull request as ready for review January 10, 2025 22:54

CusiniM requested a review from ryar9534 as a code owner January 10, 2025 22:54

Merge branch 'develop' into feat/cusini/autoMaticProcessingOfTestsLogs

bdc04c6

paveltomin approved these changes Jan 10, 2025

View reviewed changes

CusiniM added 4 commits January 10, 2025 15:18

fix compilation error.

1506ca6

Merge branch 'feat/cusini/autoMaticProcessingOfTestsLogs' of github.c…

9fa9f9a

…om:GEOS-DEV/GEOS into feat/cusini/autoMaticProcessingOfTestsLogs

Move the location of the script back.

cd4f923

revert unwanted changes.

e55a3e5

CusiniM added the ci: run CUDA builds Allows to triggers (costly) CUDA jobs label Jan 11, 2025

revert change.

8178c36

CusiniM added the ci: run code coverage enables running of the code coverage CI jobs label Jan 12, 2025

CusiniM merged commit fd36d2a into develop Jan 12, 2025
25 checks passed

CusiniM deleted the feat/cusini/autoMaticProcessingOfTestsLogs branch January 12, 2025 18:08

danielemoretto44 pushed a commit that referenced this pull request Sep 22, 2025

feat: add automatic postprocessing of integratedTestsLogs. (#3509)

6d8f9f9

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

feat: add automatic postprocessing of integratedTestsLogs. #3509

feat: add automatic postprocessing of integratedTestsLogs. #3509

Uh oh!

CusiniM commented Jan 8, 2025 •

edited

Loading

Uh oh!

CusiniM commented Jan 10, 2025

Uh oh!

CusiniM commented Jan 10, 2025 •

edited

Loading

Uh oh!

codecov bot commented Jan 12, 2025 •

edited

Loading

Uh oh!

Uh oh!

rrsettgast commented Jan 13, 2025

Uh oh!

CusiniM commented Jan 13, 2025

Uh oh!

rrsettgast commented Jan 13, 2025 •

edited

Loading

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

feat: add automatic postprocessing of integratedTestsLogs. #3509

feat: add automatic postprocessing of integratedTestsLogs. #3509

Uh oh!

Conversation

CusiniM commented Jan 8, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

CusiniM commented Jan 10, 2025

Uh oh!

CusiniM commented Jan 10, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

codecov bot commented Jan 12, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

Uh oh!

rrsettgast commented Jan 13, 2025

Uh oh!

CusiniM commented Jan 13, 2025

Uh oh!

rrsettgast commented Jan 13, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

CusiniM commented Jan 8, 2025 •

edited

Loading

CusiniM commented Jan 10, 2025 •

edited

Loading

codecov bot commented Jan 12, 2025 •

edited

Loading

rrsettgast commented Jan 13, 2025 •

edited

Loading