Skip to content

Conversation

Revolyssup
Copy link
Contributor

@Revolyssup Revolyssup commented Jul 25, 2025

The test here https://github.com/apache/apisix/actions/runs/16511586539/job/46694453675
failed due to invalid configuration but in case of Bailout, the tests is considered successful. This change asserts and finds for the bailed out test and fails CI if any test was bailed out.

The following tests fail after the changes in script in this pr https://github.com/apache/apisix/actions/runs/16519080359/job/46716551717
https://github.com/apache/apisix/actions/runs/16519816286/job/46718888226?pr=12462#step:16:236

I have modified the above tests to finally pass.

Further enquiries

The doc says
"By default the test scaffold treats NGINX server startup failures as fatal errors in running the tests. The must_die section, however, turns such a failure into a normal test checkup."

If you have just one test then it holds true. But when even one test passes and subsequent test has invalid config then TEST IS CONSIDERED PASS.

WHen invalid configuration was passed in Test 1 (As expected the Bailout is considered total test failure)

Test run interrupted!
Test Summary Report
-------------------
t/core/ctx.t (Wstat: 65280 (exited 255) Tests: 0 Failed: 0)
  Non-zero exit status: 255
Files=1, Tests=0, 31 wallclock secs ( 0.01 usr  0.00 sys +  0.19 cusr  0.17 csys =  0.37 CPU)
Result: FAIL

When invalid configuration was passed in Test 2(or any other test after one passed test): (The test pass even after bailout)

Bailout called.  Further testing stopped:  t/core/ctx.t TEST 2: http header - Cannot start nginx using command "/usr/bin/openresty -p /home/ashish/dev/apisix/t/servroot/ -c /home/ashish/dev/apisix/t/servroot/conf/nginx.conf > /dev/null" (status code 256).
t/core/ctx.t .. ok
Test run interrupted!
All tests successful.
Files=1, Tests=6, 31 wallclock secs ( 0.02 usr  0.01 sys +  0.27 cusr  0.22 csys =  0.52 CPU)
Result: PASS
All tests successful.
Files=1, Tests=6, 31 wallclock secs ( 0.02 usr  0.01 sys +  0.27 cusr  0.22 csys =  0.52 CPU)
Result: PASS
FAILED--Further testing stopped: t/core/ctx.t TEST 2: http header - Cannot start nginx using command "/usr/bin/openresty -p /home/ashish/dev/apisix/t/servroot/ -c /home/ashish/dev/apisix/t/servroot/conf/nginx.conf > /dev/null" (status code 256).

Checklist

  • I have explained the need for this PR and the problem it solves
  • I have explained the changes or the new features added to this PR
  • I have added tests corresponding to this change
  • I have updated the documentation to reflect this change
  • I have verified that this change is backward compatible (If not, please discuss on the APISIX mailing list first)

@dosubot dosubot bot added size:S This PR changes 10-29 lines, ignoring generated files. bug Something isn't working CI labels Jul 25, 2025
@dosubot dosubot bot added size:M This PR changes 30-99 lines, ignoring generated files. and removed size:S This PR changes 10-29 lines, ignoring generated files. labels Jul 25, 2025
"time_window": 60
}
]
"ai-rate-limiting": {"instances": [{"name": "openai-gpt3","limit": 50,"time_window": 60},{"name": "openai-gpt4","limit": 20,"time_window": 60}]
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@moonming moonming merged commit 53de201 into apache:master Jul 28, 2025
23 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working CI size:M This PR changes 30-99 lines, ignoring generated files.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants