[flink/ci] Add watchdog for flink timeout failure stack trace at 95 percent time completion #1280
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
@wuchong, I have tried to implement it in the most basic way to get started and kept it specifically for Flink tests. Please review whenever you get a chance. Thank you! |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Purpose
Linked issue: close #1031
Add watchdog monitoring for Flink tests to capture debugging information when CI builds timeout, helping identify root causes of stuck test processes.
Brief change log
Added tools/ci/flink_watchdog.sh: New watchdog script that monitors Flink test execution and captures thread dumps when tests approach timeout limits
Modified .github/workflows/ci.yaml: Updated CI workflow to use watchdog specifically for Flink module tests (matrix.module == 'flink')
Enhanced debugging capabilities: Watchdog captures Java process information and thread dumps (jstack) at 95% of timeout and just before process termination
Artifact collection: Added new CI step to upload debug artifacts (thread dumps, execution logs) when Flink tests fail due to timeout
This introduces a new CI debugging feature for Flink tests. Key aspects:
Debug artifacts: When Flink tests timeout, three files are created and uploaded:
Artifact download: Failed Flink CI runs will have downloadable flink-debug-{run-number} artifacts containing all debugging information
Tests
API and Format
None
Documentation
None