Updating the mirror of test-amd.yaml as of 2025-11-18 #29016
Conversation
Signed-off-by: Alexei V. Ivanov <alexei.ivanov@amd.com>
|
This pull request has merge conflicts that must be resolved before it can be |
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Code Review
This pull request updates the CI configuration in .buildkite/test-amd.yaml. The changes involve adding new tests, updating test filters, and disabling some tests, likely to sync with the current state of the test suite. My review found one issue: an outdated comment that incorrectly states a test is skipped, which could cause confusion. I've provided a comment to address this.
| commands: # LMEval | ||
| # Transcription WER check is skipped because encoder-decoder models are not supported on ROCm, see https://github.com/vllm-project/vllm/issues/27442 | ||
| - pytest -s entrypoints/openai/correctness/ --ignore entrypoints/openai/correctness/test_transcription_api_correctness.py | ||
| - pytest -s entrypoints/openai/correctness/ |
There was a problem hiding this comment.
With the removal of the --ignore flag, test_transcription_api_correctness.py will now be executed. However, the comment on the preceding line (713) which states that this test is skipped is now outdated. This can be misleading for developers. Please remove the outdated comment to maintain clarity in the CI configuration.
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
| - vllm/model_executor/models/whisper.py | ||
| commands: # LMEval | ||
| # Transcription WER check is skipped because encoder-decoder models are not supported on ROCm, see https://github.com/vllm-project/vllm/issues/27442 | ||
| - pytest -s entrypoints/openai/correctness/ --ignore entrypoints/openai/correctness/test_transcription_api_correctness.py | ||
| - pytest -s entrypoints/openai/correctness/ |
There was a problem hiding this comment.
The OpenAI API correctness step now runs the entire entrypoints/openai/correctness directory, but the comment above still states that transcription WER checks must be skipped because encoder–decoder models are unsupported on ROCm. The previously ignored tests/entrypoints/openai/correctness/test_transcription_api_correctness.py contains an unconditional integration test that downloads audio datasets and depends on packages like librosa and evaluate without any ROCm guard. Running it on the AMD agents will likely fail due to missing dependencies or unsupported model execution. If the ROCm limitation still applies, the command should continue to ignore that file or add an explicit skip inside the test.
Useful? React with 👍 / 👎.
| - pytest -s entrypoints/openai/correctness/ | ||
|
|
||
| - label: OpenAI-Compatible Tool Use # 23 min | ||
| timeout_in_minutes: 35 |
There was a problem hiding this comment.
we need to ignore this since encoder-decoder models are not supported on ROCm
There was a problem hiding this comment.
There is an effort to support specifically OpenAI Whisper (e.g. #28376)
for which Word Error Rate (WER) is the predominant quality metric.
There was a problem hiding this comment.
@Alexei-V-Ivanov-AMD Could we disable the test first? The PR are still under reviewing. It will take some time.
Signed-off-by: Alexei V. Ivanov <alexei.ivanov@amd.com>
Signed-off-by: Alexei V. Ivanov <alexei.ivanov@amd.com>
…9016) Signed-off-by: Alexei V. Ivanov <alexei.ivanov@amd.com> Signed-off-by: LuminolT <lumischen01@gmail.com>
…9016) Signed-off-by: Alexei V. Ivanov <alexei.ivanov@amd.com>
…9016) Signed-off-by: Alexei V. Ivanov <alexei.ivanov@amd.com> Signed-off-by: Runkai Tao <rt572@physics.rutgers.edu>
…9016) Signed-off-by: Alexei V. Ivanov <alexei.ivanov@amd.com>
…9016) Signed-off-by: Alexei V. Ivanov <alexei.ivanov@amd.com>
…9016) Signed-off-by: Alexei V. Ivanov <alexei.ivanov@amd.com>
…9016) Signed-off-by: Alexei V. Ivanov <alexei.ivanov@amd.com> Signed-off-by: Xingyu Liu <charlotteliu12x@gmail.com>
4 participants
Updating the mirror of test-amd.yaml as of 2025-11-18
Signed-off-by: Alexei V. Ivanov alexei.ivanov@amd.com