[Bug]: VBAQ causes severe blur on I-frames during motion in AMF encoder

[Ryu0833]

Describe the bug
With my testing of the encoder (visual test), it feels like AMF_VIDEO_ENCODER_ENABLE_VBAQ has a problem with I-frames. When there is motion in the video and an I-frame is inserted, the entire frame becomes blurred. So either VBAQ cannot properly handle I-frames (causing the frame to blur completely), or there is a problem with the VBAQ implementation.

In my test, I changed the keyframe interval from 2s
When the image is static, there is no problem (just a little blur, barely noticeable).
But when moving, at the moment an I-frame is inserted, the image becomes very blurry.

To Reproduce
Steps to reproduce the behavior using obs:

1. Set the following properties 'MaxNumRefFrames=3 RateControlPreanalysisEnable=1 EnableVBAQ=false HighMotionQualityBoostEnable=true ' when test the VBAQ false
2. start the encoding with resolution=1920*1080 bitrates=8000kbps the test.mkv (the test.mkv has fix QP=16 for high quality )
3. test with VBAQ following properties 'MaxNumRefFrames=3 RateControlPreanalysisEnable=1 EnableVBAQ=true HighMotionQualityBoostEnable=true '
4. start the encoding resolution=1920*1080 bitrates=8000kbps test.mkv

Setup:

• OS: Windows 11 24h2
• Driver Version 25.9.1
• GPU RX 5500 XT
• Which component has the issue Encoder

Expected behavior
When VBAQ is enabled, I-frames should remain clear (similar to when VBAQ is disabled), and the entire frame should not become blurred.

Additional context
This problem is consistent at both high and low bitrates — the blur issue always occurs.

[MikhailAMD]

Can you share command line for TranscodeHW sample or for FFmpeg to repro together with source file and both outputs? We need to see whole set of parameters.

[Ryu0833]

i did use obs last master and with:
1)testing with VBAQ=true 'MaxNumRefFrames=3 RateControlPreanalysisEnable=1 EnableVBAQ=false HighMotionQualityBoostEnable=true '
2)testing with VBAQ=true 'MaxNumRefFrames=3 RateControlPreanalysisEnable=1 EnableVBAQ=true HighMotionQualityBoostEnable=true '

but i cant send the videos of the test with the result test

[Ryu0833]

I forgot to mention that all the tests, both with VBAQ=true and VBAQ=false, were done using CBR.

[MikhailAMD]

I've asked rate control gurus to check.