FFmpeg+Intel VAAPI decoding of certain h264/mp4 videos fails in 6.6.x+

Our application uses the QML MediaPlayer element to allow users to play back H264 videos that were muxed into .mp4 using the FFmpeg APIs (standalone from Qt).

In the jump to 6.6.x+, we found that some of our videos fail to play back when the Qt FFmpeg backend chooses VAAPI + the intel media driver (iHD). This only happens on certain mp4 files, and it isn't clear what the difference is between video files that cause vs don't cause the issue. I've attached two sample videos, one that triggers the issue, and one that doesn't.

When the decoding error occurs, I get a stream of FFmpeg errors that looks like:

[h264 @ 0x5594b8ae5c80] get_buffer() failed                                
[h264 @ 0x5594b8ae5c80] thread_get_buffer() failed                           
[h264 @ 0x5594b8ae5c80] decode_slice_header error                          
[h264 @ 0x5594b8ae5c80] no frame!                                
[h264 @ 0x5594b94202c0] get_buffer() failed   
[h264 @ 0x5594b94202c0] thread_get_buffer() failed 
[h264 @ 0x5594b94202c0] decode_slice_header error 
[h264 @ 0x5594b94202c0] no frame!                                   
[h264 @ 0x5594b8b20b00] get_buffer() failed   
[h264 @ 0x5594b8b20b00] thread_get_buffer() failed              
[h264 @ 0x5594b8b20b00] decode_slice_header error                          
[h264 @ 0x5594b8b20b00] no frame!        
[15:25:52.062][debug]: Invalid data found when processing input 

The first few frames of the video play normally, then it locks up when the "Invalid data found when processing input" message appears, a fraction of a second into the video.

Some notes:

• This doesn't happen in Qt6.5.2, but happens in all 6.6.x versions I've tried
• This does not appeared to be tied to the version of FFmpeg, at least there is no difference between FFmpeg 5.0 and 6.0
• The issue does not present itself when I allow FFmpeg to select the "cuda" HwAccel backend or an Nvidia VAAPI driver, only the Intel VAAPI driver
• The behavior is present when playing back several of our videos, and always manifests in the same way (locks up within the first fraction of a second - there is no apparent variation in timing per video file)
• This happens on multiple machines, some with only Intel integrated graphics (which is how we uncovered the error), and others that have both Intel integrated graphics and an Nvidia graphics card.
  • When an Nvidia card is present, qtmultimedia/ffmpeg prefers to select the "cuda" hardware decoder, which works, so we manually override the selection using environment variables, such as:

export QT_MEDIA_BACKEND=ffmpeg

export SYSTEM_LIBVA_DRIVERS_PATH="/usr/lib/x86_64-linux-gnu/dri"
export BUNDLED_LIBVA_DRIVERS_PATH="${CURRENT_DIR}/opt/mr/lib/thirdparty/dri"

export QT_FFMPEG_DECODING_HW_DEVICE_TYPES="vaapi"
export LIBVA_DRIVERS_PATH="${SYSTEM_LIBVA_DRIVERS_PATH}"
export LIBVA_DRIVER_NAME="iHD" 

• We provide our own built-from-source ffmpeg, libva, and intel-media-driver libraries/drivers, using Conan ( "ffmpeg/6.0.1", "libva/2.16.0", "intel-media-driver/22.6.6-dev")

• • However, it doesn't matter whether we allow the application to load these or the stock Kubuntu 22.04 apt-installed equivalents, the issue is still present
• The video files can be played fine in any other media player
• We also have our own hand-written real-time decoding + display pipeline within our app that uses the FFmpeg avcodec API with VAAPI+Intel iHD (to decode and display video streaming from a remote system) using the exact same libraries, and it has no issue decoding the h264 stream as it comes in. The only Qt-related part of this pipeline is the rendering of the decoded texture to a Qml VideoOutput element. This gives us confidence that the underlying H264 is probably fine, but maybe qtmultimedia could be interacting poorly with something in the mp4 container.

Ultimately, we are fairly certain that there has been some change in behavior, perhaps a regression, or perhaps some more aggressive error-checking, between 6.5 to 6.6, as our application can work with both versions, using all of the permutations of self-provided libraries or default Kubuntu libraries, except for Qt6.6+ with Intel VAAPI.