QVideoSink with Qt 6 on Android ignores requested camera settings

We're attempting to migrate our app, Trimble Access, from Qt 5.15.x to Qt 6.x. One of the last remaining areas of concern for us is capturing video frames from the camera. In Qt 5.15, we were able to control the frame rate, frame size, and frame pixel format. In Qt 6 (so far we've tried this in Qt 6.4 and Qt 6.5.0), the reworked video capture framework seems to largely ignore our attempts to control these things. As a result, video frames are the wrong size and orientation, frames arrive too fast, and performance is badly affected.

I've attached our "UIVideoFrameGrabber" source modules. The code has some dependencies that may make it difficult to compile this and incorporate it in a test app, but it will at least demonstrate how we do this. It uses #ifdefs to build for both Qt 5.x and Qt 6.x, so you can see both the "old" and "new" approaches in the source code.

I've tested this on a custom Trimble device (TSC5), and on a phone (Google Pixel 5), with similar results in both cases.

In Qt 5.15.1, when we queried for the camera capabilities, we received the following specifications:

Camera Supports pixel formats :19 (Format_YV12), 23 (Format_NV21)

Camera Supports resolutions :[160,120],[176,144],[320,240],[240,320],[352,288],[480,320],[480,360],[640,360],[640,480],[480,640],[720,480],[800,480],[640,640],[864,480],[800,600],[1280,400],[1280,480],[1024,720],[1024,768],[1280,720],[1280,768],[1280,960],[1200,1200],[1440,1080],[1600,1200],[1920,1080]

Camera Supports frame rates :[15,15],[20,20],[7,24],[24,24],[7,30],[30,30] 

We would then request the camera format we wanted:

    settings.setResolution(QSize(640, 480));
    settings.setMinimumFrameRate(15.0);
    settings.setMaximumFrameRate(15.0);
    settings.setPixelAspectRatio(QSize(1, 1));
    settings.setPixelFormat(QVideoFrame::PixelFormat::Format_NV21); 

After initiating video streaming, our override of QAbstractVideoSurface:: present() would be called, and the frames would be delivered exactly as requested, at 15 Hz intervals:

Frame format = 23, frame size = (640, 480), image size = (640, 480) 

(Note that 'image' here is the result of calling frame.image().)

In Qt 6.5.0, when we query for the camera capabilities, we receive a list of camera formats that looks broadly equivalent to the Qt 5.15 specs (if represented a bit differently):

 Camera supported video formats (Pixel format | Resolution | Frame rates):
  15 | [160,120] | [7 - 30]
  19 | [160,120] | [7 - 30]
  15 | [176,144] | [7 - 30]
  19 | [176,144] | [7 - 30]
  15 | [240,320] | [7 - 30]
  19 | [240,320] | [7 - 30]
  ... several elided here...
  15 | [640,480] | [7 - 30]
  19 | [640,480] | [7 - 30]
  ... and many more rows after this one - a total of 62 camera formats

We select the format we want to use:

  19 (Format_NV21) | [640,480] | [7 - 30] 

Disturbingly, there is no way to request 15 fps - we can only pick from one of the camera formats that was reported to us, all of which specify fps = [7 - 30].

We then hook up a hander for the videoFrameChanged signal. When our onVideoFrameChanged() handler is called, the received frame has the following characteristics:

Frame format = 9 (Format_RGBA8888), frame size = (1200, 1600), image size = (1200, 1600) 

(Note that 'image' here is the result of calling frame.toImage().)

So - the frame is much larger than we want, has a portrait rather than landscape orientation, does not use the requested frame pixel format, and the frames appear to arrive "ASAP".

We can probably write additional code to overcome all of these issues, resizing and rotating the image. Our code already uses a timer to discard some of the frames, thereby effectively throttling the frame rate back to what we want (around 15 Hz). However, it would appear that there is significant overhead in having the multimedia framework deliver such high-resolution frames at such a high frame rate (even though we are discarding/ignoring many of the frames), as our whole app performs very badly when streaming video with Qt 6, which was never a problem with Qt 5. If we add more code to scale and/or rotate the images to produce the size and orientation we want, this would presumably further degrade performance. If the performance can't be improved, this would potentially be a show-stopper for our migration to Qt 6.

What we would really like is to be able to control the frame rate, frame pixel format, and frame size in Qt 6, and have the multimedia framework then deliver QVideoFrames that conform to the settings we've configured. That simply doesn't appear to be the case at present.