Spatial Audio relatively high latency in QtQuick3D.Xr

When using Spatial Audio in a QtQuick3D scene and observing a spatial sound source from POV via a Meta Quest 3. The latency of the playback matters quite a bit in selling the spatial audio effect. Right now it is not that useful with as much latency as the output has.

Whilst rotating your head and observing a SpatialSound with a fixed location in a room you will notice as if the sound source is not coming from the physical location in the room during movement of your head. This is kind of a problem as it is disorienting for the user.  I suspect this due to how the Multimedia player is utilized under the hood by SpatialAudio.

Could you help me find avanues for lowering the latency of the audio playback by SpatialAudio? For example I am perfectly fine with only being able to use WAV audio formats and find ways to reduce the latency like how SoundEffect does already.

Some directions I'm looking into:

Finding ways to reduce the buffersize of the audiosink in use:

https://doc.qt.io/qt-6/qaudiosink.html#bufferSize

Update: found a solution to lower the latency as described in the comments below.