The org.bluez.ProfileManager1 interface and org.bluez.Profile1 interfaces have to be used.
Currently QtBluetooth directly interacts with the SDP server (not using the DBUS interfaces) on the client side to find the service. Normally the SDP discovery would provide the channel for SPP profiles which could be used to directly connect to the remote profile. If bluetoothd is not running in compat mode the sdp server does not accept requests anymore as seen under:
The client side of QBluetoothSocket needs to be modified to use the above mentioned ProfileManager1/Profile1 interfaces.
On the server side might have similar restrictions.
The current workaround is to follow the suggestions in the above link.