Set widget window flag to Qt::Tool before showing it. All behaves correctly that window is on top of application but behind native windows.
Set widget window flag first to some other (Qt::Widget) and set window visible. Set now window flag to Qt::Tool and see that window stays on top of all desktop windows what is wrong.
Easy to test with the attached sample.
There's also error when trying to change window type from widget to tool:
NSWindow does not support utility styleMask 0x10