BypassWindowManagerHint might make applications freeze

QtWayland doesn't create a shell surface for windows with BypassWindowManagerHint. This can result in a freeze of the application. Attached is a small qml file which illustrates the issue.

Run with:
qmlscene freeze.qml

The QML lets the color of the background switch between red and blue every half a second. As soon as one clicks the button "Freeze", the application freezes on Wayland (tested with KWin and Weston), on X11 another window is opened.

The backtrace of the freeze is:

Thread 6 (Thread 0x7fffc7fff700 (LWP 27703)):
#0 0x00007ffff4c5e52d in poll () at ../sysdeps/unix/syscall-template.S:81
#1 0x00007fffee85d618 in poll (__timeout=-1, __nfds=1, __fds=0x7fffc7ffe180) at /usr/include/x86_64-linux-gnu/bits/poll2.h:46
#2 wl_display_dispatch_queue (display=0x7fffe4003460, queue=0x48ea70) at ../src/wayland-client.c:1574
#3 0x00007fffed6160bb in get_back_bo (dri2_surf=dri2_surf@entry=0x7fffc0003500) at ../../../../../../src/egl/drivers/dri2/platform_wayland.c:329
#4 0x00007fffed6168c4 in update_buffers (dri2_surf=dri2_surf@entry=0x7fffc0003500) at ../../../../../../src/egl/drivers/dri2/platform_wayland.c:444
#5 0x00007fffed616914 in image_get_buffers (driDrawable=<optimized out>, format=<optimized out>, stamp=<optimized out>, loaderPrivate=0x7fffc0003500, buffer_mask=<optimized out>, buffers=0x7fffc7ffe280)
at ../../../../../../src/egl/drivers/dri2/platform_wayland.c:563
#6 0x00007fffde591abb in ?? () from /usr/lib/x86_64-linux-gnu/dri/i965_dri.so
#7 0x00007fffde591e11 in ?? () from /usr/lib/x86_64-linux-gnu/dri/i965_dri.so
#8 0x00007fffde585b20 in ?? () from /usr/lib/x86_64-linux-gnu/dri/i965_dri.so
#9 0x00007ffff79edc42 in QOpenGLFunctions::glClear (this=0x7fffc0003f00, mask=17664) at /home/martin/src/qt5/qtbase/include/QtGui/../../src/gui/opengl/qopenglfunctions.h:596
#10 0x00007ffff7a01545 in QSGBindable::clear (this=0x7fffc7ffea50, mode=...) at scenegraph/coreapi/qsgrenderer.cpp:64
#11 0x00007ffff79e85aa in QSGBatchRenderer::Renderer::renderBatches (this=0x7fffc0004be0) at scenegraph/coreapi/qsgbatchrenderer.cpp:2475
#12 0x00007ffff79e9ddf in QSGBatchRenderer::Renderer::render (this=0x7fffc0004be0) at scenegraph/coreapi/qsgbatchrenderer.cpp:2697
#13 0x00007ffff7a01bb2 in QSGRenderer::renderScene (this=0x7fffc0004be0, bindable=warning: RTTI symbol not found for class 'QSGRenderer::renderScene(unsigned int)::B'
...) at scenegraph/coreapi/qsgrenderer.cpp:217
#14 0x00007ffff7a0199d in QSGRenderer::renderScene (this=0x7fffc0004be0, fboId=0) at scenegraph/coreapi/qsgrenderer.cpp:177
#15 0x00007ffff7a1d35b in QSGRenderContext::renderNextFrame (this=0x114dbc0, renderer=0x7fffc0004be0, fboId=0) at scenegraph/qsgcontext.cpp:555
#16 0x00007ffff7a79f52 in QQuickWindowPrivate::renderSceneGraph (this=0x114e330, size=...) at items/qquickwindow.cpp:384
#17 0x00007ffff7a419ad in QSGRenderThread::syncAndRender (this=0xd1f2b0) at scenegraph/qsgthreadedrenderloop.cpp:623
#18 0x00007ffff7a4236d in QSGRenderThread::run (this=0xd1f2b0) at scenegraph/qsgthreadedrenderloop.cpp:704
#19 0x00007ffff5d6fc65 in QThreadPrivate::start (arg=0xd1f2b0) at thread/qthread_unix.cpp:329
#20 0x00007ffff57c40a4 in start_thread (arg=0x7fffc7fff700) at pthread_create.c:309
#21 0x00007ffff4c6706d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111

Thread 5 (Thread 0x7fffd1e6b700 (LWP 27701)):
#0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
#1 0x00007ffff5d713a2 in QWaitConditionPrivate::wait (this=0x1231e70, time=18446744073709551615) at thread/qwaitcondition_unix.cpp:136
#2 0x00007ffff5d71175 in QWaitCondition::wait (this=0x126ea08, mutex=0x126ea00, time=18446744073709551615) at thread/qwaitcondition_unix.cpp:208
#3 0x00007ffff7a46231 in QSGRenderThreadEventQueue::takeEvent (this=0x126e9f8, wait=true) at scenegraph/qsgthreadedrenderloop.cpp:236
#4 0x00007ffff7a42084 in QSGRenderThread::processEventsAndWaitForMore (this=0x126e980) at scenegraph/qsgthreadedrenderloop.cpp:684
#5 0x00007ffff7a42481 in QSGRenderThread::run (this=0x126e980) at scenegraph/qsgthreadedrenderloop.cpp:713
#6 0x00007ffff5d6fc65 in QThreadPrivate::start (arg=0x126e980) at thread/qthread_unix.cpp:329
#7 0x00007ffff57c40a4 in start_thread (arg=0x7fffd1e6b700) at pthread_create.c:309
#8 0x00007ffff4c6706d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111

Thread 4 (Thread 0x7fffdb99b700 (LWP 27700)):
#0 0x00007ffff4c5e52d in poll () at ../sysdeps/unix/syscall-template.S:81
#1 0x00007ffff173a1dc in ?? () from /lib/x86_64-linux-gnu/libglib-2.0.so.0
#2 0x00007ffff173a2ec in g_main_context_iteration () from /lib/x86_64-linux-gnu/libglib-2.0.so.0
#3 0x00007ffff6017545 in QEventDispatcherGlib::processEvents (this=0x7fffcc0008c0, flags=...) at kernel/qeventdispatcher_glib.cpp:417
#4 0x00007ffff5f9a764 in QEventLoop::processEvents (this=0x7fffdb99ade0, flags=...) at kernel/qeventloop.cpp:128
#5 0x00007ffff5f9aa60 in QEventLoop::exec (this=0x7fffdb99ade0, flags=...) at kernel/qeventloop.cpp:204
#6 0x00007ffff5d67e6e in QThread::exec (this=0xbfc9a0) at thread/qthread.cpp:503
#7 0x00007ffff668101b in QQmlThreadPrivate::run (this=0xbfc9a0) at qml/ftw/qqmlthread.cpp:141
#8 0x00007ffff5d6fc65 in QThreadPrivate::start (arg=0xbfc9a0) at thread/qthread_unix.cpp:329
#9 0x00007ffff57c40a4 in start_thread (arg=0x7fffdb99b700) at pthread_create.c:309
#10 0x00007ffff4c6706d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111

Thread 3 (Thread 0x7fffdc19c700 (LWP 27699)):
#0 0x00007ffff4c5e52d in poll () at ../sysdeps/unix/syscall-template.S:81
#1 0x00007ffff173a1dc in ?? () from /lib/x86_64-linux-gnu/libglib-2.0.so.0
#2 0x00007ffff173a2ec in g_main_context_iteration () from /lib/x86_64-linux-gnu/libglib-2.0.so.0
#3 0x00007ffff6017545 in QEventDispatcherGlib::processEvents (this=0x7fffd40008c0, flags=...) at kernel/qeventdispatcher_glib.cpp:417
#4 0x00007ffff5f9a764 in QEventLoop::processEvents (this=0x7fffdc19bdb0, flags=...) at kernel/qeventloop.cpp:128
#5 0x00007ffff5f9aa60 in QEventLoop::exec (this=0x7fffdc19bdb0, flags=...) at kernel/qeventloop.cpp:204
#6 0x00007ffff5d67e6e in QThread::exec (this=0x7fffee167920 <(anonymous namespace)::Q_QGS__q_manager::innerFunction()::holder>) at thread/qthread.cpp:503
#7 0x00007fffee0d13da in QDBusConnectionManager::run (this=0x7fffee167920 <(anonymous namespace)::Q_QGS__q_manager::innerFunction()::holder>) at qdbusconnection.cpp:152
#8 0x00007ffff5d6fc65 in QThreadPrivate::start (arg=0x7fffee167920 <(anonymous namespace)::Q_QGS__q_manager::innerFunction()::holder>) at thread/qthread_unix.cpp:329
#9 0x00007ffff57c40a4 in start_thread (arg=0x7fffdc19c700) at pthread_create.c:309
#10 0x00007ffff4c6706d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111

Thread 2 (Thread 0x7fffeb579700 (LWP 27698)):
#0 0x00007ffff4c5e52d in poll () at ../sysdeps/unix/syscall-template.S:81
#1 0x00007ffff173a1dc in ?? () from /lib/x86_64-linux-gnu/libglib-2.0.so.0
#2 0x00007ffff173a2ec in g_main_context_iteration () from /lib/x86_64-linux-gnu/libglib-2.0.so.0
#3 0x00007ffff6017545 in QEventDispatcherGlib::processEvents (this=0x7fffe40008c0, flags=...) at kernel/qeventdispatcher_glib.cpp:417
#4 0x00007ffff5f9a764 in QEventLoop::processEvents (this=0x7fffeb578de0, flags=...) at kernel/qeventloop.cpp:128
#5 0x00007ffff5f9aa60 in QEventLoop::exec (this=0x7fffeb578de0, flags=...) at kernel/qeventloop.cpp:204
#6 0x00007ffff5d67e6e in QThread::exec (this=0x431b50) at thread/qthread.cpp:503
#7 0x00007ffff5d68010 in QThread::run (this=0x431b50) at thread/qthread.cpp:570
#8 0x00007ffff5d6fc65 in QThreadPrivate::start (arg=0x431b50) at thread/qthread_unix.cpp:329
#9 0x00007ffff57c40a4 in start_thread (arg=0x7fffeb579700) at pthread_create.c:309
#10 0x00007ffff4c6706d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111

Thread 1 (Thread 0x7ffff7e19780 (LWP 27693)):
#0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
#1 0x00007ffff5d713a2 in QWaitConditionPrivate::wait (this=0x1247f00, time=18446744073709551615) at thread/qwaitcondition_unix.cpp:136
#2 0x00007ffff5d71175 in QWaitCondition::wait (this=0xd1f2f8, mutex=0xd1f2f0, time=18446744073709551615) at thread/qwaitcondition_unix.cpp:208
#3 0x00007ffff7a4506b in QSGThreadedRenderLoop::polishAndSync (this=0x114d9f0, w=0x126a110, inExpose=false) at scenegraph/qsgthreadedrenderloop.cpp:1177
#4 0x00007ffff7a43f00 in QSGThreadedRenderLoop::handleUpdateRequest (this=0x114d9f0, window=0x114d760) at scenegraph/qsgthreadedrenderloop.cpp:1004
#5 0x00007ffff7a7e2b8 in QQuickWindow::event (this=0x114d760, e=0x7fffffffcb50) at items/qquickwindow.cpp:1401
#6 0x00007ffff71aecd2 in QApplicationPrivate::notify_helper (this=0x42d260, receiver=0x114d760, e=0x7fffffffcb50) at kernel/qapplication.cpp:3717
#7 0x00007ffff71ac3fd in QApplication::notify (this=0x7fffffffd880, receiver=0x114d760, e=0x7fffffffcb50) at kernel/qapplication.cpp:3160
#8 0x00007ffff5f9dec0 in QCoreApplication::notifyInternal2 (receiver=0x114d760, event=0x7fffffffcb50) at kernel/qcoreapplication.cpp:1002
#9 0x00007ffff69d6c96 in QCoreApplication::sendEvent (receiver=0x114d760, event=0x7fffffffcb50) at ../../include/QtCore/../../src/corelib/kernel/qcoreapplication.h:227
#10 0x00007ffff6a025db in QWindowPrivate::deliverUpdateRequest (this=0x114e330) at kernel/qwindow.cpp:2123
#11 0x00007ffff6a0255c in QWindow::event (this=0x114d760, ev=0x7fffffffd3c0) at kernel/qwindow.cpp:2105
#12 0x00007ffff7a7e315 in QQuickWindow::event (this=0x114d760, e=0x7fffffffd3c0) at items/qquickwindow.cpp:1416
#13 0x00007ffff71aecd2 in QApplicationPrivate::notify_helper (this=0x42d260, receiver=0x114d760, e=0x7fffffffd3c0) at kernel/qapplication.cpp:3717
#14 0x00007ffff71ac3fd in QApplication::notify (this=0x7fffffffd880, receiver=0x114d760, e=0x7fffffffd3c0) at kernel/qapplication.cpp:3160
#15 0x00007ffff5f9dec0 in QCoreApplication::notifyInternal2 (receiver=0x114d760, event=0x7fffffffd3c0) at kernel/qcoreapplication.cpp:1002
#16 0x00007ffff5fa1ef6 in QCoreApplication::sendEvent (receiver=0x114d760, event=0x7fffffffd3c0) at ../../include/QtCore/../../src/corelib/kernel/qcoreapplication.h:227
#17 0x00007ffff601585a in QTimerInfoList::activateTimers (this=0x479520) at kernel/qtimerinfo_unix.cpp:637
#18 0x00007ffff6016b78 in timerSourceDispatch (source=0x4794c0) at kernel/qeventdispatcher_glib.cpp:176
#19 0x00007ffff1739fe7 in g_main_context_dispatch () from /lib/x86_64-linux-gnu/libglib-2.0.so.0
#20 0x00007ffff173a240 in ?? () from /lib/x86_64-linux-gnu/libglib-2.0.so.0
#21 0x00007ffff173a2ec in g_main_context_iteration () from /lib/x86_64-linux-gnu/libglib-2.0.so.0
#22 0x00007ffff6017545 in QEventDispatcherGlib::processEvents (this=0x473f20, flags=...) at kernel/qeventdispatcher_glib.cpp:417
#23 0x00007fffeeae57aa in QPAEventDispatcherGlib::processEvents (this=0x473f20, flags=...) at eventdispatchers/qeventdispatcher_glib.cpp:115
#24 0x00007ffff5f9a764 in QEventLoop::processEvents (this=0x7fffffffd6d0, flags=...) at kernel/qeventloop.cpp:128
#25 0x00007ffff5f9aa60 in QEventLoop::exec (this=0x7fffffffd6d0, flags=...) at kernel/qeventloop.cpp:204
#26 0x00007ffff5f9e60a in QCoreApplication::exec () at kernel/qcoreapplication.cpp:1272
#27 0x00007ffff69f101e in QGuiApplication::exec () at kernel/qguiapplication.cpp:1571
#28 0x00007ffff71abde1 in QApplication::exec () at kernel/qapplication.cpp:2976
#29 0x00000000004081c1 in main (argc=2, argv=0x7fffffffde78) at main.cpp:626

The problem here is that the wl_surface gets created and it's rendered to. But as no wl_shell_surface is created the Compositor doesn't know anything about the "window", doesn't render it and will never trigger the frame rendered callback. Mesa internally recognizes it rendered a frame, but didn't get the callback yet, thus blocks the rendering thread till it gets the callback (which it never will). The main gui thread on the other hand is blocked on waiting for the rendering thread. The problem can also be reproduced with QSG_RENDER_LOOP=basic with a slightly different backtrace.