Python 2.7 or 3.6.6 built against 10.6 SDK (https://www.python.org/ftp/python/3.6.6/python-3.6.6-macosx10.6.pkg)
Qt 5.12 against 10.12 SDK
PySide2 5.11.0 against 10.12 SDK
When building PySide (5.11 or dev) against Qt 5.12 (dev) on macOS (10.12), using the official Python (2 or 3 doesn't matter) package which was built against a 10.6 macOS SDK, no widgets are painted, all GUI examples are broken, a bunch of tests in the CI hang.
If you use a self-compiled Python, or a newer Python package that is built against a newer macOS SDK (10.9), everything works properly.
Current workaround to make tests and CI integrations pass is found at https://codereview.qt-project.org/#/c/233830/
Below will follow the story of investigating this obscure bug.
The python script used for testing was in the pyside-setup repo, examples/tutorial/t1.py
After some poking around the Qt cocoa QPA plugin, and enabling the following logging categories
as well as sprinkling some additional qCDebug()s inside qnsview_drawing.mm,
the first observed issue was that -[QNSView drawLayer] was not called, and hence no widget content was displayed. Much later I noticed that the backingLayer was not created at all, which proved to be crucial.
After realllly many build tests I determined that the widgets got painted (and drawLayer called) if the used official Python was built with minimum deployment target set to 10.9, but not 10.6.
And yet if I built Python myself with deployment target set to 10.6, everything worked. So that ended up being a bit of a dead end. We will come back to this later.
At the same time I was doing tests with Qt 5.11, and there everything worked, even with deployment target 10.6. After some advice from fellow colleagues and scouring of the git log, the initial culprit commit in qtbase was https://codereview.qt-project.org/#/c/230735/ .
Qt switched to using layer-backed NSViews and that somehow broke rendering when using Python.
The first thing to try, and the already temporary committed workaround is to set the environment variable QT_MAC_WANTS_LAYER=0, which disabled layer-backed views, and the issue went away!
Still I didn't know why layer-backed NSViews didn't work.
At some point it occurred to me try and and build against Qt 5.11, and explicitly enable (opt-in) to layer-backed NSViews. To my surprise everything worked, even with the official 10.6-built python!
That means there must have been some change done that regressed from Qt 5.11 to Qt 5.12, because layer backing worked!
After a lot of disassembly debugging and comparing the execution flow and re-reading the relevant commits I found it: https://codereview.qt-project.org/#/c/223718/4
If I revert the commit by explicitly calling -[NSView setWantsLayer], then the backingLayer is created, -[QNSView displayLayer] is called, drawing works!
Apparently it is not enough to just override -[NSView wantsLayer], calling setWantsLayer has side-effects which in our case are important.
But remember that this issue only happened for official Python 10.6-built, so it doesn't explain why I would need to call setWantsLayer explicitly for 10.6, but not for any other python version.
I went back to disassembling and execution tracing between the working and non-working case, to see what was the difference. In the working case something must be calling -[NSView setWantsLayer] for us, or something similar to it.
After some time I found it, and this is the relevant decompiled source code:
For the 10.6 non-working case __NSViewLayerBackWindowFrame() returned "0", which means it skipped over calling setWantsLayer which I noted above.
The code for "__NSViewLayerBackWindowFrame" is :
Googling for "NSViewLayerBackWindowFrame" returned a single result, unsurprisingly in Chromium https://bugs.chromium.org/p/chromium/issues/detail?id=312462
Reading through that bug report and some other pages regarding NSGetBoolAppConfig, I found a way to enable this app config property via "defaults write" command line application.
First I enabled
and reran the example with Python 10.6-based to see this:
I then reran the same with Python 10.9-based, and diff-ed through the list to see the different config values. After some trial and error I tried this:
And then the example worked on Python 10.6-based!
Apparently it's not the minimum deployment target that is important, but the version of the SDK against which you build! Depending on the SDK you build against, certain silent behaviour changes can happen via these secret / private App config values.
Official Python is built on a 10.6 OS X version against 10.6 SDK. Whereas I was building with SDK 10.12, which explains why I couldn't reproduce the issue with a self-built Python.
The cleanest solution for us to make layer-backed NSViews work with a Python built against an old SDK, is to revert https://codereview.qt-project.org/#/c/223718/4 and explicitly call -[NSView setWantsLayer] ourselves.
Here is the disassembly of -[NSView setWantsLayer] and -[NSView _doSetWantsLayerYES] and why it's important to call the former explicitly. It calls "_doSetWantsLayerYES" which ends up creating the backingLayer, and for the 10.6 sdk case because neither Qt code nor AppKit code called setWantsLayer, no backing layer was created, and nothing was drawn.