Crash on macOS when using winId() after hiding/showing QMainWindow (introduced in Qt 5.11)

Summary:

Starting with Qt 5.11, winId() seems to return a freed pointer when the QMainWindow is hidden and then showed which led to a crash when used. This issue could not be reproduced on Qt 5.9 and Qt 5.10 and has been introduced in Qt 5.11.

See attached screencast: screencast.mov

Steps to reproduce:

1. Download the attached application QtBlackTitlebar QtBlackTitlebar_src.zip
2. Compile the application on macOS 10.13.5 and run it
3. Close the window
4. Using the QSystemTray, display the window

Result: The application crashes.

More information:

The application tries to determine if the new Key NSWindow is the main window in:

- (void)windowDidBecomeKey:(NSNotification*)notification

In order to do so, it gets the singleton instance of the MainWindow (which is a QMainWindow) and gets the winId() pointer. It casts the winId() pointer to an NSView and compares the NSWindow to the NSWindow passed in the NSNotification.

Using winId() like this is probably not ideal but the value returned by winId() should never be a freed pointer. If the underlaying object has been released, winId() should probably return a NULL pointer.

Also I could not find any proper API to detect if the NSWindow received in windowDidBecomeKey: is the main QMainWindow. A possible workaround could be to check the class type of the NSWindow like the following, although it is again not clean:

- (BOOL)isMainWindow:(NSWindow*)inWindow
{
if(inWindow == nil)
return NO;

// Only the MainWindow is an instance of QNSWindow
if ([NSStringFromClass([inWindow class]) isEqualToString:@"QNSWindow"])
{
return YES;
}

return NO;
}