There's a memory leak in [QNSWindowHelper initWithNSWindow:] method in qtbase/src/plugins/platforms/cocoa/qcocoawindow.mm file (line 145 in 5.9.4 branch):

_window.delegate = [[QNSWindowDelegate alloc] initWithQCocoaWindow:_platformWindow];

Looks like a missing autorelease.

Attached example can be used to see this. 

It creates 5000 dialogs one by one, show, close and release them. After that it waits giving a chance to release resources asynchronously just in case. It's visible that memory grows during dialogs creation/releasing and never restores back when all dialogs are released. 'leaks' utility report showing QNSWindowDelegate leaks is also attached.