[REG->5.14] QWindowsContext::registerWindowClass can fail when the Qt libs are different versions

We have a 3DS Max 2020 Plugin that uses the Qt UI. We use a custom Qt namespace in our compilation to avoid any potential conflicts between both.

3DS Max currently uses Qt, and obviously it runs and loads its own Windows platform plugin first before calling any other 3rd party plugins.

Whenever our Plugin starts, it runs into several problems when creating any Window components via the QWindowsContext::registerWindowClass method. The checks for avoiding any potential conflicts are skipped, and therefore, all attempts to create several window classes with the same name result in failure. Our clients reported this issue first as a "white window" glitch whenever our Qt UI was used.

After several hours of investigating the issue, I believe I've determined the main cause as to why the conflict checks are skipped.

Here's the relevant snippet of code for the method:

...
QString QWindowsContext::registerWindowClass(QString cname,
                                             WNDPROC proc,
                                             unsigned style,
                                             HBRUSH brush,
                                             bool icon)
{
    // since multiple Qt versions can be used in one process
    // each one has to have window class names with a unique name
    // The first instance gets the unmodified name; if the class
    // has already been registered by another instance of Qt then
    // add a UUID.
    static int classExists = -1;
    const auto appInstance = static_cast<HINSTANCE>(GetModuleHandle(nullptr));
    if (classExists == -1) {
        WNDCLASS wcinfo;
        classExists = GetClassInfo(appInstance, reinterpret_cast<LPCWSTR>(cname.utf16()), &wcinfo);
        classExists = classExists && wcinfo.lpfnWndProc != proc;
    }

  if (classExists)
   cname += QUuid::createUuid().toString();

  if (d->m_registeredWindowClassNames.contains(cname))
    return cname;
 
  WNDCLASSEX wc;
  wc.cbSize       = sizeof(WNDCLASSEX);

...

This seems fine at first glance: if an existing class is found that has the same name, it adds a random string generated by an UUID to avoid the issue. However, given the use of the static number, this check is only performed for the first class that is registered.

If the first time this method is called a class conflict is not found, it assumes no conflict will be found for any future calls. If instead it results in a class conflict, it will always add the UUID in any future calls regardless of the existence of a class conflict.

This check has failed for me under Qt 5.14.0 and the latest 3DS Max 2020 update, as it seems this Qt version calls the method first for registering "QtPowerDummyWindow", which 3DS Max 2020 has not previously registered since it's currently on Qt 5.12.4. This makes the static value false, so the next attempt at registering the class fails completely with a Windows API error (for example on Qt5ClipboardView).

I don't really understand the purpose of the static variable in this method. I assume it's some sort of optimization to avoid further checks, but it seems like it's not very resilient against potential changes between Qt versions like the addition of the "Power Notifications" as shown above.

I've verified that removing the static attribute from the variable fixes any conflicts and allows both the Plugin and 3DS Max to work properly. I'm not sure if this is the preferred way to fix it, since the if classExists == -1 check does become redundant after all. This would result in an increased number of GetClassInfo calls, but I don't see a solution unless the first call to the method is forced to always be the same regardless of the Qt version.