Loading of OpenSsl DLLs do not respect application's call to SetDllDirectory

Short summary: When the network code attempts to dynamically load the OpenSsl DLLs, it will fail if the application separates out the EXE from it's DLLs into different folders.

Here's the issue we are attempting to solve. Our application, OneDrive, installs its primary executable (OneDrive.exe) in a directory separate from the DLLs it uses. It’s a similar setup to how Google Chrome works.

All the program’s DLLs, including all the Qt5*.dll files, are in a version-specific subdirectory. The EXE just does some basic initialization and then loads a DLL from the subfolder and invokes the real “main” that’s in the DLL. But not before calling the Win32 API, SetDllDirectory, such that all subsequent LoadLibrary calls (and dependent DLLs) will search the version-specific subfolder. In other words, the OneDrive.exe is in the parent folder, which is what windows considers the “application folder” with regards to the DLL search path. The call to SetDllDirectory fixes up the search path used by LoadLibrary and the loader for dependent DLLs.

In our next release, we want to use open HTTPS and need to rebuild Qt5Network with SSL support. And then put the OpenSSL binaries (libeay32.dll and ssleay32.dll) in the version-specific folder with all the other DLLs.

First issue:
When placing our locally-built libeay32 and ssleay32.dll into the version-specific folder, I got a crash later on. The reason was that tryToLoadOpenSslWin32Library uses the QSystemLibrary class to load the DLL using its own algorithm of searching well known folders and the application path (location of the EXE), before searching the user’s PATH. The problem is that QSystemLibrary class does not know about the SetDllDirectory call that the application makes very early at startup.

What ended up happening was that the first load of ssleay32.dll wasn’t found in the application or system folder, so QSystemLibrary::load searches the PATH environment. It happened to find another copy of ssleay32.dll in my Ruby\bin directory which was in my PATH. So it calls LoadLibrary(c:\ruby\bin\ssleay32.dll). ssleay32.dll has a dependency on libeay32.dll. So the Windows loader, using the native DLL search algorithm, resolves the dependency by searching for the corresponding copy of libeay32.dll. But in this case, since the loader does respect the SetDllDirectory call, it finds libeay32.dll in the OneDrive version folder. And this kind of works. But then the tryLoadToOpenSslWin32Library will use QSystemLibrary again to load libeay32.dll. This again calls LoadLibrary with an absolute path back to the Ruby directory. This time loading a different copy of libeay32.dll. Eventually, when SSL calls are actually invoked, a crash occurs. The mismatched versions of libeay32.dll and libssl32.dll call into each other and hit some undefined behavior.

The simple fix to avoid the crash is to explicitly load libeay32.dll first. Because the subsequent load of ssleay32.dll will trigger the Windows loader to use the copy of libeay32.dll already loaded. This doesn’t fix the original problem of trying to load OpenSSL from the subfolder, but it avoids the crash.

Second issue:

We really don’t want an external copy of the SSL binaries getting loaded from anywhere other than our OneDrive version subfolder. But QSystemLibrary class doesn’t respect the SetDllDirectory call. So in the most general case, without the user having OpenSSL installed in a PATH folder, the loading of OpenSSL binaries will fail. Hence, no HTTPS support There’s several ways to fix this. Includes any combination of the following:

a. Switch to using QLibrary instead of QSystemLibrary. (i.e. Just use LoadLibrary without an absolute path: LoadLibrary(“libeay32.dll”); )
On modern versions of Windows, “safe search” for dll resolution is on by default which might mitigate some of the concerns QSystemLibrary was built for (maybe because Win XP didn't initially have this mode?). It would find the copy of libeay32.dll in the directory that the app specified with SetDllDirectory.

b. Modify QSystemLibrary::load to call the Win32 API, GetDllDirectory, and use the result of that call to influence the DLL search path.

c. Modify tryToLoadOpenSslWin32Library in qtbase\src\network\ssl\qsslsocket_openssl_symbols.cpp to not use QSystemLibrary, but instead use LoadLibraryEx with the appropriate flags. Requires some investigation (note: LoadLibraryEx is not available on XP).

d. tryToLoadOpenSslWin32Library obtains the path to the instance of Qt5Network.dll that is loaded and uses the path returned from that call to be used in the search order for the DLL.

e. Allow the application to do its own LoadLibrary calls for the OpenSSL binaries. Qt code could call GetModuleHandle("libeay32.dll") before attempting to do its own attempt to load.