[macOS 15 Beta 6] SslInvalidUserDataError on second connection with QSslServer

Background

Currently, I have a multiplatform application (Windows and macOS) that is expected to receive TCP connections from third-party applications in the same machine. That is, no real network connections, but local ones.

For security, the application is expected to encrypt the TCP connections via an SSL certificate. The SSL certificate is official, issued by DigiCert. I'm fully aware that these kind of certificates are usually provided for web servers, but I have the specific need of using a real certificate.

Until recently, the application has been working just fine on all my systems (Windows 10 and 11, macOS 10.14 -> 14). No issues. The problem I have is with macOS 15.

I just received reports from some users that my application is having trouble in such OS version, and just that version.

However, after confirming the issue, I also confirmed that older versions of my application don't present the error.

The setup

The older version of my application that is NOT failing in macOS 15 is built using Qt 6.2.2 and OpenSSL 1.x. The extra OpenSSL library included is because I need to encrypt some data pieces in BlowFish (I know, is not recommended anymore). This version of the application works just fine under macOS 15 Beta 5.

The newest version of my application that IS failing in macOS 15 is built using Qt 6.7.2 and OpenSSL 3.x. The extra OpenSSL library included is due to the same requirement (BlowFish encryption). I load the "legacy" providers when needed and then unload stuff. This version of the application is presenting a weird error that is 100% reproducible on my end.

The error

From my original source code, the application detects the first incoming connection, receives it and processes it. The connection is encrypted and data is transfered successfully. However, right after that, if a second application attempts to connect to my application, the connection fails.

This is 100% repro rate on my setup. No matter what is connected to my server first, the problem is when the second connection is attempted.

Here, you can find the two most relevant classes of the latest attempts I have tried (class ClientConnection and class SslServer, attached to the ticket):

clientconnection.cc

The "SslServer" class is only a sort of "wrapper" for the QSslServer class. After launch, it receives the corresponding call to load the SSL certificate files I have (a CRT and a KEY files), which are embedded as resources.

The load goes well and the setup happens, where I set the corresponding QSslConfiguration instance to the QSslServer instance. Then the corresponding call to start listening happens.

After this, the application is listening.

When the first connection happens, the application receives the connection request and the slot onServerNewConnection is called.  After it, the slot onServerPendingConnectionAvailable from the SslServer class is called. In this call, the QSslSocket of the connection is extracted and sent to a new instance of a secondary custom class, named ClientConnection (from which I also provide the source).

Then a signal is emitted to store the client connection in a list hashed by numeric IDs. The connection is now able to communicate correctly. I have no issues here.

Then, a second connection is attempted. When the second connection is starting, I receive the call to the slot onServerNewConnection, but then, inmediately, I receive errors from the QSslServer class instance.

This is the example of what I see from the Terminal screen during the first connection:

LOG onServerNewConnection New connection
LOG onSocketStartedEncryptionHandshake Started encryption
LOG onServerPendingConnectionAvailable Pending connection available
LOG onServerPendingConnectionAvailable Checking for pending connection
LOG onServerPendingConnectionAvailable Creating PendingConnection instance
LOG onServerPendingConnectionAvailable Setting ID: 1
LOG (1) setID
LOG onServerPendingConnectionAvailable Setting QSslSocket
LOG (1) setConnectionSocket Starting
LOG (1) setConnectionSocket Storing and connecting
LOG (1) setConnectionSocket Connections done, checking for available bytes
LOG (1) setConnectionSocket Pending bytes so far: 0
LOG (1) setConnectionSocket Nothing to read, stop new client operation
LOG onServerPendingConnectionAvailable Notifying about new client

Then, comes the second connection:

LOG onServerNewConnection New connection
LOG onSocketConnection Error Connection problem found.
LOG onSocketConnectionError Problem=SSL Invalid User Data.
LOG onSocketConnectionError Connection problem found.
LOG onSocketConnectionError Problem=SSL Internal Error.
LOG onSocketStartedEncryptionHandshake Started encryption 

The actual text of "SSL Invalid User Data" corresponds to the result of a function I have that converts the actual Qt error value QAbstractSocket::SslInvalidUserDataError. The other message, "SSL Internal Error", comes from the value QAbstractSocket::SslInternalError.

System information

The machine that compiled the old version of my application (the one that works on macOS 15) and the newest (the one that doesn't work on macOS 15) is the exact same:

• macOS 13.x
• XCode 14.3.1
• Intel machine
• Compiled universal (Intel+ARM) of my application
• Signed and notarized by Apple with valid credentials.

Just in case, I checked all the information I thought would be useful to know, during each connection detection, as to identify if there is any difference (in case something changes between connections), but so far, it appears that no changes are detected:

LOG incommingConnection QSslSocket active backend: securetransport
LOG incommingConnection Available backend: securetransport
LOG incommingConnection Available backend: cert-only
LOG incommingConnection Ssl Build Version number: 0
LOG incommingConnection Ssl Build Version string: Secure Transport, macOS 15.0
LOG incommingConnection Ssl Version number: 0
LOG incommingConnection Ssl Version string: Secure Transport, macOS 15.0
LOG incommingConnection Supports SSL: 1
LOG incommingConnection Ssl Class provided: Socket
LOG incommingConnection Ssl Class provided: Certificate
LOG incommingConnection Ssl Class provided: Key
LOG incommingConnection Ssl Protocol supported: Any
LOG incommingConnection Ssl Protocol supported: SecureProtocols
LOG incommingConnection Ssl Protocol supported: TlsV1_0 DEPRECATED
LOG incommingConnection Ssl Protocol supported: TlsV1_0OrLater DEPRECATED
LOG incommingConnection Ssl Protocol supported: TlsV1_1 DEPRECATED
LOG incommingConnection Ssl Protocol supported: TlsV1_1OrLater DEPRECATED
LOG incommingConnection Ssl Protocol supported: TlsV1_2
LOG incommingConnection Ssl Protocol supported: TlsV1_2OrLater
LOG incommingConnection Ssl Supported Feature: ClientSideAlpn 

From what I can see, the only relevant element is that, effectively, Qt is using the Secure Transport backend at all times.

What I have tested

• The original code (that is working in the older Qt) uses a QTcpServer and reimplements the incomingConnection slot to assign the socket handle directly to a QSslSocket instance.
• Already attempted to switch to the more clean QSslServer (as seen in the provided code examples). It fails in the same way.
• Already attempted to change the backend. The "cert" backend causes issues in the application when doing other network stuff (TLS errors about missing functions).
• Already attempted to change the way I setup the certificates:
  • Directly to the QSslServer (as seen in the example provided).
  • Directly to the {{QSslSocket }}once got.
  • Globally to the "setDefaultConfiguration" member function from QSslConfiguration, before the {{QSslServer }}or {{QTcpServer }}start.
• Already attempted to change the Protocol used in the QSslConfiguration.

I believe it's also relevant to mention that I also already attempted to use the latest XCode, version 16, Beta 5. I get the exact same errors.

—

What could be the error I'm having? What else could I try?