TLS Certificate chain validation fails with invalid purpose if the root CA contains a NetscapeCertType extension that does not contain server bit when schannel is used as TLS backend on windows

Subcomponent

TLS schannel backend

Symptom

We ran into a critical issue, after we updated our app from Qt 6.2 to Qt 6.5, thus it seems that deploying openssl 3 became a new requirement when the openssl TLS backend is used. Thus we unintendly switched our app to the schannel backend since we only deployed openssl 1.1.

But when using the schannel backend, at least one of our customers can no longer connect to some company internal services that use an internal CA for certificate signing. 

When using the openssl backend (after deploying opennssl 3), it just works as expected.

Cause

The certificate used by the TLS endpoint has a root CA at the end of its chain of trust that has the long deprecated netscapeCertType key extension (2.16.840.1.113730.1.1). (a good summary can be found here: https://github.com/opnsense/core/issues/2459). The value of this key extension inside the CA cert is correct, because it denotes its a CA and its used for signing and verification, but it has not the "server use" bit set.

It seems there is a extension in the TLS schannel backend implementation (that exists since ages like this) which will validate the netscapeCertType key extension and generate an invalid purpose error, but it will only check for the server bit, also for root CA.

The check function is in qsslsocket_schannel.cpp, line 300+, see the comment for the issue if it is supposed to be fixed:

/* lines 300+ */
bool netscapeWrongCertType(const QList<QSslCertificateExtension> &extensions, bool isClient)
{
    const auto netscapeIt = std::find_if(
            extensions.cbegin(), extensions.cend(),
            [](const QSslCertificateExtension &extension) {
                const auto netscapeCertType = QStringLiteral("2.16.840.1.113730.1.1");
                return extension.oid() == netscapeCertType;
            });
    if (netscapeIt != extensions.cend()) {
        const QByteArray netscapeCertTypeByte = netscapeIt->value().toByteArray();
        int netscapeCertType = 0;
        QDataStream dataStream(netscapeCertTypeByte);
        dataStream >> netscapeCertType;
        if (dataStream.status() != QDataStream::Status::Ok)
            return true;
// This check fails for root CA, for root CA, it should be checked that the type is 
// i.e. NETSCAPE_SSL_CA_CERT_TYPE

const int expectedPeerCertType = isClient ? NETSCAPE_SSL_SERVER_AUTH_CERT_TYPE : NETSCAPE_SSL_CLIENT_AUTH_CERT_TYPE;
        if ((netscapeCertType & expectedPeerCertType) == 0)
            return true;
    }
    return false;
}/* and in line 1959+ */
// While netscape shouldn't be relevant now it defined an extension which is
// still in use. Schannel does not check this automatically, so we do it here.
// It is used to differentiate between client and server certificates.
if (netscapeWrongCertType(extensions, isClient))
    element->TrustStatus.dwErrorStatus |= CERT_TRUST_IS_NOT_VALID_FOR_USAGE; 

Proposed solution

Remove the whole netscapeCertType related code to no longer overrule the tlschannel verification and to follow the implementation of openssl that simply ignores this key extension.

If for some reason I'm not aware of, this still needs to be supported, provide a way to turn it off, in this case i would also fix the issue to handle the rootCA correctly.

Reproduction 

I havent't found a web site that uses this keyExtension so i did not create an example application. Reproducing this would be easy though – just make sure the schannel backend is used on windows and connect to the TLS service that has this extension in its rootCA.