Looks like I stumbled on an issue with masking inside QtWebsockets. It pop up during SSL protected WS connection between QtWebsocket & boost driven websocketpp.
My problem that websocketpp is dropping incoming connection because in the stream of masked WS frames it found frame without mask.
Inside QWebSocketPrivate::getFrameHeader() implementation it turns out addition of mask flags and fields depends on key value and not m_mustMask member variable:
...
if (maskingKey != 0)
byte |= 0x80;
...
if (maskingKey != 0)

{ const quint32 mask = qToBigEndian<quint32>(maskingKey); header.append(static_cast<const char *>(static_cast<const void *>(&mask)), sizeof(quint32)); }

...

Thus if generator (default one) can produce a zero key, possibility of rare unmasked frame inside a stream of masked frames occur. I don't really know if it is a big fault or not at all but websocketpp is definitely check for this kind of situation and treat it like error.
Fix looks trivial: to use m_mustMask instead of maskingKey to find out if masking is needed.