Problem description:
With limited size of the read buffer, when buffer fills up hardware handshake is triggered. RTS(CTS) goes low properly. When application is ready to receive data and when all data is read from internal buffers operating system buffer is not read (no ReadFile call) so it stalls. RTS(CTS) stays low. Future calls to any read commands or waitForReadyRead do nothing. No further reads are possible.

Steps to reproduce:
Set limited read buffer size (eg. 1024) and enable hardware handshake. Allow read buffer to fill until hardware handshake is triggered. No further read is possible.

Analysis:
Initial debugging reveals that in this condition ReadFile is never called again, so hardware handshake cannot be unblocked.

See simple demo attached (tested using com0com & Eltima virtual serial ports)

Simple hack by adding a timer in startAsyncRead() helps, but it's not a proper solution and I'm not best at QSerialPort internals to fix it.

 
bool QSerialPortPrivate::_q_startAsyncRead()
{
    return startAsyncRead();
}

bool QSerialPortPrivate::startAsyncRead()
{
    Q_Q(QSerialPort);

    DWORD bytesToRead = policy == QSerialPort::IgnorePolicy ? ReadChunkSize : 1;

    if (readBufferMaxSize && bytesToRead > (readBufferMaxSize - readBuffer.size())) {
        bytesToRead = readBufferMaxSize - readBuffer.size();
        if (bytesToRead == 0) {
            // Buffer is full. User must read data from the buffer
            // before we can read more from the port.
            QTimer::singleShot(100,q,SLOT(_q_startAsyncRead()));
            return false;
        }
    }
[...]