In DirectWriteFontFileStream::ReadFileFragment() line 331, accessing  m_fontData.data() duplicates the font data for every font face. 
This will result in significant memory usage when loading large font files (e.g. loading SourceHanSans-Regular.ttc eats up extra ~150MB private bytes).

Adding a const qualifier to the m_fontData member should fix this issue:

class DirectWriteFontFileStream: public IDWriteFontFileStream
{
...
private:
    const QByteArray m_fontData;
    ULONG m_referenceCount;
};