I profiled this using a rather simple app and the Instruments tool.
I used the "System/IO" and the "Time Profiler" instruments.

More than 40 percent of time (for a not very complex QML app) is spent in:

QCoreTextFontDatabase::populateFontDatabase() (taking ~870 msec)
calling QCoreTextFontDatabase::populateFromDescriptor(__CTFontDescriptor const*)
calling TCollection::CopyMatchingFontDescriptors(__CFDictionary const*) const

and in:
QCoreTextFontDatabase::fallbacksForFamily(QString const&, QFont::Style, QFont::StyleHint, QChar::Script) const

Both of those functions are called at some point in initialization by QQuickTextEdit (I'm using TextEdit in QML).

From the "System/IO" profile tool I can see that all fonts in /System/Library/Fonts are opened and examined.

If I replace the TextEdit by TextInput it is still slow.
If I replace the TextEdit by Rectangle it is faster (the populateFontDatabase() still seems to be called, but maybe much less).