Details
-
Bug
-
Resolution: Incomplete
-
P2: Important
-
None
-
5.9.1, 5.9
-
Linux/Other display system
Description
We use QT on the service to render a lot of multi-language texts in parallel. We have a bunch of fonts installed on the system, so that we could render most of characters. As a result we have quite big font cache.
Recently we added even more fonts and increased amount of text rendering requests. We noticed that quite often our service is crashing, and when we check core dump, the stack is corrupted:
@ 0000000000000000 (unknown) @ 0000000000000000 (unknown)
So it looked like we were writing somewhere where we shouldn't. We turned on Address Sanitiser, and found this problem. It happens when we check if we can render 1 single character:
1. First, we allocated font engine with fallback families:
previously allocated by thread T69 (Text Renderer) here:
#0 0x8168ef7 in __interceptor_malloc (/packages/service+0x8168ef7)
#1 0x7a06341 in QArrayData::allocate(unsigned long, unsigned long, unsigned long, QFlags<QArrayData::AllocationOption>) /home/qt/5.9.1/src/qt-everywhere-opensource-src-5.9.1/qtbase/src/corelib/tools/qarraydata.cpp:118:60
#2 0x77ce661 in QTypedArrayData<QFontEngine*>::allocate(unsigned long, QFlags<QArrayData::AllocationOption>) /home/qt/5.9.1/src/qt-everywhere-opensource-src-5.9.1/qtbase/src/corelib/tools/qarraydata.h:222:67
#3 0x77ce661 in QVector<QFontEngine*>::reallocData(int, int, QFlags<QArrayData::AllocationOption>) /home/qt/5.9.1/src/qt-everywhere-opensource-src-5.9.1/qtbase/src/corelib/tools/qvector.h:544:35
#4 0x77cc5a8 in QVector<QFontEngine*>::resize(int) /home/qt/5.9.1/src/qt-everywhere-opensource-src-5.9.1/qtbase/src/corelib/tools/qvector.h:423:5
#5 0x77cc5a8 in QFontEngineMulti::setFallbackFamiliesList(QStringList const&) /home/qt/5.9.1/src/qt-everywhere-opensource-src-5.9.1/qtbase/src/gui/text/qfontengine.cpp:1823:25
#6 0x77cca9d in QFontEngineMulti::ensureFallbackFamiliesQueried() /home/qt/5.9.1/src/qt-everywhere-opensource-src-5.9.1/qtbase/src/gui/text/qfontengine.cpp:1807:28
#7 0x77cca9d in QFontEngineMulti::glyphIndex(unsigned int) const /home/qt/5.9.1/src/qt-everywhere-opensource-src-5.9.1/qtbase/src/gui/text/qfontengine.cpp:1869:80
#8 0x77cca9d in QFontEngineMulti::glyphIndex(unsigned int) const /home/qt/5.9.1/src/qt-everywhere-opensource-src-5.9.1/qtbase/src/gui/text/qfontengine.cpp:1860:9
#9 0x77d20ff in QFontEngine::canRender(unsigned int) const /home/qt/5.9.1/src/qt-everywhere-opensource-src-5.9.1/qtbase/src/gui/text/qfontengine_p.h:229:63
#10 0x77d20ff in QFontMetrics::inFontUcs4(unsigned int) const /home/qt/5.9.1/src/qt-everywhere-opensource-src-5.9.1/qtbase/src/gui/text/qfontmetrics.cpp:444:29
2. Then, when we checked if we can render this character with other fonts (fallbacks), we loaded another engine. But we also had to decrease cache. When we decreased cache, we deleted some QFontEngineMulti (freed m_engines):
0x6190012e7698 is located 24 bytes inside of 1024-byte region [0x6190012e7680,0x6190012e7a80) freed by thread T69 (Text Renderer) here: #0 0x8168ce8 in __interceptor_free (/packages/service+0x8168ce8) #1 0x77cb896 in QTypedArrayData<QFontEngine*>::deallocate(QArrayData*) /home/qt/5.9.1/src/qt-everywhere-opensource-src-5.9.1/qtbase/src/corelib/tools/qarraydata.h:237:31 #2 0x77cb896 in QVector<QFontEngine*>::freeData(QTypedArrayData<QFontEngine*>*) /home/qt/5.9.1/src/qt-everywhere-opensource-src-5.9.1/qtbase/src/corelib/tools/qvector.h:529:21 #3 0x77cb896 in _ZN7QVectorIP11QFontEngineED4Ev /home/qt/5.9.1/src/qt-everywhere-opensource-src-5.9.1/qtbase/src/corelib/tools/qvector.h:75:46 #4 0x77cb896 in QFontEngineMulti::~QFontEngineMulti() /home/qt/5.9.1/src/qt-everywhere-opensource-src-5.9.1/qtbase/src/gui/text/qfontengine.cpp:1790:37 #5 0x76c7510 in QFontEngineMultiFontConfig::~QFontEngineMultiFontConfig() /home/qt/5.9.1/src/qt-everywhere-opensource-src-5.9.1/qtbase/src/platformsupport/fontdatabases/fontconfig/qfontenginemultifontconfig.cpp:57:1 #6 0x77bddf5 in QFontCache::decreaseCache() /home/qt/5.9.1/src/qt-everywhere-opensource-src-5.9.1/qtbase/src/gui/text/qfont.cpp:3077:20 #7 0x77bed97 in QFontCache::insertEngine(QFontCache::Key const&, QFontEngine*, bool) /home/qt/5.9.1/src/qt-everywhere-opensource-src-5.9.1/qtbase/src/gui/text/qfont.cpp:2872:22 #8 0x77de770 in loadSingleEngine /home/qt/5.9.1/src/qt-everywhere-opensource-src-5.9.1/qtbase/src/gui/text/qfontdatabase.cpp:982:36 #9 0x77de770 in loadEngine(int, QFontDef const&, QtFontFamily*, QtFontFoundry*, QtFontStyle*, QtFontSize*) (.isra.117) /home/qt/5.9.1/src/qt-everywhere-opensource-src-5.9.1/qtbase/src/gui/text/qfontdatabase.cpp:1000:43 #10 0x77debc7 in QFontDatabase::findFont(QFontDef const&, int) /home/qt/5.9.1/src/qt-everywhere-opensource-src-5.9.1/qtbase/src/gui/text/qfontdatabase.cpp:2686:28 #11 0x77cb993 in QFontEngineMulti::loadEngine(int) /home/qt/5.9.1/src/qt-everywhere-opensource-src-5.9.1/qtbase/src/gui/text/qfontengine.cpp:1850:54 #12 0x77cc7c6 in QFontEngineMulti::ensureEngineAt(int) /home/qt/5.9.1/src/qt-everywhere-opensource-src-5.9.1/qtbase/src/gui/text/qfontengine.cpp:1835:41 #13 0x77ccb35 in QFontEngineMulti::glyphIndex(unsigned int) const /home/qt/5.9.1/src/qt-everywhere-opensource-src-5.9.1/qtbase/src/gui/text/qfontengine.cpp:1875:69 #14 0x77ccb35 in QFontEngineMulti::glyphIndex(unsigned int) const /home/qt/5.9.1/src/qt-everywhere-opensource-src-5.9.1/qtbase/src/gui/text/qfontengine.cpp:1860:9 #15 0x77d20ff in QFontEngine::canRender(unsigned int) const /home/qt/5.9.1/src/qt-everywhere-opensource-src-5.9.1/qtbase/src/gui/text/qfontengine_p.h:229:63 #16 0x77d20ff in QFontMetrics::inFontUcs4(unsigned int) const /home/qt/5.9.1/src/qt-everywhere-opensource-src-5.9.1/qtbase/src/gui/text/qfontmetrics.cpp:444:29
3. Now, in the same frame 0x77ccb35 of QFontEngineMulti::glyphIndex, we check other engine. We load it, and put into vector: m_engines[at] = engine; There QVector is resized, and we try to access data that was already freed. Which causes this problem:
==221==ERROR: AddressSanitizer: heap-use-after-free on address 0x6190012e7698 at pc 0x0000080caf26 bp 0x7efbd44c9010 sp 0x7efbd44c87c0 READ of size 840 at 0x6190012e7698 thread T69 (Text Renderer) SCARINESS: 54 (multi-byte-read-heap-use-after-free) #0 0x80caf25 in __interceptor_memcpy.part.45 (/packages/service+0x80caf25) #1 0x77ce699 in memcpy /home/glibc/2.26/platform007/f259413/include/bits/string_fortified.h:34:33 #2 0x77ce699 in QVector<QFontEngine*>::reallocData(int, int, QFlags<QArrayData::AllocationOption>) /home/qt/5.9.1/src/qt-everywhere-opensource-src-5.9.1/qtbase/src/corelib/tools/qvector.h:563:29 #3 0x77cc850 in QVector<QFontEngine*>::detach() /home/qt/5.9.1/src/qt-everywhere-opensource-src-5.9.1/qtbase/src/corelib/tools/qvector.h:391:13 #4 0x77cc850 in QVector<QFontEngine*>::data() /home/qt/5.9.1/src/qt-everywhere-opensource-src-5.9.1/qtbase/src/corelib/tools/qvector.h:129:24 #5 0x77cc850 in QVector<QFontEngine*>::operator[](int) /home/qt/5.9.1/src/qt-everywhere-opensource-src-5.9.1/qtbase/src/corelib/tools/qvector.h:439:16 #6 0x77cc850 in QFontEngineMulti::ensureEngineAt(int) /home/qt/5.9.1/src/qt-everywhere-opensource-src-5.9.1/qtbase/src/gui/text/qfontengine.cpp:1840:21 #7 0x77ccb35 in QFontEngineMulti::glyphIndex(unsigned int) const /home/qt/5.9.1/src/qt-everywhere-opensource-src-5.9.1/qtbase/src/gui/text/qfontengine.cpp:1875:69 #8 0x77ccb35 in QFontEngineMulti::glyphIndex(unsigned int) const /home/qt/5.9.1/src/qt-everywhere-opensource-src-5.9.1/qtbase/src/gui/text/qfontengine.cpp:1860:9 #9 0x77d20ff in QFontEngine::canRender(unsigned int) const /home/qt/5.9.1/src/qt-everywhere-opensource-src-5.9.1/qtbase/src/gui/text/qfontengine_p.h:229:63 #10 0x77d20ff in QFontMetrics::inFontUcs4(unsigned int) const /home/qt/5.9.1/src/qt-everywhere-opensource-src-5.9.1/qtbase/src/gui/text/qfontmetrics.cpp:444:29
We think this is dangerous, because later m_engines can be reused, and we can access the memory we shouldn't access. This can lead to a crash with corruption of the stack.
When I research this with GDB, right before the address sanitiser dies,:
(gdb) break __sanitizer::Die
(gdb) run
we can see that in the stack frame 0x77ccb35, "this" pointer (to multi engine) is good, but m_engines is already fully erased:
(gdb) print this $1 = (const QFontEngineMulti * const) 0x6611000c09580 (gdb) print this->m_engines $2 = {d = 0xbebebebebebebebe} (gdb) print *(this->m_engines.d) Cannot access memory at address 0xbebebebebebebeb
I'm not sure yet if we deleted the same engine, or "this" engine shared internal data with some other engine that we deleted.
Repro
I'm not sure if there's any easy way to reproduce this. This issue is flaky. On our service (single host) we serve ~500qps for text rendering requests in parallel threads. This crash happens every ~10min - ~2hours. Our RAM usage is at the level ~10-15 GB. We expect that most of it is taken by qt font cache.
We have 151 fonts installed on the system (almost all Noto and Roboto + a few system's default).
On our side, the caller code looks like this. Each thread gets QString& text as an input, and executes this code:
bool renderer::getUnSupportedCharacter(const QString& text) { bool supported = false; for (uint char32 : text.toUcs4()) { // ...skipped some unrelated logic QFontDatabase db; QStringList const families = db.families(); for (int i = 0; i < families.size(); i++) { QFont font(families.at(i), 10, QFont::Normal); QFontMetrics qFontMetrics(font); if (qFontMetrics.inFontUcs4(char32)) { supported = true; break; } } } return supported; }
Other versions of QT
I checked the code of QT12 and didn't found anything that could impacted this logic in qfont, qfontengine, qfontdatabase, etc. So it might be a problem in other versions of QT as well.
Questions
1. Do you know if there's an easy way to fix this?
2. Is it possible to tune some settings of QFontCache to mitigate the problem or make crashes less frequent? I noticed we can change size/timeout.
3. Can you please provide more information about how QFontCache works? I noticed there's a separate timer, that can be used to decrease font cache. Is it safe to us it when we have multiple threads using fonts?
4. Do you need any additional data to debug this?