Qt can't draw some glyphs available in installed fonts if there are too many fallback fonts

I have two systems that are practically equal, but in one of them some unicode characters are painted correctly in Qt applications (Konsole, Kate, Qt file dialogs...) while in the other those characters are shown as an empty square.

I found a quick way to reproduce it is running python3 -c "print('\uff0f')" in a konsole window which showed the character in one system and didn't in the other, while it worked in all other terminals like gnome-terminal or xterm.

Then, I did a simple testcase that shows the problem and started debugging the font related Qt classes:

int main(int argc, char *argv[])
  {
     QApplication app(argc, argv);
     QLabel *label = new QLabel(nullptr);
     label->setText("／");  /* Note that is a 0xFF0F character, not a regular slash */
     label->resize(500,500);
     label->show();
     return app.exec();
 }

QT += widgets
CONFIG += debug
SOURCES = main.cpp

I checked that in the system where the character wasn't correctly painted I had way more fonts installed than in the other, so I tried moving away some fonts from their proper location and suddenly the problem was fixed. Putting them back brought the problem back, but I noticed that the font that was moved away didn't actually matter, so it looked like the problem was related to the number of fonts, instead of to the fonts themselves. I reduced the way to reproduce/fix the problem to moving a font to ~/.fonts to reproduce the problem and moving the font to some other directory to make Qt work properly.

in QFontEngineMulti::setFallbackFamilies I added a loop to print the whole m_fallbackFamilies list of fallback fonts that were being used as well as some other debugging information to check from what font the character was actually used.
The font where the character is defined is Droid Sans Fallback, and the contents of m_fallbackFamilies when it doesn't work is (just some elements from the middle of the list):

...
254 )  "Droid Arabic Kufi"
255 )  "Droid Arabic Naskh"
256 )  "Droid Sans Fallback"
257 )  "Droid Sans Thai"
...

While moving the font away to "fix the problem" gave:

...
252 )  "Droid Arabic Kufi"
253 )  "Droid Arabic Naskh"
254 )  "Droid Sans Fallback"
255 )  "Droid Sans Thai"
...

I then noticed that the code in QFontEngineMulti uses the highest 8 bits of glyph_t values to store the engine being used, so it's limited to using the first 255 fonts as fallback and if the fallback font that contains the character is beyond that number, then it simply doesn't work.

In order to test this, note that openSUSE Tumbleweed/Leap has a package called free-ttf-fonts which installs 375 truetype fonts. Just installing that package with zypper in free-ttf-fonts makes the problem reproducible.

I implemented 4 different solutions (I'll attach the patches here), which I'll explain next. But first, please note that most patches are not ready to be applied (which is the reason this is a bug report and not a submit to gerrit) although all of them work properly and fix the problem. I wanted to submit them with debug code so it's easier for you to try and see the debug output and also to establish a discussion and continue working on the preferred one, instead of losing time with all approaches to only use one of them at the end.

In order to prove the problem was what I thought, I made a patch fix-fonts-using-10-bits-for-engine.patch that makes Qt use the highest 10 bits instead of the highest 8 bits for the engine, thus increasing the font limit to 1024 fonts before Qt stops looking for a fallback. As expected, that fixed the problem. Or better said, just hides the problem for a while. On one hand, I don't know what's the highest glyph value that could be used, so maybe 22 bits is not enough (I guess if glyph_t represents an index in a font table that's ok, but still, it's a quint32, so it could have a value in the full range of the type). On the other hand, that just moves the limit from 256 fonts to 1024. I don't know for sure, but I guess designers might have such a number of fonts installed in their systems and it seems like imposing a limit so I tried a second approach.

In fix-fonts-using-qglyph-struct.patch I replaced glyph_t with a struct I called QGlyph containing an engine_t (another quint32) and glyph_t variable members and then I renamed glyph_t to glyph_t to make the compiler to give errors when glyph_t is used and force me to read the code and and fix it by either replacing it with QGlyph or with glypht as appropriate. Of course, glyph_t should then be replaced back to glyph_t but I left that for later since I guessed someone looking at the patches might want to try them in their systems too. This fix is far larger than the previous one, changes far more code (I took care not to modify any public interface, though) and uses a bit more memory, but is future-proof.

About the memory used, which is what mostly worried me while I was doing these changes, I checked that opening a large file like src/widgets/graphicsview/qgraphicsitem.cpp (379KB) with kate using a regular Qt uses 412024 KiB (vsize) / 99996 KiB (rss) while using a patched Qt with QGlyph structs for glyphs uses 415956 KiB (vsize) / 102652 KiB (rss) and in the case of the simple testcase above which just shows a QLabel with one character that has to fallback, the memory sizes are 374648 KiB / 58760 KiB for the unpatched Qt and 378220 KiB / 64288 KiB when using the patched Qt . As you see, the increase of memory usage seems very similar (always around 4 KB), which lets me think it probably won't increase much more for larger applications (I also tested dolphin, with similar values).

Still, I thought even if this patch works and should be the right approach IMHO, this patch is really large and changes too much code which makes it quite risky to apply (leaving aside that someone would need to also adapt code for other platforms that I can't test and would need to be modified too). So I looked for an alternative solution.

After doing some tests, I found the 0xFF0F character is included only in the following fonts in my system:

So fix-fonts-using-special-treatment-fallback-fonts.patch is a much simpler patch (and should be even smaller when debug code is removed) and is based on the knowledge that those fonts are known to have extra glyphs. So in QFontEngineMulti:: setFallbackFamiliesList this patch makes sure that if any of those fonts appear beyond the 255 border, they're moved before the limit in the same order they're found. This ensures that if those fonts are in the fallback list, they're found when iterating up to the 255th engine.

The problem with this approach is that it introduces a new list of "Specially treated fonts" which should be platform-specific and should be done in a more generic way than I did in my patch.

Which leads to my preferred solution, which I implemented in force-fallback-fonts-to-be-reachable.patch .

I noticed that once the fallback engine that contains a glyph we need is found beyond the 255 limit, most engines below the limit are still unused so... why not moving the engine below the limit before it's used? Which is exactly what this patch does. The engine we need is swapped with an unused engine at the end of the range (note we cannot move engines around, since that would modify the indexes of engines that might be in use, and that would break glyph indexes already stored somewhere else).

Note that this might change the order of fonts. For example, if a glyph we need is found in the font engine at position 270, it would be moved to 255 and from that point on, it would have preference over the rest of fonts beyond 255. If still there's a glyph it doesn't contain and is contained in a font in position 280, it would be moved to 254 and from that point on, this would have preference even over 270. Note that the already stored glyphs from 255/270 would still be used from that font, but when if we search again for a font containing such glyphs now the font in 254/280 would be used (if it contains it, of course). I can't think of any satisfying solution to this except noting that the order of fallbackFamilies fonts is only suggestive and it might be reordered at any point. Maybe just selecting the substitute position from the beginning of the list, in order would be better, since this would at least guarantee that once a glyph is taken from a substitute font, it will always be used from it, and also would minimize the number of fallback fonts, since it wouldn't try new fonts if any of the already used fallback fonts contains the new glyph being searched for. If this is preferred (and I'm starting to like the idea while I write this) I can modify the patch to do that.

A nice benefit of this patch is that it lets 255 possible fallback fonts to be used before running out of space, which at least makes glyphs visible, it doesn't establish a list of special-handled fonts and should work well in most situations while being a quite small patch (in fact, this is the only one without debug code, mostly ready to be applied save minor changes).

So, would it be ok to submit force-fallback-fonts-to-be-reachable.patch as it is? Should I change it to select the new position for fonts from the beginning of the engines list in order instead of from the end backwards? Or would you prefer any of the other solutions?