Very slow QFileIconProvider since Qt6

In a PyQt program of mine, I noticed a performance regression when porting from Qt5 to Qt6. Though I can't test with a C++ equivalent, I believe it's unrelated to the use of Python language.
I was able to narrow down to a minimal example like this:

import os
from PyQt6.QtWidgets
import QApplication
from PyQt6.QtGui import QFileSystemModel
from PyQt6.QtWidgets import QTreeView

app = QApplication([])
tv = QTreeView()
mdl = QFileSystemModel()
tv.setModel(mdl)
qidx = mdl.setRootPath(os.environ["HOME"])
tv.setRootIndex(qidx)
tv.show()
app.exec()

With Qt5, when running the program, the app takes less than a second to display entirely and be responsive.
With Qt6 though, the app is quick to show an empty window but I have to wait more than 10 seconds for it to be responsive and its contents to be effectively shown.

When running strace on the app and quitting it as soon as possible, I can see strace spits ~300 times more log lines when running under Qt6! With Qt6, many lines are repeated hundreds of times, like:

access("/home/x/.local/share/icons/hicolor/96x96@2/emotes/image.png", F_OK) = -1 ENOENT

For example, this random line appeared more than 600 times before I killed the app since it was still not responsive (it's a little slower under strace, logs were piling up, so I finally killed it)

Another app seems to have a similar issue: https://github.com/qutebrowser/qutebrowser/pull/7925