QFileSystemModel shows hidden files after QDir::Hidden filter is set and unset again

If I set the filter of the QFileSystemModel using the QFileSystemModel::setFilter(...) function to QDir::Hidden and then unset it again, sometimes several hidden directories remain shown in the QListView that its connected to. After repeating, usually some different directories will be shown. Strangely, this only happens when I scroll down the list, never when I switch while at the beginning (in that case no hidden files remain shown).

The first screenshot is with the hidden filter on and scrollbar at the beginning. If I'd toggle hidden files like this, everything would be ok. Second screenshot shows the situation after I scrolled down a bit and toggled hidden files.

Due to the strange dependency on how far I've scrolled in the QListView, I marked the component as Maemo 5.

The application I'm developing is case file manager and, in case you would like, the source code is in git in its garage page: https://garage.maemo.org/plugins/ggit/browse.php/?p=case