QML Engine deadlocks/freezes on the long term

This is a long story of a multi-year research. I apologize for the long story but it needs a lot of context.

The backstory

Back then at 5.6.x series we implemented Particle System simulations for weather applications, basically wind simulators (there is even a quick talk on SF QtWS about it) but somewhere we migrated to 5.9 series. We immediately discovered lockups on our applications after some hours, as a lot of code changed on our part we basically ignored the lockups, reverted to 5.6 and went on. After that, we read about some changes in core QML engine and some optimizations, we suspected some changes to Particle System code were made (we were wrong). After 5.10 series were a bit worse, we started research on what was going on, ...

The research

We started by running or code on 5.6, 5.9 and 5.10 series, then we wen into testing 5.11 and 5.12, every single Qt version was affected after 5.6 (beware we never tested 5.7).

The testbeds, OS and Qt version Matrix

Back then testing was long and tedious, it involved running our weather simulator and letting it run for hours, it froze at about 6 to 10 hours depending on Qt version. 

Test hardware:

1. HP All In One, 8Gb RAM, SSD. Intel Graphics
2. Dell AIO, 8 Gb RAM SATA nVidia GPU
3. nVidia Jetson TX2
4. Intel core i5 assembled computer, 16 Gb RAM, Sata hard disk, powerful nVidia GPU

Test OS

1. Kubuntu 16.04 64 bit
2. Kubuntu 18.04 64 bit
3. KDE Neon (2019) 64 bit
4. Yocto image (for Jetson)

We ran tests on every piece of hardware on every OS combination, over each public Qt version since: 5.6 except 5.7 and 5.8. Last one tested is Qt 5.13. We even reinstalled clean OS images to avoid noise. After some months we just found everything failed.

In-depth research to discard particle-related code

We had some time at mid 2018, so we dedicated some time to scanning particle related code which might have been changed, besides a one-liner or two, we found nothing changed on memory management or deep code changes. Particles were not the cuprit.

In-depth research to minimize reproducible bug

As our codebase was big, we started trimming down our code to find the minimal reproducible bug. We settled on a single custom affector & emitter and feedback-GLSL shader. We ran the whole test matrix again! all failed.

Then we removed custom emitter, custom affector and feedback shader trying to find a really minimal code. We ran few tests (not the whole matrix) and we found the minimal code.

We were still reluctant to send bug as we needed a culprit to blame, it was something on QML Engine but we had no idea what.

Anyway, we tested 5.13, it failed too.

Then we found comments on colleagues that many dynamic elements create freezes, they called gc() to make it behave. We repeated some tests using gc() but now definitely think is something about QML engine dynamic object creation & deletion. Particles just make it more obvious.

The Bug

Something inside QML engine changed after 5.6 Series and before 5.9 series. After talking on IRC channel I was told it might be a deadlock as it does not consume 100% CPU nor 100% RAM, I just found it uses 100% of a single CPU core when it freezes. It just freezes after a nearly identical amounts of time, quite different for every Qt version.

We think this will affect every QML application beyond Particles specially on long running process that generate dynamic elements.

Reproduce the bug

We crafted this code to reproduce the behavior, but be aware it requires a lot of patience, as it will take hours to freeze.

https://github.com/desert/particleBenchmarkTest

We have tested reproducibility even on Qt Creator Debug mode 

(code has a comment about garbage collector, but it will freeze with or without it)