Uploaded image for project: 'Qt'
  1. Qt
  2. QTBUG-77824

QML Engine deadlocks/freezes on the long term

    XMLWordPrintable

Details

    • Linux/X11, Linux/Other display system

    Description

      This is a long story of a multi-year research. I apologize for the long story but it needs a lot of context.

      The backstory

      Back then at 5.6.x series we implemented Particle System simulations for weather applications, basically wind simulators (there is even a quick talk on SF QtWS about it) but somewhere we migrated to 5.9 series. We immediately discovered lockups on our applications after some hours, as a lot of code changed on our part we basically ignored the lockups, reverted to 5.6 and went on. After that, we read about some changes in core QML engine and some optimizations, we suspected some changes to Particle System code were made (we were wrong). After 5.10 series were a bit worse, we started research on what was going on, ...

      The research

      We started by running or code on 5.6, 5.9 and 5.10 series, then we wen into testing 5.11 and 5.12, every single Qt version was affected after 5.6 (beware we never tested 5.7).

      The testbeds, OS and Qt version Matrix

      Back then testing was long and tedious, it involved running our weather simulator and letting it run for hours, it froze at about 6 to 10 hours depending on Qt version. 

      Test hardware:

      1. HP All In One, 8Gb RAM, SSD. Intel Graphics
      2. Dell AIO, 8 Gb RAM SATA nVidia GPU
      3. nVidia Jetson TX2
      4. Intel core i5 assembled computer, 16 Gb RAM, Sata hard disk, powerful nVidia GPU

      Test OS

      1. Kubuntu 16.04 64 bit
      2. Kubuntu 18.04 64 bit
      3. KDE Neon (2019) 64 bit
      4. Yocto image (for Jetson)

      We ran tests on every piece of hardware on every OS combination, over each public Qt version since: 5.6 except 5.7 and 5.8. Last one tested is Qt 5.13. We even reinstalled clean OS images to avoid noise. After some months we just found everything failed.

      In-depth research to discard particle-related code

      We had some time at mid 2018, so we dedicated some time to scanning particle related code which might have been changed, besides a one-liner or two, we found nothing changed on memory management or deep code changes. Particles were not the cuprit.

      In-depth research to minimize reproducible bug

      As our codebase was big, we started trimming down our code to find the minimal reproducible bug. We settled on a single custom affector & emitter and feedback-GLSL shader. We ran the whole test matrix again! all failed.

      Then we removed custom emitter, custom affector and feedback shader trying to find a really minimal code. We ran few tests (not the whole matrix) and we found the minimal code.

      We were still reluctant to send bug as we needed a culprit to blame, it was something on QML Engine but we had no idea what.

      Anyway, we tested 5.13, it failed too.

      Then we found comments on colleagues that many dynamic elements create freezes, they called gc() to make it behave. We repeated some tests using gc() but now definitely think is something about QML engine dynamic object creation & deletion. Particles just make it more obvious.

      The Bug

      Something inside QML engine changed after 5.6 Series and before 5.9 series. After talking on IRC channel I was told it might be a deadlock as it does not consume 100% CPU nor 100% RAM, I just found it uses 100% of a single CPU core when it freezes. It just freezes after a nearly identical amounts of time, quite different for every Qt version.

      We think this will affect every QML application beyond Particles specially on long running process that generate dynamic elements.

      Reproduce the bug

      We crafted this code to reproduce the behavior, but be aware it requires a lot of patience, as it will take hours to freeze.

      https://github.com/desert/particleBenchmarkTest

      We have tested reproducibility even on Qt Creator Debug mode 

      (code has a comment about garbage collector, but it will freeze with or without it)

       

       

       

       

       

       

      Attachments

        1. GDB_thread-apply-all-bt.txt
          6 kB
          Ariel Molina R.
        2. Screenshot_20190826_112117.png
          321 kB
          Ariel Molina R.
        3. Screenshot_20190826_112126.png
          243 kB
          Ariel Molina R.
        4. Screenshot_20190826_112131.png
          257 kB
          Ariel Molina R.
        5. Screenshot_20190826_112145.png
          259 kB
          Ariel Molina R.
        6. Screenshot_20190826_112149.png
          258 kB
          Ariel Molina R.

        Issue Links

          No reviews matched the request. Check your Options in the drop-down menu of this sections header.

          Activity

            People

              fabiankosmale Fabian Kosmale
              desert Ariel Molina R.
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Gerrit Reviews

                  There are no open Gerrit changes