Uploaded image for project: 'Qt Quality Assurance Infrastructure'
  1. Qt Quality Assurance Infrastructure
  2. QTQAINFRA-5509

Categorize flaky/random fails in CI

XMLWordPrintable

    • Icon: Epic Epic
    • Resolution: Unresolved
    • Icon: P2: Important P2: Important
    • None
    • None
    • CI / Test infrastructure
    • None
    • Categorize flaky/random fails in CI

      Currently in is unclear if a flakiness is due to "bad test case" or "inconsistent CI system". Categorizing would help identifying the improvement actions. Categorization would also make the follow-up easier, to see if improvement actions made any differences.

      Known reasons for a fail:

      • test fails
      • test fails because of test specific timeout
      • test crashes
      • test execution is stopped because no output in the log for 15 minutes
      • VM appears to be frozen
      • max test execution timeout reaches (max VM lifetime)

      example of crashes:

      https://testresults.qt.io/coin/api/log/qt/qtwebengine/30969ebdf0b030d2f34f3f405497211315510fab/MacOSMacOS_13arm64MacOSMacOS_13arm64Clangqtci-macos-13-arm64Sccache_TestOnly/99104407198c599dae20bfda131d512a61052790/forceRun/test_1690141264/log.txt.gz
      (not stored in in influx)

        No reviews matched the request. Check your Options in the drop-down menu of this sections header.

            anwojcie Anna Wojciechowska
            jujokini Jukka Jokiniva
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

              Created:
              Updated:

                There are no open Gerrit changes