Uploaded image for project: 'Qt Quality Assurance Infrastructure'
  1. Qt Quality Assurance Infrastructure
  2. QTQAINFRA-5509

Categorize flaky/random fails in CI

    XMLWordPrintable

Details

    • Epic
    • Resolution: Unresolved
    • P2: Important
    • None
    • None
    • None
    • Categorize flaky/random fails in CI

    Description

      Currently in is unclear if a flakiness is due to "bad test case" or "inconsistent CI system". Categorizing would help identifying the improvement actions. Categorization would also make the follow-up easier, to see if improvement actions made any differences.

      Known reasons for a fail:

      • test fails
      • test fails because of test specific timeout
      • test crashes
      • test execution is stopped because no output in the log for 15 minutes
      • VM appears to be frozen
      • max test execution timeout reaches (max VM lifetime)

      example of crashes:

      https://testresults.qt.io/coin/api/log/qt/qtwebengine/30969ebdf0b030d2f34f3f405497211315510fab/MacOSMacOS_13arm64MacOSMacOS_13arm64Clangqtci-macos-13-arm64Sccache_TestOnly/99104407198c599dae20bfda131d512a61052790/forceRun/test_1690141264/log.txt.gz
      (not stored in in influx)

      Attachments

        Issue Links

          No reviews matched the request. Check your Options in the drop-down menu of this sections header.

          Activity

            People

              anwojcie Anna Wojciechowska
              jujokini Jukka Jokiniva
              Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

                Created:
                Updated:

                Gerrit Reviews

                  There are no open Gerrit changes