Uploaded image for project: 'Qt'
  1. Qt
  2. QTBUG-96353

Test run procedure and flaky detection

    XMLWordPrintable

Details

    • Bug
    • Resolution: Fixed
    • P3: Somewhat important
    • None
    • 6.3
    • Build System: CMake
    • None

    Description

      Here are some differences I notice in the qmake (coin) vs cmake test run procedure, regarding failed tests:

      When a test fails then it re-runs up to 4 extra times, in order to decide if the failure was a flake (flaky test)

      • 2 failures used to be enough to fail the integration. Now with cmake it seems that after 2 failures the test runs again and passes the 3rd time and the integration continues. For example search for tst_QSocks5SocketEngine::simpleConnectToIMAP in this qmake run and in this cmake run.
      • We used to re-run only the test function that failed, for example tst_progname funcname; now cmake re-runs everything. It's not clear to me what should be done if on the 2nd run, a different function fails.

      Attachments

        Issue Links

          is blocked by

          Bug - A problem which impairs or prevents the functions of the product. QTQAINFRA-4681 All tests are super flaky on Android

          • P1: Critical - Urgent and Important, will STOP the release if matched with set FixVersion field. This includes regressions from the last version that are not edge cases; Data loss; Build issues; All but the most unlikely crashes/lockups/hanging; Serious usability issues and embarrassing bugs & Bugs critical to a deliverable. This is the highest possible priority for requirements.
          • Closed
          relates to

          Task - A task that needs to be done. QTBUG-96844 Rationalize logging in QtTestLib

          • P2: Important - Urgent, should be fixed, but will not stop the release.
          • Closed

          Bug - A problem which impairs or prevents the functions of the product. COIN-728 Logging categories no longer enabled when re-running failed tests

          • P2: Important - Urgent, should be fixed, but will not stop the release.
          • Closed
          resulted from

          Bug - A problem which impairs or prevents the functions of the product. QTBUG-96345 tst_QSocks5SocketEngine::simpleConnectToIMAP() is flaky on Ubuntu

          • P3: Somewhat important - Should be fixed, but doesn't affect the release in any way.
          • Closed
          resulted in

          Bug - A problem which impairs or prevents the functions of the product. QTQAINFRA-4687 Android test results are not written to influx

          • P2: Important - Urgent, should be fixed, but will not stop the release.
          • Closed
          mentioned in

          Page Loading...

          No reviews matched the request. Check your Options in the drop-down menu of this sections header.

          Activity

            People

              jimis Dimitrios Apostolou
              jimis Dimitrios Apostolou
              Jukka Jokiniva Jukka Jokiniva
              Votes:
              0 Vote for this issue
              Watchers:
              8 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Gerrit Reviews

                  There are no open Gerrit changes