Uploaded image for project: 'Qt'
  1. Qt
  2. QTBUG-96353

Test run procedure and flaky detection

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Fixed
    • Icon: P3: Somewhat important P3: Somewhat important
    • None
    • 6.3
    • Build System: CMake
    • None

      Here are some differences I notice in the qmake (coin) vs cmake test run procedure, regarding failed tests:

      When a test fails then it re-runs up to 4 extra times, in order to decide if the failure was a flake (flaky test)

      • 2 failures used to be enough to fail the integration. Now with cmake it seems that after 2 failures the test runs again and passes the 3rd time and the integration continues. For example search for tst_QSocks5SocketEngine::simpleConnectToIMAP in this qmake run and in this cmake run.
      • We used to re-run only the test function that failed, for example tst_progname funcname; now cmake re-runs everything. It's not clear to me what should be done if on the 2nd run, a different function fails.

        No reviews matched the request. Check your Options in the drop-down menu of this sections header.

            jimis Dimitrios Apostolou
            jimis Dimitrios Apostolou
            Jukka Jokiniva Jukka Jokiniva
            Votes:
            0 Vote for this issue
            Watchers:
            8 Start watching this issue

              Created:
              Updated:
              Resolved:

                There are no open Gerrit changes