Uploaded image for project: 'Qt'
  1. Qt
  2. QTBUG-96353

Test run procedure and flaky detection

    XMLWordPrintable

Details

    • Bug
    • Status: Closed
    • P3: Somewhat important
    • Resolution: Fixed
    • 6.3
    • None
    • Build System: CMake
    • None

    Description

      Here are some differences I notice in the qmake (coin) vs cmake test run procedure, regarding failed tests:

      When a test fails then it re-runs up to 4 extra times, in order to decide if the failure was a flake (flaky test)

      • 2 failures used to be enough to fail the integration. Now with cmake it seems that after 2 failures the test runs again and passes the 3rd time and the integration continues. For example search for tst_QSocks5SocketEngine::simpleConnectToIMAP in this qmake run and in this cmake run.
      • We used to re-run only the test function that failed, for example tst_progname funcname; now cmake re-runs everything. It's not clear to me what should be done if on the 2nd run, a different function fails.

      Attachments

        Issue Links

          No reviews matched the request. Check your Options in the drop-down menu of this sections header.

          Activity

            People

              jimis Dimitrios Apostolou
              jimis Dimitrios Apostolou
              Jukka Jokiniva Jukka Jokiniva
              Votes:
              0 Vote for this issue
              Watchers:
              8 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Gerrit Reviews

                  There are no open Gerrit changes