-
Bug
-
Resolution: Fixed
-
P3: Somewhat important
-
None
-
6.3
-
None
Here are some differences I notice in the qmake (coin) vs cmake test run procedure, regarding failed tests:
When a test fails then it re-runs up to 4 extra times, in order to decide if the failure was a flake (flaky test)
- 2 failures used to be enough to fail the integration. Now with cmake it seems that after 2 failures the test runs again and passes the 3rd time and the integration continues. For example search for tst_QSocks5SocketEngine::simpleConnectToIMAP in this qmake run and in this cmake run.
- We used to re-run only the test function that failed, for example tst_progname funcname; now cmake re-runs everything. It's not clear to me what should be done if on the 2nd run, a different function fails.
- is blocked by
-
-
- Closed
-
- relates to
-
QTBUG-96844 Rationalize logging in QtTestLib
-
- Closed
-
-
-
- Closed
-
- resulted from
-
-
- Closed
-
- resulted in
-
-
- Closed
-
- mentioned in
-
Page Loading...
| For Gerrit Dashboard: QTBUG-96353 | ||||||
|---|---|---|---|---|---|---|
| # | Subject | Branch | Project | Status | CR | V |
| 374555,8 | Update flaky test logic | master | qtqa/tqtc-coin-ci | Status: MERGED | +2 | 0 |
| 383444,27 | Introduce qt-testrunner.py | dev | qt/qtbase | Status: MERGED | +2 | 0 |
| 407393,2 | Cherry-pick qt-testrunner.py from dev branch | 6.3 | qt/qtbase | Status: MERGED | +2 | 0 |
| 407397,4 | Cherry-pick qt-testrunner.py from dev branch | tqtc/lts-6.2 | qt/tqtc-qtbase | Status: MERGED | -1 | 0 |