Details
-
Bug
-
Resolution: Cannot Reproduce
-
Not Evaluated
-
None
-
5.0.0
-
None
Description
This autotest is flaky on Windows, sometimes giving different results on consecutive runs.
Out of a sample of 500 build logs, I found 45 logs where the test had been run twice consecutively with different results. Here's a few of the most recent:
- http://testresults.qt-project.org/ci/QtBase_master_Integration/build_02982/win32-msvc2010_Windows_7/log.txt.gz
- http://testresults.qt-project.org/ci/QtBase_master_Integration/build_02980/win32-msvc2010_Windows_7/log.txt.gz
- http://testresults.qt-project.org/ci/QtBase_master_Integration/build_02972/win32-msvc2010_Windows_7/log.txt.gz
The test log is:
6/1/12 3:53:17 AM EST: QtQA::App::TestRunner: begin [release\tst_qtoolbutton.exe] [-silent] [-o] [C:\test\recipes\187514714\base\_artifacts\test-logs\tst_qtoolbutton.exe-testresults-00.xml,xml] [-o] [-,txt] 6/1/12 3:53:17 AM EST: Testing tst_QToolButton 6/1/12 3:53:22 AM EST: FAIL! : tst_QToolButton::task176137_autoRepeatOfAction() 'qAbs( (expected - repeatSpy.count()) / expected) < 0.1' returned FALSE. () 6/1/12 3:53:22 AM EST: tst_qtoolbutton.cpp(222) : failure location 6/1/12 3:53:22 AM EST: Totals: 6 passed, 1 failed, 0 skipped 6/1/12 3:53:22 AM EST: QtQA::App::TestRunner: test failed, running again to see if it is flaky... 6/1/12 3:53:22 AM EST: Testing tst_QToolButton 6/1/12 3:53:27 AM EST: Totals: 7 passed, 0 failed, 0 skipped 6/1/12 3:53:27 AM EST: QtQA::App::TestRunner: test failed on first attempt and passed on second attempt! 6/1/12 3:53:27 AM EST: QtQA::App::TestRunner: first attempt: exited with exit code 1 6/1/12 3:53:27 AM EST: QtQA::App::TestRunner: the test seems to be flaky, please fix this 6/1/12 3:53:27 AM EST: QtQA::App::TestRunner: this flaky test is being treated as a PASS 6/1/12 3:53:27 AM EST: QtQA::App::TestRunner: end [release\tst_qtoolbutton.exe] [-silent] [-o] [C:\test\recipes\187514714\base\_artifacts\test-logs\tst_qtoolbutton.exe-testresults-00.xml,xml] [-o] [-,txt], 11 seconds, exit code 0