Details
-
Task
-
Resolution: Out of scope
-
P2: Important
-
None
-
None
-
None
-
c07eef8a9ec3629d5e9c130e56184651b8f46e55
Description
qtbase/tests/auto/opengl/qgl/qgl.pro is marked as an insignificant_test, meaning that the results are discarded and the test does not help the CI system to block regressions.
Examining recent CI logs shows that the test passes on Mac OS X 10.7, crashes on Windows 7, passes on Ubuntu 11.10 and fails on Ubuntu 10.04.
4/13/12 4:10:24 PM EST: Testing tst_QGL 4/13/12 4:10:24 PM EST: QtQA::App::TestRunner: Process exited with exit code 0xC0000005 (STATUS_ACCESS_VIOLATION) 4/13/12 4:10:24 PM EST: QtQA::App::TestRunner: test failed, running again to see if it is flaky... 4/13/12 4:10:24 PM EST: No platform plugin argument was specified, defaulting to "windows". 4/13/12 4:10:24 PM EST: Testing tst_QGL 4/13/12 4:10:24 PM EST: QtQA::App::TestRunner: Process exited with exit code 0xC0000005 (STATUS_ACCESS_VIOLATION) 4/13/12 4:10:24 PM EST: QtQA::App::TestRunner: test failure could be reproduced twice consecutively
4/13/12 10:09:07 AM EST: Testing tst_QGL 4/13/12 10:09:07 AM EST: FAIL! : tst_QGL::graphicsViewClipping() Pixel [0,0]: Max fuzz (1) exceeded: (255,255,255,255) vs (0,0,0,255) 4/13/12 10:09:07 AM EST: Loc: [tst_qgl.cpp(879)] 4/13/12 10:09:07 AM EST: FAIL! : tst_QGL::glWidgetRendering() Pixel Max fuzz (1) exceeded: (239,239,239,255) vs (255,0,0,255) 4/13/12 10:09:07 AM EST: Loc: [tst_qgl.cpp(1007)] 4/13/12 10:09:07 AM EST: FAIL! : tst_QGL::glWidgetRendering() Pixel Max fuzz (1) exceeded: (239,239,239,255) vs (255,0,0,255) 4/13/12 10:09:07 AM EST: Loc: [tst_qgl.cpp(1008)] ... 4/13/12 10:09:10 AM EST: FAIL! : tst_QGL::clipTest() Pixel Max fuzz (1) exceeded: (239,239,239,255) vs (255,255,255,255) 4/13/12 10:09:10 AM EST: Loc: [tst_qgl.cpp(1812)] 4/13/12 10:09:10 AM EST: FAIL! : tst_QGL::clipTest() Pixel Max fuzz (1) exceeded: (239,239,239,255) vs (255,255,255,255) 4/13/12 10:09:10 AM EST: Loc: [tst_qgl.cpp(1812)] 4/13/12 10:09:13 AM EST: Totals: 19 passed, 1575 failed, 3 skipped
For the platforms where the test passes consistently, the insignificant_test marker can be removed immediately. For the other platforms, the failures should be investigated to determine whether the problem lies in the test, the code under test or the CI machine configuration. If the test is mostly good, the insignificant_test marker could be replaced with QEXPECT__FAIL calls for known issues that aren't going to be fixed immediately.