Details
-
Task
-
Resolution: Done
-
Not Evaluated
-
None
-
None
-
Linux (since this is currently the only platform in Qt 5 CI)
-
See comment
Description
Currently the insignificant_test flag allows us to disable (i.e. ignore failures in) a test case as a whole, but not individual test functions. This reduces the effective test coverage more than necessary since many "unproblematic" test functions would not be allowed to block CI even when they should. In other words: a real bug in Qt may in this case slip through CI, even when it was actually caught by an existing test function.
A rough analysis of recent pulse logs in qtbase#master (runs 730..829) shows the following situation for test cases that contain test functions of only two types, those that are passing in all runs and those that are failing in all runs:
test case | failing functions | total functions |
---|---|---|
tst_qabstractprintdialog | 2 | 3 |
tst_exceptionsafety_objects | 1 | 5 |
tst_qopengl | 2 | 5 |
tst_qinputcontext | 3 | 6 |
tst_qglthreads | 1 | 7 |
tst_qdialog | 1 | 14 |
tst_qprinter | 1 | 18 |
tst_qtextbrowser | 10 | 18 |
tst_qdoublespinbox | 1 | 23 |
tst_qmenu | 2 | 24 |
tst_qlistview | 1 | 44 |
tst_qgraphicswidget | 2 | 67 |
tst_qgraphicsscene | 1 | 80 |
tst_qvariant | 1 | 85 |
tst_qgraphicsview | 2 | 91 |
tst_qgraphicsitem | 1 | 168 |
Total: | 32 | 658 |
This essentially means that for these test cases, 626 out of 658 test functions (95%) are turned off for no good reason.
The ideal way to remedy this situation is of course to fix the failing test functions, and get rid of the insignificant_test flag without any further ado.
In the meantime, until the proper fix is in place, I propose to remove the insignificant_test flag, and instead disable the failing test functions individually using QEXPECT_FAIL. (Note that the failing test functions in this case are assumed to fail in every run, so we would not expect any XPASS'es).
Attachments
For Gerrit Dashboard: QTBUG-22453 | ||||||
---|---|---|---|---|---|---|
# | Subject | Branch | Project | Status | CR | V |
7714,2 | Re-enabled passing tests in tst_qgraphicsitem | master | qt/qtbase | Status: MERGED | +2 | 0 |
7717,2 | Re-enabled passing tests in tst_qgraphicsview | master | qt/qtbase | Status: MERGED | +2 | 0 |
7724,2 | Re-enabled passing tests in tst_qgraphicsscene | master | qt/qtbase | Status: MERGED | +2 | 0 |
7741,1 | Re-enabled passing tests in tst_qgraphicswidget | master | qt/qtbase | Status: MERGED | +2 | 0 |
7879,2 | Re-enabled passing tests in tst_qlistview | master | qt/qtbase | Status: MERGED | +2 | 0 |
8051,2 | Re-enabled passing tests in tst_qvariant | master | qt/qtbase | Status: MERGED | +2 | 0 |
8159,1 | Re-enabled passing tests in tst_qprinter | master | qt/qtbase | Status: MERGED | +2 | 0 |
8172,1 | Re-enabled passing tests in tst_qinputcontext | master | qt/qtbase | Status: MERGED | +2 | 0 |
8175,1 | Re-enabled passing tests in tst_qmenu | master | qt/qtbase | Status: MERGED | +2 | 0 |
8248,1 | Re-enabled tst_qdialog | master | qt/qtbase | Status: MERGED | +2 | 0 |
8377,2 | Re-enabled passing tests in tst_qopengl | master | qt/qtbase | Status: MERGED | +2 | 0 |
8499,1 | Re-enabled passing tests in tst_exceptionsafety_objects | master | qt/qtbase | Status: MERGED | +2 | 0 |
8511,1 | Re-enabled passing tests in tst_qabstractprintdialog | master | qt/qtbase | Status: MERGED | +2 | 0 |
16135,2 | Re-enabled passing tests in tst_qvariant | 4.8 | qt/qt | Status: MERGED | +2 | 0 |