Details
-
Task
-
Resolution: Done
-
P2: Important
-
None
-
None
Description
Metrics for test run- and pass-rates are significantly affected by three long-standing problems with the test infrastructure:
1. Some tests can't be run on some paltforms, either because they don't compile, crash or otherwise don't run to completion.
2. Some tests are either explicitly skipped (via QSKIP) or implicitly skipped (empty test functions, or ifdef'd to empty).
3. The metrics system is unable to accurately determine which tests are valid for a particular platform.
We now have some projects where these issues mean that significant evaluation/processing of run- and pass rates is needed before meaningful and acceptable figures can be presented to those people who make decisions about when a product is ready to ship.
We should be able to deal with test cases more intelligently. Specifically, we should know exactly how many test-cases/test-functions are supposed to run on a specific platform and use that number to calculate the run-rate. So when we do everything well we should be able to achieve 100% run-rate on every platform, even though not every platform is running exactly the same tests.
Attachments
Issue Links
- depends on
-
QTBUG-21652 Remove SkipAll option from QSKIP
-
- Closed
-
Gerrit Reviews
For Gerrit Dashboard: QTBUG-21848 | ||||||
---|---|---|---|---|---|---|
# | Subject | Branch | Project | Status | CR | V |
5824,2 | Avoid using QSKIP in lieu of compile-time checks | master | qt/qtbase | Status: MERGED | +2 | 0 |
6743,2 | Remove QTEST_NOOP_MAIN from qtestlib API. | master | qt/qtbase | Status: MERGED | +2 | 0 |
6805,1 | Remove DEPENDS_ON from qtestlib API. | master | qt/qtbase | Status: MERGED | +2 | 0 |
6884,3 | Remove QTest::SkipMode from qtestlib API. | master | qt/qtbase | Status: MERGED | +2 | 0 |
6885,3 | Remove SkipMode parameter from QSKIP calls. | master | qt/qtbase | Status: MERGED | +2 | 0 |
6887,2 | Remove SkipMode from qtestlib API. | master | qt/qtdeclarative | Status: MERGED | +2 | 0 |
6890,2 | Remove SkipMode parameter from QSKIP. | master | qt/qtxmlpatterns | Status: MERGED | +2 | 0 |
6893,1 | Remove SkipMode parameter from QSKIP. | master | qt/qttools | Status: MERGED | +2 | 0 |
6894,1 | Remove SkipMode parameter from QSKIP. | qml2 | {graveyard}/quick3d | Status: MERGED | +2 | 0 |
6895,2 | Remove SkipMode parameter from QSKIP. | master | qt/qtqa | Status: MERGED | +2 | 0 |
6896,1 | Remove SkipMode parameter from QSKIP. | master | qt/qtpim | Status: ABANDONED | -1 | 0 |
6897,2 | Remove SkipMode parameter from QSKIP. | master | {graveyard}/qtmultimediakit | Status: ABANDONED | +2 | 0 |
6898,1 | Remove SkipMode parameter from QSKIP. | master | qt/qtlocation | Status: MERGED | -1 | 0 |
6899,1 | Remove SkipMode parameter from QSKIP. | master | qt/qtfeedback | Status: MERGED | +2 | 0 |
6900,1 | Remove SkipMode parameter from QSKIP. | master | qt/qtconnectivity | Status: MERGED | +2 | 0 |
8441,1 | Remove SkipMode parameter from QSKIP. | master | qt/qtmultimedia | Status: MERGED | +2 | 0 |
14702,1 | Add testlib selftest for counting of test results. | master | qt/qtbase | Status: MERGED | +2 | 0 |
15299,2 | testlib: Count passes, fails and skips consistently. | master | qt/qtbase | Status: MERGED | +2 | 0 |
15896,1 | qmltest: Count passes, fails and skips consistently. | master | qt/qtdeclarative | Status: MERGED | +2 | 0 |