Details
-
Task
-
Resolution: Done
-
P1: Critical
-
None
-
None
Description
All autotests should be running in Qt5 CI.
I would like to do this a bit differently than for Qt4.
The Qt4 way:
- compile, run all autotests via (private) scripts
- get results in testlib XML files
- upload results to (private) database
- download results from (private) database frontend
- subtract results for tests marked as unstable in a (private) blacklist
- compare munged results against last results from (private) database, consider test run "successful" iff no new regressions appeared
The Qt5 way should be:
- compile, run all autotests, via scripts in public qtqa module
- leveraging build system as much as possible (i.e. `make -j1 -k check')
- test output goes to two places: stdout as plain text, and to XML log files
- any test exiting with a non-zero exit code means the test run fails
- unstable tests are handled by marking them as unstable directly in the source (e.g. by extending QEXPECT_FAIL)
- unstable tests which may crash may need something more, e.g. CONFIG+=insignificant_test in .pro file
- XML log files may be uploaded to (private and/or public) results handling systems
- only for gathering of metrics - test results database no longer handles any regression detection logic
Probably the best way to get this going is to pick a small module (qtsvg ?) where all the tests are stable and passing, and implement the basic test running for that module first, then gradually extend to other modules.