Details
-
Task
-
Resolution: Fixed
-
P2: Important
-
None
-
None
-
None
-
19aa21b59 (dev), d15a32251 (6.6), ac4bc3df8 (tqtc/lts-6.5)
Description
A test might have PASSed all its testcases, but might still exit with != 0, i.e. fail, for example because of a crash in the destructor. qt-testrunner will detect that and re-run the full test executable. But such events should be logged.
So qt-testrunner should output a machine-readable log file, mentioning
- BEGIN
- What was tested
- How many re-runs
- (maybe intermediate results)
- Final outcome
- END
qt-testrunner should flush that file to disk frequently. If the END is missing one can assume a crash/cancellation or other irregular termination, outside of qt-testrunner's control.
The filename should be placed into COIN_CTEST_RESULTSDIR and it should be named uniquely, since qt-testrunner is being executed once for each test.
Newer ctest can also return a JUnit XML log
--output-junit <file>
New in version 3.21.Write test results in JUnit format.
This option tells CTest to write test results to <file> in JUnit XML format. If <file> already exists, it will be overwritten. If using the -S option to run a dashboard script, use the OUTPUT_JUNIT keyword with the ctest_test() command instead.
ctest is the command that does all test running. Coin-agent executes ctest once, and then ctest loops through all tests and runs each one after wrapping it with any commands in the environment variable $TESTRUNNER. As such, it doesn't know of any possible re-runs that the TESTRUNNER did.
Nevertheless it might be useful to get that XML file, given that it's a summary of the total test run in the workitem. For example it will catch what is reported at the 1st line of the description of the ticket, i.e. "a test might have PASSed all its testcases, but still exit with != 0".
Attachments
Issue Links
- relates to
-
QTBUG-109441 When running tests in CI, only show output when a test fails
- Closed
- resulted in
-
QTQAINFRA-5941 coin_testresults_parser: add support for parsing the JUnit XML file generated by CTest
- Closed
For Gerrit Dashboard: QTQAINFRA-5146 | ||||||
---|---|---|---|---|---|---|
# | Subject | Branch | Project | Status | CR | V |
516436,4 | CTest should write a JUnit XML test summary | dev | qt/qtbase | Status: MERGED | +2 | 0 |
516901,3 | CTest should write a JUnit XML test summary | 6.6 | qt/qtbase | Status: MERGED | +2 | 0 |
516965,2 | CTest should write a JUnit XML test summary | tqtc/lts-6.5 | qt/tqtc-qtbase | Status: MERGED | +2 | 0 |