qt-testrunner.py (or ctest) should write a status file

A test might have PASSed all its testcases, but might still exit with != 0, i.e. fail, for example because of a crash in the destructor. qt-testrunner will detect that and re-run the full test executable. But such events should be logged.

So qt-testrunner should output a machine-readable log file, mentioning

• BEGIN
• What was tested
• How many re-runs
• (maybe intermediate results)
• Final outcome
• END

qt-testrunner should flush that file to disk frequently. If the END is missing one can assume a crash/cancellation or other irregular termination, outside of qt-testrunner's control.

The filename should be placed into COIN_CTEST_RESULTSDIR and it should be named uniquely, since qt-testrunner is being executed once for each test.

Newer ctest can also return a JUnit XML log

--output-junit <file>
New in version 3.21.

Write test results in JUnit format.

This option tells CTest to write test results to <file> in JUnit XML format. If <file> already exists, it will be overwritten. If using the -S option to run a dashboard script, use the OUTPUT_JUNIT keyword with the ctest_test() command instead.

ctest is the command that does all test running. Coin-agent executes ctest once, and then ctest loops through all tests and runs each one after wrapping it with any commands in the environment variable $TESTRUNNER. As such, it doesn't know of any possible re-runs that the TESTRUNNER did.

Nevertheless it might be useful to get that XML file, given that it's a summary of the total test run in the workitem. For example it will catch what is reported at the 1st line of the description of the ticket, i.e. "a test might have PASSed all its testcases, but still exit with != 0".