When Qt is built with qmake, Coin's legacy instructions has a bunch of hardcoded behaviour to create a test plan, run the test, extract the test results from xml files, and feed that data to Influx / Grafana for flaky test inspection.
Check https://testresults.qt.io/coin/logview/qt/qtsvg/09dd827c922c06e6d8d995e039b9de61df32ca3e/LinuxCentOS_8_1x86_64LinuxCentOS_8_1x86_64GCCqtci-linux-CentOS-8.1-x86_64-50-64ffe9UseLegacyInstructions/ed6289f7b0615a85920b2dcf137f65842e4fe1c7/build_1606211670 for example
and search for TestPlan and UploadTestPlan keywords, and the instructions between them.
Also check https://testresults.qt.io/coin/logview/qt/qtsvg/09dd827c922c06e6d8d995e039b9de61df32ca3e/LinuxCentOS_8_1x86_64LinuxCentOS_8_1x86_64GCCqtci-linux-CentOS-8.1-x86_64-50-64ffe9UseLegacyInstructions/ed6289f7b0615a85920b2dcf137f65842e4fe1c7/test_1606211671
and see how each test is ran as a separate instruction (RunQtUnitTest) and the qmake built test outputs the test result to an xml file
These xml files are then collected and uploaded by the Upload test results instruction for further processing in Coin, and feeding InfluxDB.
When Qt is built with CMake, all of the above currently does not run.
Qt + CMake is not built using hardcoded Legacy instructions, but rather yaml files, and we can't really hardcode running each test as a separate RunQtUnitTest instruction.
Ideally we don't need to create a testplan, and can just run ctest directly to run all tests.
The remaining question for the build system is how to create the xml files.
There's an open feature request to add junit xml output to ctest, which is not yet implemented https://gitlab.kitware.com/cmake/cmake/-/issues/18654 . I spoke with Tor Arne, and he says it's not worth fixing it upstream, and we should instead use the QtTestLib provided xml generation capabilities.
So we will most likely have to create a shell script wrapper for each test (maybe using CMake as the script language) like qmake does, so that TESTARGS and TESTRUNNER environment variables (like the ones above) are taken into account, and thus let the QtTestLib executables create the xml files.
An open question is what kind of parameters they should be, if Coin expects anything in particular, can we replicate the same ones as done by Coin for qmake integrations.
Coin will also have to be adapted to use the new ctest approach (if it works), bypassing the testplan creation.
Necessary coin changes tracked in