Details
-
Task
-
Resolution: Unresolved
-
P2: Important
-
None
-
None
-
None
Description
With the ctest_results table we can see find the integrations that have zero entries in it. These are CI-flaky since the test run never completed.
- has a "fail" ctest result recorded but
no JUnit XML result OR a pass-outcome result? - ctest_message is Timeout
- the workitem result is fail/error AND type=Test AND ctest-junit-xml file was expected (i.e. not the legacy platform) but missing
TODO a crashed_tests_2 TABLE that contain crashed tests without XML files, known only to ctest_results
0. if a workitem has no XML results at all, then it's OK (possibly ASAN build or a non-qt project)
1. test/emulator crash: failure in ctest_results, missing test XML file, not ASAN testrun or other type that does not generate XML (i.e. workitem must have produced other XMLs)
2. timeout/other crash: existing XML results for workitem but missing ctest_result
3. failed workitem without individually failed test in neither ctest or QTest -> CI flakiness
For the association between qtest and ctest results to happen, we need to match the TestCase as reported by qtest, with the testcase as reported by ctest (is that the test executable?)
- TODO might need to INDEX ctest_results based on ctest_testcase_n