Details
-
Task
-
Resolution: Fixed
-
P2: Important
-
None
-
None
-
None
Description
Create a script that, given a specific workitem ID (aka buildKey):
- it fetches the testresults tarball
- it reads the XML test files
- It prints data points about whether tests were passing/failing or being flaky
- (Later) it inserts the points to a database
Example workitem ID (buildKey):
qt/qtlanguageserver/19b874f185b768bd902a23e40e9a305998bf28ff/MacOSMacOS_12x86_64AndroidAndroid_ANYx86_64Clangqtci-macos-12-x86_64-102-09b1e8DisableTests_Sccache_UseConfigure/f96bee4fbb6540effbede89dc0816a3fc1bf6bc9/build_1661782709
NOTES
- A test_name consists of test_executable+test_function+test_datatag
- A test_name may re-run up to 5 times in order to combat flakiness. The final run is what sets the test_outcome.
- Possible test outcomes: pass, fail, crash
- Every single run of a test_name can have a test_result.
- These are printed listed in the XML file: FAIL,XPASS, PASS,XFAIL,BFAIL,BPASS,BXFAIL,BXPASS,SKIP...
- The number of re-runs should also be recorded.
See suggested schema for storing all that in QTQAINFRA-4563.
Attachments
Issue Links
- resulted in
-
QTQAINFRA-5291 Testresult parser improvements
- Closed
-
QTBUG-107722 Timestamps for each incident are missing in the QTest XML output
- Reported
- split from
-
QTQAINFRA-4566 Implement test result parser & db importer
- Closed