Details
-
Bug
-
Resolution: Unresolved
-
P2: Important
-
None
-
dev
-
None
Description
When looking into a flaky test issue on testresults.qt.io (e.g https://testresults.qt.io/grafana/d/3dhio4K7k/fastcheck-ci-test-info?var-testCase=tst_qcombobox&var-testFunction=popupPositionAfterStyleChange&orgId=1), the first thing you typically want to know, is which QCOMPARE or QVERFIY that tends to fail. In order to figure this out today, you need to navigate the testresults page to find the correct menu, and then click on the correct link. This will take you to a log in the browser which is really slow to navigate and search manually. It would be nice if this information was instead provided automatically on the testresults page, similar to how we do it on patches on codereview.qt-project.org for changes that fails to merge. I think this will save quite some time and laborious manual work when investigating flaky tests.
Anna edit:
The information can be parsed from test result XML file:
<Incident type="fail" file="/home/qt/work/qt/qtbase/tests/auto/widgets/widgets/qcombobox/tst_qcombobox.cpp" line="3470"> <Description><![CDATA[Compared values are not the same Actual (box.currentText()) : "first" Expected (QStringLiteral("last")): "last"]]></Description>
so it would require modifying of your testreults parser scripts (see attached example) . Of course not all flaky test would provide this nice output, but many would.