Uploaded image for project: 'Qt Quality Assurance Infrastructure'
  1. Qt Quality Assurance Infrastructure
  2. QTQAINFRA-6230

flaky tests: show the line number of the failing test directly on the testresults page

    XMLWordPrintable

Details

    • Bug
    • Resolution: Unresolved
    • P2: Important
    • None
    • dev
    • Metrics / Test Results
    • None

    Description

      When looking into a flaky test issue on testresults.qt.io (e.g https://testresults.qt.io/grafana/d/3dhio4K7k/fastcheck-ci-test-info?var-testCase=tst_qcombobox&var-testFunction=popupPositionAfterStyleChange&orgId=1), the first thing you typically want to know, is which QCOMPARE or QVERFIY that tends to fail. In order to figure this out today, you need to navigate the testresults page to find the correct menu, and then click on the correct link. This will take you to a log in the browser which is really slow to navigate and search manually. It would be nice if this information was instead provided automatically on the testresults page, similar to how we do it on patches on codereview.qt-project.org for changes that fails to merge. I think this will save quite some time and laborious manual work when investigating flaky tests.

      Anna edit:
      The information can be parsed from test result XML file:

      <Incident type="fail" file="/home/qt/work/qt/qtbase/tests/auto/widgets/widgets/qcombobox/tst_qcombobox.cpp" line="3470">
      <Description><![CDATA[Compared values are not the same
      Actual (box.currentText()) : "first"
      Expected (QStringLiteral("last")): "last"]]></Description>
      

      so it would require modifying of your testreults parser scripts (see attached example) . Of course not all flaky test would provide this nice output, but many would.

      Attachments

        1. baa84681-d1d4-4fd6-8f8b-2b6d0b608e77.jpg
          35 kB
          Richard Moe Gustavsen
        2. tst_qcombobox-1711628082584.xml
          2 kB
          Anna Wojciechowska
        No reviews matched the request. Check your Options in the drop-down menu of this sections header.

        Activity

          People

            jujokini Jukka Jokiniva
            richard Richard Moe Gustavsen
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

              Created:
              Updated:

              Gerrit Reviews

                There are no open Gerrit changes