Uploaded image for project: 'Coin'
  1. Coin
  2. COIN-898

In some cases testfunctions are marked flaky when they are actually failing

    XMLWordPrintable

Details

    • Bug
    • Resolution: Unresolved
    • P2: Important
    • None
    • master, 1.5
    • Influxdb
    • None
    • All

    Description

      The https://testresults.qt.io/coin/api/results/qt/qtdeclarative/a6526c5c966b87a394cec71a79386dd4b95a117d/MacOSMacOS_11_00arm64MacOSMacOS_11_00arm64Clangqtci-macos-11.0-arm64TestOnly/986900e09b34d8efb74398ad9d7bf5bb1b85b9d7/test_1658235870/log.txt.gz

      workitem failed after re-running failed testcase. With first run there was two functions failing:

      FAIL!  : tst_HoverHandler::deviceCursor(synth, tablet wins)
      FAIL!  : tst_HoverHandler::deviceCursor(synth, mouse wins)

      No with re-runs only

      FAIL!  : tst_HoverHandler::deviceCursor(synth, tablet wins)

      kept failing.
      Even that the test failed, tests functions was marked as flaky in influxdb.
      Now when COIN parses the testresult XML files it checks for testcase name and test function name. In this specific case the test function names were the same. So even if the deviceCursor() kept failing, deviceCursor was also among the passed test functions so COIN considers it as flaky. Which is wrong.

      Attachments

        Issue Links

          No reviews matched the request. Check your Options in the drop-down menu of this sections header.

          Activity

            People

              falt Simo Fält
              falt Simo Fält
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

                Created:
                Updated:

                Gerrit Reviews

                  There are no open Gerrit changes