Uploaded image for project: 'Coin'
  1. Coin
  2. COIN-898

In some cases testfunctions are marked flaky when they are actually failing

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Fixed
    • Icon: P2: Important P2: Important
    • None
    • master, 1.5
    • Influxdb
    • None
    • All

      The https://testresults.qt.io/coin/api/results/qt/qtdeclarative/a6526c5c966b87a394cec71a79386dd4b95a117d/MacOSMacOS_11_00arm64MacOSMacOS_11_00arm64Clangqtci-macos-11.0-arm64TestOnly/986900e09b34d8efb74398ad9d7bf5bb1b85b9d7/test_1658235870/log.txt.gz

      workitem failed after re-running failed testcase. With first run there was two functions failing:

      FAIL!  : tst_HoverHandler::deviceCursor(synth, tablet wins)
      FAIL!  : tst_HoverHandler::deviceCursor(synth, mouse wins)

      No with re-runs only

      FAIL!  : tst_HoverHandler::deviceCursor(synth, tablet wins)

      kept failing.
      Even that the test failed, tests functions was marked as flaky in influxdb.
      Now when COIN parses the testresult XML files it checks for testcase name and test function name. In this specific case the test function names were the same. So even if the deviceCursor() kept failing, deviceCursor was also among the passed test functions so COIN considers it as flaky. Which is wrong.

        No reviews matched the request. Check your Options in the drop-down menu of this sections header.

            falt Simo Fält
            falt Simo Fält
            Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

              Created:
              Updated:
              Resolved:

                There are no open Gerrit changes