Uploaded image for project: 'Coin'
  1. Coin
  2. COIN-898

In some cases testfunctions are marked flaky when they are actually failing

    XMLWordPrintable

Details

    • Bug
    • Resolution: Unresolved
    • P2: Important
    • None
    • master, 1.5
    • Influxdb
    • None
    • All

    Description

      The https://testresults.qt.io/coin/api/results/qt/qtdeclarative/a6526c5c966b87a394cec71a79386dd4b95a117d/MacOSMacOS_11_00arm64MacOSMacOS_11_00arm64Clangqtci-macos-11.0-arm64TestOnly/986900e09b34d8efb74398ad9d7bf5bb1b85b9d7/test_1658235870/log.txt.gz

      workitem failed after re-running failed testcase. With first run there was two functions failing:

      FAIL!  : tst_HoverHandler::deviceCursor(synth, tablet wins)
      FAIL!  : tst_HoverHandler::deviceCursor(synth, mouse wins)

      No with re-runs only

      FAIL!  : tst_HoverHandler::deviceCursor(synth, tablet wins)

      kept failing.
      Even that the test failed, tests functions was marked as flaky in influxdb.
      Now when COIN parses the testresult XML files it checks for testcase name and test function name. In this specific case the test function names were the same. So even if the deviceCursor() kept failing, deviceCursor was also among the passed test functions so COIN considers it as flaky. Which is wrong.

      Attachments

        No reviews matched the request. Check your Options in the drop-down menu of this sections header.

        Activity

          People

            falt Simo Fält
            falt Simo Fält
            Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

              Created:
              Updated:

              Gerrit Reviews

                There are no open Gerrit changes