Uploaded image for project: 'Qt'
  1. Qt
  2. QTBUG-95661

Test result counts are incorrect

    XMLWordPrintable

    Details

    • Platform/s:
      All
    • Story Points:
      8
    • Commits:
      13d2e13290 (qt/qtbase/dev) 13d2e13290 (qt/tqtc-qtbase/dev)
    • Sprint:
      Qt6_Foundation_Sprint 40, Foundation_Sprint 41 (Team 1), Team 1 Foundation_Sprint 42, Team 1 Foundation_Sprint 43, Team 1 Foundation_Sprint 44, Team 1 Foundation_Sprint 45, Team 1 Foundation_Sprint 46, Team 1 Foundation_Sprint 47, Team 1 Foundation_Sprint 48, Team One Foundation Sprint 49

      Description

      https://codereview.qt-project.org/c/qt/qtbase/+/253709 introduced incorrect behavior for counting tests. A blacklisted test will be counted twice now.

      There are also issues with expect-fail tests (both blacklisted and not), where an expected pass will count as a failure, but with Continue will go on testing, possibly introducing more expected passes, each one increasing the failure count, and perhaps finally resulting in a failed QVERIFY, furthermore increasing the count. The real count should increase by 1, since these are all failures in the same test. (we count tests, not verifications).

        Attachments

          Issue Links

          is duplicated by

          Bug - A problem which impairs or prevents the functions of the product. QTBUG-98454 QtTestlib JUnitXML reporter generates a failure and a skip in the same test case

          • P2: Important - Urgent, should be fixed, but will not stop the release.
          • Reported
          is required for

          Task - A task that needs to be done. QTBUG-96844 Rationalize logging in QtTestLib

          • P2: Important - Urgent, should be fixed, but will not stop the release.
          • Closed
          No reviews matched the request. Check your Options in the drop-down menu of this sections header.

            Activity

              People

              Assignee:
              Eddy Edward Welbourne
              Reporter:
              vestbo Tor Arne Vestbø
              PM Owner:
              Vladimir Minenko Vladimir Minenko
              RnD Owner:
              Alex Blasche Alex Blasche
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved: