Uploaded image for project: 'Qt Quality Assurance Infrastructure'
  1. Qt Quality Assurance Infrastructure
  2. QTQAINFRA-3070

Create a QUIP that documents the rules surrounding the management of failing/flaky auto tests

    XMLWordPrintable

Details

    • Task
    • Resolution: Unresolved
    • P2: Important
    • None
    • None
    • Test management
    • None

    Description

      We should add a QUIP that instructs developers how to handle failing and flaky tests.

      Things that should be covered:

      • When and how to use QSKIP
      • When and how to use QEXPECT_FAIL
      • When and how to use BLACKLIST

      Quoting https://lists.qt-project.org/pipermail/development/2017-February/028715.html:

      Anybody who identifies a flaky test (ie. a test that is randomly failing in CI), can blacklist that test; under one condition. He needs to at the same time create a P0 bug report about it. Please also add the labels ‘autotest’ and ‘flaky’ to the bug report, so that we can follow up on those.

      Flaky tests might be badly written tests, but they can also hide real problems in Qt. So we need to have a very high priority for looking at them and keeping our blacklist minimal. Making them a P0 ensures that the issue will be looked quickly (as they will block the next release).

      It could also link to https://wiki.qt.io/How_to_reproduce_autotest_fails as additional reading.

      Attachments

        Issue Links

          No reviews matched the request. Check your Options in the drop-down menu of this sections header.

          Activity

            People

              assaarel Asmo Saarela (Inactive)
              mitch_curtis Mitch Curtis
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

                Created:
                Updated:

                Gerrit Reviews

                  There are no open Gerrit changes