Uploaded image for project: 'Qt'
  1. Qt
  2. QTBUG-109441

When running tests in CI, only show output when a test fails

    XMLWordPrintable

Details

    Description

      Currently our test logs are very big, they contain the output of running all tests.
      This makes it hard to find the exact location of the failure in the log when something fails.
      One idea would be to buffer the output of each test, and only print it if a failure happened.

      This could be implemented either at ctest level using it's --output-on-failure option, or at qt-testrunner.py level.

      Doing at ctest level is simple to understand, but it will be harder to know which tests flaky passed, because we don't see the reruns that qt-testrunner is doing.
      The advantage of doing it at qt-testrunner level, is that we could detect flaky passes and print a short one liner to the ci log mentioning that a test not only passed, but it actually flaky passed with x out of y tries.

      Attachments

        Issue Links

          No reviews matched the request. Check your Options in the drop-down menu of this sections header.

          Activity

            People

              qtbuildsystem Qt Build System Team
              alexandru.croitor Alexandru Croitor
              Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

                Created:
                Updated:

                Gerrit Reviews

                  There are no open Gerrit changes