Details

    • Task
    • Resolution: Done
    • P1: Critical
    • 2011q2
    • None
    • Test scripts
    • None

    Description

      All autotests should be running in Qt5 CI.

      I would like to do this a bit differently than for Qt4.

      The Qt4 way:

      • compile, run all autotests via (private) scripts
      • get results in testlib XML files
      • upload results to (private) database
      • download results from (private) database frontend
      • subtract results for tests marked as unstable in a (private) blacklist
      • compare munged results against last results from (private) database, consider test run "successful" iff no new regressions appeared

      The Qt5 way should be:

      • compile, run all autotests, via scripts in public qtqa module
        • leveraging build system as much as possible (i.e. `make -j1 -k check')
      • test output goes to two places: stdout as plain text, and to XML log files
      • any test exiting with a non-zero exit code means the test run fails
      • unstable tests are handled by marking them as unstable directly in the source (e.g. by extending QEXPECT_FAIL)
        • unstable tests which may crash may need something more, e.g. CONFIG+=insignificant_test in .pro file
      • XML log files may be uploaded to (private and/or public) results handling systems
        • only for gathering of metrics - test results database no longer handles any regression detection logic

      Probably the best way to get this going is to pick a small module (qtsvg ?) where all the tests are stable and passing, and implement the basic test running for that module first, then gradually extend to other modules.

      Attachments

        No reviews matched the request. Check your Options in the drop-down menu of this sections header.

        Activity

          People

            rmcgover Rohan McGovern (Inactive)
            rmcgover Rohan McGovern (Inactive)
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Gerrit Reviews

                There are no open Gerrit changes