Uploaded image for project: 'Qt Quality Assurance Infrastructure'
  1. Qt Quality Assurance Infrastructure
  2. QTQAINFRA-287 Handle false positives
  3. QTQAINFRA-243

Add reference benchmarks to detect anomalies in test environment

    XMLWordPrintable

Details

    • Sub-task
    • Resolution: Out of scope
    • P1: Critical
    • 2012q4
    • Performance tracking
    • None

    Description

      The BM tool should track a set of reference benchmarks to detect anomalies/instabilities in the test environment. These benchmarks should test aspects of the system that are independent of Qt:

      • Looping a primitive statement (like an assignment or a no-op) a fixed number of times.
      • Invoking system call X a fixed number of times.
      • Ditto for system call Y, and so on.

      The idea is that whenever a performance regression is detected in one of the regular benchmarks, we can check against the reference benchmarks to see if the regression could be caused by a change in the test environment.

      As an additional verification, each of the reference benchmarks could have two variants, one of which is designed to show e.g. twice the performance of the other one.

      Note that the test environment includes QTestLib, so a regression in the reference benchmark could in theory be caused by a change in QTestLib rather than in the system hardware/software.

      Attachments

        No reviews matched the request. Check your Options in the drop-down menu of this sections header.

        Activity

          People

            bdo Jo Asplin (Inactive)
            bdo Jo Asplin (Inactive)
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Time Tracking

                Estimated:
                Original Estimate - 1 day
                1d
                Remaining:
                Remaining Estimate - 1 day
                1d
                Logged:
                Time Spent - Not Specified
                Not Specified

                Gerrit Reviews

                  There are no open Gerrit changes