Uploaded image for project: 'Qt'
  1. Qt
  2. QTBUG-123040

Add error-bars for benchmark results

    XMLWordPrintable

Details

    • Suggestion
    • Resolution: Unresolved
    • P3: Somewhat important
    • None
    • 4.6.0, 6.7
    • Testing: qtestlib
    • None
    • All
    • 5
    • Foundation PM Staging

    Description

      At present, QtTest's benchmarking only records the median result from however many runs of the test it has done. Our graphs of results across successive nightly builds have quite spiky variation from night to night. Knowing whether such variations from one night to another are large or small when compared to the variation within each night's run would make it easier to distinguish signal from noise in this spikiness.

      The relevant code in qtbase/src/testlib/qtestcase.cpp is, in TestMethods::invokeTestOnData(int), right at the end,

          // If the test is a benchmark, finalize the result after all iterations have finished.
          if (isBenchmark) {
              bool testPassed = !QTestResult::skipCurrentTest() && !QTestResult::currentTestFailed();
              QTestResult::finishedCurrentTestDataCleanup();
              // Only report benchmark figures if the test passed
              if (testPassed && QBenchmarkTestMethodData::current->resultsAccepted())
                  QTestLog::addBenchmarkResults(qMedian(resultsList));
          }
      

      We have a list of results, resultsList, from which we currently just report the median; we could perfectly well also report the other two quartile boundaries (sufficient to make conventional blox plots), or the inner boundaries of the first and last deciles (for the 10% to 90% range as error bar). This would give us a reasonable error bar on each run to help us understand the scale of variations between runs.

      Attachments

        No reviews matched the request. Check your Options in the drop-down menu of this sections header.

        Activity

          People

            cnn Qt Core & Network
            Eddy Edward Welbourne
            Vladimir Minenko Vladimir Minenko
            Alex Blasche Alex Blasche
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

              Created:
              Updated:

              Gerrit Reviews

                There are no open Gerrit changes