Details
-
Suggestion
-
Resolution: Unresolved
-
P3: Somewhat important
-
None
-
4.6.0, 6.7
-
None
-
-
5
-
Foundation PM Staging
Description
At present, QtTest's benchmarking only records the median result from however many runs of the test it has done. Our graphs of results across successive nightly builds have quite spiky variation from night to night. Knowing whether such variations from one night to another are large or small when compared to the variation within each night's run would make it easier to distinguish signal from noise in this spikiness.
The relevant code in qtbase/src/testlib/qtestcase.cpp is, in TestMethods::invokeTestOnData(int), right at the end,
// If the test is a benchmark, finalize the result after all iterations have finished. if (isBenchmark) { bool testPassed = !QTestResult::skipCurrentTest() && !QTestResult::currentTestFailed(); QTestResult::finishedCurrentTestDataCleanup(); // Only report benchmark figures if the test passed if (testPassed && QBenchmarkTestMethodData::current->resultsAccepted()) QTestLog::addBenchmarkResults(qMedian(resultsList)); }
We have a list of results, resultsList, from which we currently just report the median; we could perfectly well also report the other two quartile boundaries (sufficient to make conventional blox plots), or the inner boundaries of the first and last deciles (for the 10% to 90% range as error bar). This would give us a reasonable error bar on each run to help us understand the scale of variations between runs.