Details
-
Sub-task
-
Resolution: Out of scope
-
P1: Critical
-
None
Description
The BM tool should track a set of reference benchmarks to detect anomalies/instabilities in the test environment. These benchmarks should test aspects of the system that are independent of Qt:
- Looping a primitive statement (like an assignment or a no-op) a fixed number of times.
- Invoking system call X a fixed number of times.
- Ditto for system call Y, and so on.
The idea is that whenever a performance regression is detected in one of the regular benchmarks, we can check against the reference benchmarks to see if the regression could be caused by a change in the test environment.
As an additional verification, each of the reference benchmarks could have two variants, one of which is designed to show e.g. twice the performance of the other one.
Note that the test environment includes QTestLib, so a regression in the reference benchmark could in theory be caused by a change in QTestLib rather than in the system hardware/software.