Details
-
Sub-task
-
Resolution: Out of scope
-
P1: Critical
-
None
-
None
-
None
Description
In order to calculate each platform's run-rate for autotests, the list of tests executed in a CI run must be compared with a list of all tests that are expected to run for that platform. The current technique for generating the expected list is flawed and may generate a list that contains tests that are not valid for a particular platform.
Given the limited development lifetime of Qt4 and its low rate of autotest creation, it is acceptable to provide a "quick-and-dirty" solution for those branches and developer a more elegant solution for Qt5.
The issue could be solved either by hard-coding a list of exclusions (things we know aren't valid autotests) into the metrics script and/or by including a flag in the .pro file of each non-test project in the tests/auto tree. The list of non-tests can be determined by examining current test results to determine which tests don't currently run for each platform, inspecting those tests to decide whether they should run.