Details
-
Sub-task
-
Resolution: Out of scope
-
P2: Important
-
None
Description
(Note: For a more detailed discussion, please see the email attached as the first comment to this task)
Background: We need a better way of defining the appropriate subset of benchmarks to track. In particular, we need a good way to avoid tracking a lot of redundant benchmarks.
Proposal:
- First build all benchmarks (listed in SUBDIRS) using make. Do not run benchmarks automatically using make check or make check-trusted. The SUBDIRS variable thus represents an initial test case level filtering mechanism.
- Keep a tests/benchmarks/trackable.xml file formatted like this:
<testcase name="..." dir="..." enabled="false" comment="..."> <function name="..." enabled="false" comment="..."> <tag name="..." enabled="false" comment="..."/> ... </function> ... </testcase> ...
This file reflects the hierarchical relationship between test cases, test functions, and data tags, and is subject to the following rules:
- the enabled attribute is optional with a default value of true
- if an element has an enabled attribute with value false (or 0)
then no benchmarks in that subtree are executed (regardless of
whether a child in that subtree (effectively) has enabled="true" - the optional comment attribute can be used e.g. to indicate the
reason for disabling the benchmark(s)
- A test runner executes a test if and only if:
- it is explicitly listed in trackable.xml, and
- it is enabled (either explicitly or implicitly)
Note: Since a given benchmark may need to be considered differently on differnt platforms (wrt e.g. applicability and stability), it would be useful to somehow indicate this directly in trackable.xml.
Attachments
Issue Links
- relates to
-
QTQAINFRA-218 Avoid performance tracking of redundant data tags
-
- Closed
-