(Note: For a more detailed discussion, please see the email attached as the first comment to this task)

Background: We need a better way of defining the appropriate subset of benchmarks to track. In particular, we need a good way to avoid tracking a lot of redundant benchmarks.

Proposal:

1. First build all benchmarks (listed in SUBDIRS) using make. Do not run benchmarks automatically using make check or make check-trusted. The SUBDIRS variable thus represents an initial test case level filtering mechanism.
   
   
2. Keep a tests/benchmarks/trackable.xml file formatted like this:
   
   

     <testcase name="..." dir="..." enabled="false" comment="...">
         <function name="..." enabled="false" comment="...">
             <tag name="..." enabled="false" comment="..."/>
             ...
         </function>
         ...
     </testcase>
     ...
   

   This file reflects the hierarchical relationship between test cases, test functions, and data tags, and is subject to the following rules:

   • the enabled attribute is optional with a default value of true
   • if an element has an enabled attribute with value false (or 0)
     then no benchmarks in that subtree are executed (regardless of
     whether a child in that subtree (effectively) has enabled="true"
   • the optional comment attribute can be used e.g. to indicate the
     reason for disabling the benchmark(s)
3. A test runner executes a test if and only if:

• it is explicitly listed in trackable.xml, and
• it is enabled (either explicitly or implicitly)

Note: Since a given benchmark may need to be considered differently on differnt platforms (wrt e.g. applicability and stability), it would be useful to somehow indicate this directly in trackable.xml.