Details

    • Sub-task
    • Resolution: Out of scope
    • P2: Important
    • 2012q4
    • Performance tracking
    • None

    Description

      (Note: For a more detailed discussion, please see the email attached as the first comment to this task)

      Background: We need a better way of defining the appropriate subset of benchmarks to track. In particular, we need a good way to avoid tracking a lot of redundant benchmarks.

      Proposal:

      1. First build all benchmarks (listed in SUBDIRS) using make. Do not run benchmarks automatically using make check or make check-trusted. The SUBDIRS variable thus represents an initial test case level filtering mechanism.

      2. Keep a tests/benchmarks/trackable.xml file formatted like this:

          <testcase name="..." dir="..." enabled="false" comment="...">
              <function name="..." enabled="false" comment="...">
                  <tag name="..." enabled="false" comment="..."/>
                  ...
              </function>
              ...
          </testcase>
          ...
        

        This file reflects the hierarchical relationship between test cases, test functions, and data tags, and is subject to the following rules:

        • the enabled attribute is optional with a default value of true
        • if an element has an enabled attribute with value false (or 0)
          then no benchmarks in that subtree are executed (regardless of
          whether a child in that subtree (effectively) has enabled="true"
        • the optional comment attribute can be used e.g. to indicate the
          reason for disabling the benchmark(s)
      3. A test runner executes a test if and only if:
      • it is explicitly listed in trackable.xml, and
      • it is enabled (either explicitly or implicitly)

      Note: Since a given benchmark may need to be considered differently on differnt platforms (wrt e.g. applicability and stability), it would be useful to somehow indicate this directly in trackable.xml.

      Attachments

        Issue Links

          No reviews matched the request. Check your Options in the drop-down menu of this sections header.

          Activity

            People

              bdo Jo Asplin (Inactive)
              bdo Jo Asplin (Inactive)
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Gerrit Reviews

                  There are no open Gerrit changes