Situation

Test cases (=classes), functions and their blacklisting information are currently derived from operational data. i.e. its traces in CI testresults.
There is no masterdata table, that knows existing test cases, functions and blacklisting instructions, before they are actually being run.

Impact

Deriving master data indirectly from operational data comes with accuracy and performance limitations.

Performance

In order to e.g. populate comboboxes for test case / function combinations, all non-precheck and successful work items need to be queried for distinct combinations within a defined time frame.
Such queries are complex and time consuming, given the amount of work items.

Accuracy

Test functions may exist but not be run on CI, e.g. because their build conditions are not met on CI.
That can cause existing or new test functions to be considered non-existing.

Use cases

Repeating tests

The test runner currently runs a test function 5 times and records a success if it passes at least once.
This has been implemented to tolerate minor flakiness without blocking CI.
The test runner can't distinguish, whether the function is existing or new. That's why it allows newly added test functions to integrate, even if they are flaky from the beginning - as long as they pass 1 out of 5 times.
Running tests with a pedantic-on-new-test-functions option isn't possible, because the query consumes too much time.
=> Cross checking a master data table just requires a primary key selection.

Blacklisting overview

Blacklisting instructions can be derived from operational data to the extent to which they leave traces in CI testresults.
For performance reasons, blacklisting reporting hasn't been implemented.
Even if it would be implemented based on operational data: Outdated platforms or non-existing data tags don't leave such traces and therefore remain invisible.
In order to check whether a function / data tag is blacklisted and whether the blacklisting is efficient, BLACKLIST files have to be manually inspected as a secondary data source besides testresults.
The user has to be proficient in interpreting BLACKLIST files in the same way, as the testlib.
=> Master data with blacklist instructions interpreted by testlib is efficient to read, accurate, reflecting changes.

Test coverage / consistency dashboard

Changes in CI configurations have impact on test functions being run in blocking / blacklisted mode.
A dashboard based on master data can add value e.g. by flagging

• tests that are no longer run in blocking mode
• excessive blacklisting (e.g. by wild card overdose)