Test execution

The need to write scripts to determine test plans and execute them (flaky handling) should be minimized (for example, COIN currently runs qmake -r to create a test plan). However, it should be considered what ctest provides, first.

• Add functionality create test plan and present results
• Move flaky rerun policies into testlib (note ctest also has some re-run feature)

Extend functionality

• Provide more Gui utilities (High DPI screen/window sizes, High DPI fuzz comparison, image comparison); potentially as a separate module? This would allow for removing a lot of code duplicated in the tests.
• Re-evaluate reasons why we limit executables to single test classes and if the reasons do no more apply add functionality to fully support several test classes in a single binary
  (QTest::qExec() can be used multiple times and results could be added to return the final result.. but this loses e.g. functionality of explicitly running single test functions)

Code

• Bring code up to scratch (however carefully considering performance)
• Check macros:
  • Consider using exceptions instead of void returns – this would remove the need to always use void for helper functions; it would also let us officially support the use of these macros in helper functions (they all currently document that they "can only be used in a test function that is invoked by the test framework" which isn't true, but is an attempt to say such use is not supported.)
  • Performance: Restructure QCOMPARE to only format the operands to strings in case of failure (currently, operands are always formatted, which is costly) (→ QTBUG-98874, QTBUG-101180)
  • Provide other operators (QLESS_THAN, …) which allows for removing a lot of QVERIFY and log the values in case of failure (→ QTBUG-98873)
• Data driven tests
  • Performance: Use coroutines to yield test data via some tuple (would remove the need to store a matrix of QVariants which is a performance hit, currently)
  • Provide something for resources during shared over the live time of data driven tests. This is currently not possible.