Uploaded image for project: 'Qt'
  1. Qt
  2. QTBUG-66320

Provide a way to conveniently use test helper functions with qtestlib



    • Type: Suggestion
    • Status: In Progress
    • Priority: P3: Somewhat important
    • Resolution: Unresolved
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: Testing: qtestlib
    • Labels:


      I have a helper class from which my test case derives from. The helper class sets up a bunch of stuff via non-test helper functions. I was using QVERIFY, QCOMPARE, etc. in those helper functions, until I realised that the test function keeps executing.

      In the docs, each test function has this note:

      Note: This macro can only be used in a test function that is invoked by the test framework.

      This all makes sense, but it leaves a huge gap in implementing and using helper functions in tests.

      My current workaround is to duplicate some of the macros:

      // https://bugreports.qt.io/browse/QTBUG-66320
      // Taken from qtestcase.h and modified to return bool and store a failureMessage
      // so that they can be used in helper functions (non-test functions).
      // We don't have COMPARE, because it would require us to get the failure message
      // into a string, the logic for which is hidden in testlib.
      #define VERIFY(statement) \
      do { \
          if (!static_cast<bool>(statement)) { \
              failureMessage = #statement; \
              return false; \
          } \
      } while (false)
      #define FAIL(message) \
      do { \
          failureMessage = message; \
          return false; \
      } while (false)
      #define VERIFY2(statement, description) \
      do { \
          if (!static_cast<bool>(statement)) { \
              failureMessage = description; \
              return false; \
          } \
      } while (false)
      #define TRY_LOOP_IMPL(expr, timeoutValue, step) \
          if (!(expr)) { \
              QTest::qWait(0); \
          } \
          int qt_test_i = 0; \
          for (; qt_test_i < timeoutValue && !(expr); qt_test_i += step) { \
              QTest::qWait(step); \
      #define TRY_TIMEOUT_DEBUG_IMPL(expr, timeoutValue, step)\
          if (!(expr)) { \
              TRY_LOOP_IMPL((expr), (2 * timeoutValue), step);\
              if (expr) { \
                  QString msg = QString::fromUtf8("QTestLib: This test case check (\"%1\") failed because the requested timeout (%2 ms) was too short, %3 ms would have been sufficient this time."); \
                  msg = msg.arg(QString::fromUtf8(#expr)).arg(timeoutValue).arg(timeoutValue + qt_test_i); \
                  FAIL(qPrintable(msg)); \
              } \
      // Ideally we'd use qWaitFor instead of QTRY_LOOP_IMPL, but due
      // to a compiler bug on MSVC < 2017 we can't (see QTBUG-59096)
      #define TRY_IMPL(expr, timeout)\
          const int qt_test_step = 50; \
          const int qt_test_timeoutValue = timeout; \
          TRY_LOOP_IMPL((expr), qt_test_timeoutValue, qt_test_step); \
          TRY_TIMEOUT_DEBUG_IMPL((expr), qt_test_timeoutValue, qt_test_step)\
      // Will try to wait for the expression to become true while allowing event processing
      #define TRY_VERIFY_WITH_TIMEOUT(expr, timeout) \
      do { \
          TRY_IMPL((expr), timeout);\
          VERIFY(expr); \
      } while (false)
      #define TRY_VERIFY(expr) TRY_VERIFY_WITH_TIMEOUT((expr), 5000)
      // Will try to wait for the expression to become true while allowing event processing
      #define TRY_VERIFY2_WITH_TIMEOUT(expr, messageExpression, timeout) \
      do { \
          TRY_IMPL((expr), timeout);\
          VERIFY2(expr, messageExpression); \
      } while (false)
      #define TRY_VERIFY2(expr, messageExpression) TRY_VERIFY2_WITH_TIMEOUT((expr), (messageExpression), 5000)

      I store a failureMessage string in the helper class:

      const char *failureMessage;

      I ensure that the result of the helper is used (they return bool so that the actual test function can verify that they succeed), and I give them a goofy prefix so that it's obvious that they're a helper:

      Q_REQUIRED_RESULT bool helpDeleteSavedTestGames();
      bool GameTestCase::helpDeleteSavedTestGames()
          // ...
          // ...

      For example, the VERIFY macro returns false if it fails, and stores the failure message. The client code in the actual test functions uses the helper like so:

      QVERIFY2(helpDeleteSavedTestGames(), failureMessage);

      This has worked fine so far, but it uses private API, so it would be good if there was an official way to do this.


          Issue Links

          For Gerrit Dashboard: QTBUG-66320
          # Subject Branch Project Status CR V



              Eddy Edward Welbourne
              mitch_curtis Mitch Curtis
              1 Vote for this issue
              6 Start watching this issue