Uploaded image for project: 'Qt'
  1. Qt
  2. QTBUG-66320

Provide a way to conveniently use test helper functions with qtestlib

XMLWordPrintable

    • Icon: Suggestion Suggestion
    • Resolution: Fixed
    • Icon: P3: Somewhat important P3: Somewhat important
    • 6.8.0 FF
    • None
    • Testing: qtestlib
    • None
    • e769cf026 (dev), fde57300a (dev), 45e9f5f2e (dev), 2b8cf058f (6.8), ad568b859 (dev)
    • Foundations Sprint 99, Foundation Sprint 100

      I have a helper class from which my test case derives from. The helper class sets up a bunch of stuff via non-test helper functions. I was using QVERIFY, QCOMPARE, etc. in those helper functions, until I realised that the test function keeps executing.

      In the docs, each test function has this note:

      Note: This macro can only be used in a test function that is invoked by the test framework.

      This all makes sense, but it leaves a huge gap in implementing and using helper functions in tests.

      My current workaround is to duplicate some of the macros:

      // https://bugreports.qt.io/browse/QTBUG-66320
      // Taken from qtestcase.h and modified to return bool and store a failureMessage
      // so that they can be used in helper functions (non-test functions).
      // We don't have COMPARE, because it would require us to get the failure message
      // into a string, the logic for which is hidden in testlib.
      #define VERIFY(statement) \
      do { \
          if (!static_cast<bool>(statement)) { \
              failureMessage = #statement; \
              return false; \
          } \
      } while (false)
      
      #define FAIL(message) \
      do { \
          failureMessage = message; \
          return false; \
      } while (false)
      
      #define VERIFY2(statement, description) \
      do { \
          if (!static_cast<bool>(statement)) { \
              failureMessage = description; \
              return false; \
          } \
      } while (false)
      
      #define TRY_LOOP_IMPL(expr, timeoutValue, step) \
          if (!(expr)) { \
              QTest::qWait(0); \
          } \
          int qt_test_i = 0; \
          for (; qt_test_i < timeoutValue && !(expr); qt_test_i += step) { \
              QTest::qWait(step); \
          }
      
      #define TRY_TIMEOUT_DEBUG_IMPL(expr, timeoutValue, step)\
          if (!(expr)) { \
              TRY_LOOP_IMPL((expr), (2 * timeoutValue), step);\
              if (expr) { \
                  QString msg = QString::fromUtf8("QTestLib: This test case check (\"%1\") failed because the requested timeout (%2 ms) was too short, %3 ms would have been sufficient this time."); \
                  msg = msg.arg(QString::fromUtf8(#expr)).arg(timeoutValue).arg(timeoutValue + qt_test_i); \
                  FAIL(qPrintable(msg)); \
              } \
          }
      
      // Ideally we'd use qWaitFor instead of QTRY_LOOP_IMPL, but due
      // to a compiler bug on MSVC < 2017 we can't (see QTBUG-59096)
      #define TRY_IMPL(expr, timeout)\
          const int qt_test_step = 50; \
          const int qt_test_timeoutValue = timeout; \
          TRY_LOOP_IMPL((expr), qt_test_timeoutValue, qt_test_step); \
          TRY_TIMEOUT_DEBUG_IMPL((expr), qt_test_timeoutValue, qt_test_step)\
      
      // Will try to wait for the expression to become true while allowing event processing
      #define TRY_VERIFY_WITH_TIMEOUT(expr, timeout) \
      do { \
          TRY_IMPL((expr), timeout);\
          VERIFY(expr); \
      } while (false)
      
      #define TRY_VERIFY(expr) TRY_VERIFY_WITH_TIMEOUT((expr), 5000)
      
      // Will try to wait for the expression to become true while allowing event processing
      #define TRY_VERIFY2_WITH_TIMEOUT(expr, messageExpression, timeout) \
      do { \
          TRY_IMPL((expr), timeout);\
          VERIFY2(expr, messageExpression); \
      } while (false)
      
      #define TRY_VERIFY2(expr, messageExpression) TRY_VERIFY2_WITH_TIMEOUT((expr), (messageExpression), 5000)
      

      I store a failureMessage string in the helper class:

      const char *failureMessage;
      

      I ensure that the result of the helper is used (they return bool so that the actual test function can verify that they succeed), and I give them a goofy prefix so that it's obvious that they're a helper:

      Q_REQUIRED_RESULT bool helpDeleteSavedTestGames();
      
      bool GameTestCase::helpDeleteSavedTestGames()
      {
          // ...
          VERIFY(game->view()->hasActiveFocus());
          // ...
      

      For example, the VERIFY macro returns false if it fails, and stores the failure message. The client code in the actual test functions uses the helper like so:

      QVERIFY2(helpDeleteSavedTestGames(), failureMessage);
      

      This has worked fine so far, but it uses private API, so it would be good if there was an official way to do this.

        For Gerrit Dashboard: QTBUG-66320
        # Subject Branch Project Status CR V

            mmutz Marc Mutz
            mitch_curtis Mitch Curtis
            Vladimir Minenko Vladimir Minenko
            Alex Blasche Alex Blasche
            Votes:
            1 Vote for this issue
            Watchers:
            8 Start watching this issue

              Created:
              Updated:
              Resolved: