Uploaded image for project: 'Qt'
  1. Qt
  2. QTBUG-66320

Provide a way to conveniently use test helper functions with qtestlib

    XMLWordPrintable

Details

    • Suggestion
    • Resolution: Fixed
    • P3: Somewhat important
    • 6.8.0 FF
    • None
    • Testing: qtestlib
    • None
    • e769cf026 (dev), fde57300a (dev), 45e9f5f2e (dev), 2b8cf058f (6.8), ad568b859 (dev)
    • Foundations Sprint 99, Foundation Sprint 100

    Description

      I have a helper class from which my test case derives from. The helper class sets up a bunch of stuff via non-test helper functions. I was using QVERIFY, QCOMPARE, etc. in those helper functions, until I realised that the test function keeps executing.

      In the docs, each test function has this note:

      Note: This macro can only be used in a test function that is invoked by the test framework.

      This all makes sense, but it leaves a huge gap in implementing and using helper functions in tests.

      My current workaround is to duplicate some of the macros:

      // https://bugreports.qt.io/browse/QTBUG-66320
      // Taken from qtestcase.h and modified to return bool and store a failureMessage
      // so that they can be used in helper functions (non-test functions).
      // We don't have COMPARE, because it would require us to get the failure message
      // into a string, the logic for which is hidden in testlib.
      #define VERIFY(statement) \
      do { \
          if (!static_cast<bool>(statement)) { \
              failureMessage = #statement; \
              return false; \
          } \
      } while (false)
      
      #define FAIL(message) \
      do { \
          failureMessage = message; \
          return false; \
      } while (false)
      
      #define VERIFY2(statement, description) \
      do { \
          if (!static_cast<bool>(statement)) { \
              failureMessage = description; \
              return false; \
          } \
      } while (false)
      
      #define TRY_LOOP_IMPL(expr, timeoutValue, step) \
          if (!(expr)) { \
              QTest::qWait(0); \
          } \
          int qt_test_i = 0; \
          for (; qt_test_i < timeoutValue && !(expr); qt_test_i += step) { \
              QTest::qWait(step); \
          }
      
      #define TRY_TIMEOUT_DEBUG_IMPL(expr, timeoutValue, step)\
          if (!(expr)) { \
              TRY_LOOP_IMPL((expr), (2 * timeoutValue), step);\
              if (expr) { \
                  QString msg = QString::fromUtf8("QTestLib: This test case check (\"%1\") failed because the requested timeout (%2 ms) was too short, %3 ms would have been sufficient this time."); \
                  msg = msg.arg(QString::fromUtf8(#expr)).arg(timeoutValue).arg(timeoutValue + qt_test_i); \
                  FAIL(qPrintable(msg)); \
              } \
          }
      
      // Ideally we'd use qWaitFor instead of QTRY_LOOP_IMPL, but due
      // to a compiler bug on MSVC < 2017 we can't (see QTBUG-59096)
      #define TRY_IMPL(expr, timeout)\
          const int qt_test_step = 50; \
          const int qt_test_timeoutValue = timeout; \
          TRY_LOOP_IMPL((expr), qt_test_timeoutValue, qt_test_step); \
          TRY_TIMEOUT_DEBUG_IMPL((expr), qt_test_timeoutValue, qt_test_step)\
      
      // Will try to wait for the expression to become true while allowing event processing
      #define TRY_VERIFY_WITH_TIMEOUT(expr, timeout) \
      do { \
          TRY_IMPL((expr), timeout);\
          VERIFY(expr); \
      } while (false)
      
      #define TRY_VERIFY(expr) TRY_VERIFY_WITH_TIMEOUT((expr), 5000)
      
      // Will try to wait for the expression to become true while allowing event processing
      #define TRY_VERIFY2_WITH_TIMEOUT(expr, messageExpression, timeout) \
      do { \
          TRY_IMPL((expr), timeout);\
          VERIFY2(expr, messageExpression); \
      } while (false)
      
      #define TRY_VERIFY2(expr, messageExpression) TRY_VERIFY2_WITH_TIMEOUT((expr), (messageExpression), 5000)
      

      I store a failureMessage string in the helper class:

      const char *failureMessage;
      

      I ensure that the result of the helper is used (they return bool so that the actual test function can verify that they succeed), and I give them a goofy prefix so that it's obvious that they're a helper:

      Q_REQUIRED_RESULT bool helpDeleteSavedTestGames();
      
      bool GameTestCase::helpDeleteSavedTestGames()
      {
          // ...
          VERIFY(game->view()->hasActiveFocus());
          // ...
      

      For example, the VERIFY macro returns false if it fails, and stores the failure message. The client code in the actual test functions uses the helper like so:

      QVERIFY2(helpDeleteSavedTestGames(), failureMessage);
      

      This has worked fine so far, but it uses private API, so it would be good if there was an official way to do this.

      Attachments

        Issue Links

          For Gerrit Dashboard: QTBUG-66320
          # Subject Branch Project Status CR V

          Activity

            People

              mmutz Marc Mutz
              mitch_curtis Mitch Curtis
              Vladimir Minenko Vladimir Minenko
              Alex Blasche Alex Blasche
              Votes:
              1 Vote for this issue
              Watchers:
              8 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: