Uploaded image for project: 'Qt'
  1. Qt
  2. QTBUG-70029

Provide API to fail a test if a warning occurs

    XMLWordPrintable

Details

    • efb283fb7f72e950c8ecf755b960a3c1b36b5507 (qt/qtbase/dev)

    Description

      This is useful to automatically detect warnings that should not occur, as there is often no way to do so if no state is changed as a result of the problem the warning is alerting us to.

      For warnings that should occur, QTest::ignoreMessage() can still be used as previously.

      The feature would be especially useful in QML, where it's not possible to fail on warnings.

      Here are some potential solutions:

      #1 failOnWarnings()/setFailOnWarnings(bool) (https://codereview.qt-project.org/#/c/89178/)

      + Simple API.
      + Can be used (called) from QML.
      - No way to limit which warnings cause failures.

      void initTestCase()
      {
          setFailOnWarnings(true);
      }
      
      void testSomething()
      {
          setFailOnWarnings(true);
          // ...
      }
      

      #2 failOnWarning(const QRegularExpression &) (https://codereview.qt-project.org/c/qt/qtbase/+/375915/1)

      + Simple API.
      + Can limit which warnings cause failures.
      + Can be used (called) from QML.
      - No way to stop failing on a certain warning, and as a result;
      - No way to implement a simplified "bool failOnWarnings" property for TestCase in QML.

      void initTestCase()
      {
          failOnWarning(QRegularExpression(".*"));
          // or:
          //failOnWarning(QRegularExpression("Binding loop.*"));
      }
      
      void testSomething()
      {
          // but can't fail on a warning only in this test:
          //failOnWarning(QRegularExpression("Some warning that shouldn't happen here but is OK elsewhere for some reason"));
      }
      

      #3 Context-dependent failOnWarning(const QRegularExpression &)

      + Simple API.
      + Can limit which warnings cause failures.
      + Can be used (called) from QML.
      + Will only ignore the warnings for the scope of the called function, so can be used in init(), initTestCase(), individual test functions, etc.
      - No way to stop failing on a certain warning, and as a result;
      - No way to implement a simplified "bool failOnWarnings" property for TestCase in QML.
      - As behaviour depends on where it's called from, could be slightly more difficult for users to learn.

      void initTestCase()
      {
          // These will last for the duration of the test case.
          startFailingOnWarning(QRegularExpression(".*"));
          // or: startFailingOnWarning(QRegularExpression("Binding loop.*"));
      }
      
      void testSomething()
      {
          // These will last for the duration of the test case.
          startFailingOnWarning(QRegularExpression("Some warning that shouldn't happen here but is OK elsewhere for some reason"));
      }
      

      #4 addFailOnWarning/startFailingOnWarning + removeFailOnWarning/stopFailingOnWarning

      + Simple API.
      + Can limit which warnings cause failures.
      + Can be used (called) from QML.
      + Can stop failing on a certain warning, and as a result;
      + Can implement a simplified "bool failOnWarnings" property for TestCase in QML.

      void initTestCase()
      {
          startFailingOnWarning(QRegularExpression(".*"));
          // or: startFailingOnWarning(QRegularExpression("Binding loop.*"));
      }
      
      void testSomething()
      {
          // This function runs some third party code with a lot of warnings that we can't control...
      
          const auto warningRegex = QRegularExpression(".*");
          // or: const auto warningRegex = QRegularExpression("Binding loop.*");
          stopFailingOnWarning(warningRegex);
          qScopeGuard([=](){ startFailingOnWarning(warningRegex); });
      }
      

      Attachments

        Issue Links

          No reviews matched the request. Check your Options in the drop-down menu of this sections header.

          Activity

            People

              mitch_curtis Mitch Curtis
              mitch_curtis Mitch Curtis
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Gerrit Reviews

                  There are no open Gerrit changes