Uploaded image for project: 'Qt'
  1. Qt
  2. QTBUG-22453

Increase effective test coverage in qtbase by disabling only failing test functions

    XMLWordPrintable

Details

    • Task
    • Resolution: Done
    • Not Evaluated
    • 5.0.0
    • None
    • Testing: qtestlib
    • None
    • Linux (since this is currently the only platform in Qt 5 CI)

    • See comment

    Description

      Currently the insignificant_test flag allows us to disable (i.e. ignore failures in) a test case as a whole, but not individual test functions. This reduces the effective test coverage more than necessary since many "unproblematic" test functions would not be allowed to block CI even when they should. In other words: a real bug in Qt may in this case slip through CI, even when it was actually caught by an existing test function.

      A rough analysis of recent pulse logs in qtbase#master (runs 730..829) shows the following situation for test cases that contain test functions of only two types, those that are passing in all runs and those that are failing in all runs:

      test case failing functions total functions
      tst_qabstractprintdialog 2 3
      tst_exceptionsafety_objects 1 5
      tst_qopengl 2 5
      tst_qinputcontext 3 6
      tst_qglthreads 1 7
      tst_qdialog 1 14
      tst_qprinter 1 18
      tst_qtextbrowser 10 18
      tst_qdoublespinbox 1 23
      tst_qmenu 2 24
      tst_qlistview 1 44
      tst_qgraphicswidget 2 67
      tst_qgraphicsscene 1 80
      tst_qvariant 1 85
      tst_qgraphicsview 2 91
      tst_qgraphicsitem 1 168
      Total: 32 658

      This essentially means that for these test cases, 626 out of 658 test functions (95%) are turned off for no good reason.

      The ideal way to remedy this situation is of course to fix the failing test functions, and get rid of the insignificant_test flag without any further ado.

      In the meantime, until the proper fix is in place, I propose to remove the insignificant_test flag, and instead disable the failing test functions individually using QEXPECT_FAIL. (Note that the failing test functions in this case are assumed to fail in every run, so we would not expect any XPASS'es).

      Attachments

        For Gerrit Dashboard: QTBUG-22453
        # Subject Branch Project Status CR V

        Activity

          People

            bdo Jo Asplin (Inactive)
            bdo Jo Asplin (Inactive)
            Votes:
            0 Vote for this issue
            Watchers:
            0 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: