Uploaded image for project: 'Qt'
  1. Qt
  2. QTBUG-105181

Non-portable or problematic tests in qtbase test suite

    XMLWordPrintable

Details

    • Bug
    • Resolution: Unresolved
    • P2: Important
    • None
    • 6.3.1
    • Packaging & Installer
    • None
    • Guix System
    • Linux/X11

    Description

      Hello,

      Working on upgrading qtbase package in Guix and enabling the test suite, I've come across the following tests which failed due to assumptions about the file system layout, networking, or other features of the underlying test system:

      Total Test time (real) = 102.01 sec
      
      The following tests FAILED:
                6 - tst_moc (Failed)
                7 - tst_rcc (Failed)
               60 - tst_qfile (Failed)
               69 - tst_qtemporarydir (Failed)
               70 - tst_qtemporaryfile (Failed)
               76 - tst_qdir (Failed)
               77 - tst_qresourceengine (Failed)
              173 - tst_qdatetime (Failed)
              176 - tst_qtimezone (Failed)
              249 - mockplugins (Failed)
              250 - test_import_plugins (Not Run)
              254 - test_plugin_flavor_static (Failed)
              255 - test_plugin_flavor_shared (Failed)
              256 - test_plugin_flavor_derived_from_qt_type (Failed)
              257 - tst_qaddpreroutine (Failed)
              258 - test_static_resources (Failed)
              259 - test_generating_cpp_exports (Failed)
              334 - tst_qglyphrun (Failed)
              364 - tst_qx11info (Failed)
              373 - tst_qdnslookup (Failed)
              415 - tst_qfiledialog (Failed)
              430 - tst_qgraphicswidget (Failed)
              466 - tst_qcompleter (Failed)
      

      They are a very small minority of tests that wouldn't work in the isolated build container of Guix. I'm sending the list of disabled tests and my summary explanations of their failures below. I hope this can be useful in trimming down problematic tests or rewriting them in a more portable way, making it easier for downstream packagers to enable the test suite in the future.

      ;; The tests expect to find the modules provided by this
      ;; package; extend the environment variables needed to do so.
      (setenv "CMAKE_PREFIX_PATH"
              (string-append #$output
                             ":" (getenv "CMAKE_PREFIX_PATH")))
      (setenv "QMAKEPATH" (string-append #$output "/lib/qt6"))
      (setenv "QML2_IMPORT_PATH"
              (string-append #$output "/lib/qt6/qml"))
      (setenv "QT_PLUGIN_PATH"
              (string-append #$output "/lib/qt6/plugins"))
      (setenv "QT_QPA_PLATFORM" "offscreen")
      ;; Skip tests known to fail on GNU/Linux, in a CI context or
      ;; due to bitness (see: https://code.qt.io/cgit/qt/qtbase.git
      ;; /tree/src/testlib/qtestblacklist.cpp).
      (setenv "QTEST_ENVIRONMENT" "linux ci 32bit")
      (setenv "HOME" "/tmp") ;some tests require a writable HOME
      (invoke
       "xvfb-run" "ctest" "--output-on-failure"
       "-j" (if parallel-tests?
                (number->string (parallel-job-count))
                "1")
       "-E"                 ;disable problematic tests
       (string-append
        "("
        (string-join
         (list
          ;; The 'tst_moc' test fails with "'fi.exists()' returned FALSE".
          "tst_moc"
          ;; The 'test_rcc' test fails on a comparison:
          ;; <<<<<< actual
          ;; 0x0,0x0,0x0,0x0,0x0,0x0,0x3,0xe8,
          ;; ======
          ;; 0x0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,
          ;; >>>>>> expected
          "tst_rcc"
          ;; The 'tst_qtemporarydir' and 'tst_qtemporaryfile'
          ;; tests depend on '/home' not being writable.
          "tst_qtemporarydir"
          "tst_qtemporaryfile"
          ;; The 'tst_qdir' compares two directories which are
          ;; unexpectedly different when inside the build
          ;; container.
          "tst_qdir"
          ;; This checks the last modified time of '/', and fails
          ;; because Epoch 0 is considered to be invalid.
          "tst_qresourceengine"
          ;; The 'tst_qfilesystemwatcher' installs a watcher on
          ;; '/home', which doesn't exist in the build container.
          "tst_qfilesystemwatcher"
          ;; The 'mockplugins' test fail following error: "Unknown
          ;; platform linux-g++", and the other plugin tests
          ;; depend on it.
          "mockplugins"
          "test_plugin_flavor.*"
          ;; The 'test_import_plugins' fails with "Could NOT find
          ;; Qt6MockPlugins1".
          "test_import_plugins"
          ;; The 'tst_QTimeZone::systemZone' validates the
          ;; currently set timezone and fails.
          "tst_qtimezone"
          ;; The 'tst_qdatetime' fails with:
          ;; FAIL!  : tst_QDateTime::offsetFromUtc() Compared values are not the same
          ;; Actual   (dt5.offsetFromUtc()): 0
          ;; Expected (46800)              : 46800
          "tst_qdatetime"
          ;; The 'tst_QSettings::fromFile' assumes the data
          ;; location to be relative to the root directory and
          ;; fails.
          "tst_qsettings"
          ;; The 'tst_qaddpreroutine',
          ;; 'test_generating_cpp_exports' and
          ;; 'test_static_resources' tests fail with: "Unknown
          ;; platform linux-g++.
          "tst_qaddpreroutine"
          "test_generating_cpp_exports"
          "test_static_resources"
          ;; The 'tst_qfile' fails since there is no /home in the
          ;; build container.
          "tst_qfile"
          ;; The 'tst_QGlyphRun::mixedScripts' test fails with:
          ;; Actual   (glyphRuns.size()): 1
          ;; Expected (2)               : 2
          "tst_qglyphrun"
          ;; The 'tst_qx11info' test fails with "Internal error:
          ;; QPA plugin doesn't implement generatePeekerId",
          ;; likely requires a real display.
          "tst_qx11info"
          ;; The 'tst_qgraphicswidget' test fails because "This
          ;; plugin does not support propagateSizeHints".
          "tst_qgraphicswidget"
          ;; The 'tst_qdnslookup' test requires networking.
          "tst_qdnslookup"
          ;; The 'tst_qcompleter' and 'tst_QFiledialog::completer'
          ;; attempt to complete paths they assume exist, such as
          ;; "/home", "/etc" or "/root" and fail.
          "tst_qcompleter"
          "tst_qfiledialog") "|")
        ")"))
      
      

      The full build log is included as an attachment.

      Thanks!

      Attachments

        No reviews matched the request. Check your Options in the drop-down menu of this sections header.

        Activity

          People

            releaseteam Qt Release Team
            apteryx Maxim Cournoyer
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

              Created:
              Updated:

              Gerrit Reviews

                There are no open Gerrit changes