Uploaded image for project: 'Qt'
  1. Qt
  2. QTBUG-81979

Allow configuring a global QTestLib timeout

    XMLWordPrintable

Details

    • Suggestion
    • Resolution: Unresolved
    • P3: Somewhat important
    • None
    • None
    • Testing: qtestlib
    • None

    Description

      Currently 5000 ms is hard-coded in various places in qtestlib. In addition to that, qtestlib will actually double that value in order to check if a longer timeout value would be required and inform the user of it. So, each time QTRY_VERIFY/QTRY_COMPARE times out, it takes a full 10 seconds to do so.

      That might be fine for whatever cases it was originally written for, but in the majority of my tests, it's just wasting my time. In most cases I do actually need to wait for it to finish because I don't know which check it failed for if I kill the test manually.

      As an example of how long it takes, if I want to run all tests in tst_qquickfiledialogimpl to see which ones fail, it takes 3 minutes with 11 failures:

      09:16:49: Starting /Users/mitch/dev/qt-dev-debug-non-fw/qtdeclarative/tests/auto/quickdialogs/qquickfiledialogimpl/tst_qquickfiledialogimpl...
********* Start testing of tst_QQuickFileDialogImpl *********
Config: Using QtTest library 6.10.0, Qt 6.10.0 (arm64-little_endian-lp64 shared (dynamic) debug build; by Apple LLVM 17.0.0 (clang-1700.0.13.5)), macos 15.5
PASS   : tst_QQuickFileDialogImpl::initTestCase()
PASS   : tst_QQuickFileDialogImpl::defaults()
PASS   : tst_QQuickFileDialogImpl::chooseFileViaStandardButtons()
PASS   : tst_QQuickFileDialogImpl::chooseFileViaDoubleClick()
PASS   : tst_QQuickFileDialogImpl::chooseFileViaTextEdit()
PASS   : tst_QQuickFileDialogImpl::chooseFileViaEnter()
PASS   : tst_QQuickFileDialogImpl::bindCurrentFolder(sub-dir)
FAIL!  : tst_QQuickFileDialogImpl::bindCurrentFolder(sub-dir/sub-file1.txt) 'verifyFileDialogDelegates(dialogHelper.fileDialogListView, expectedVisibleFiles, failureMessage)' returned FALSE. (Mismatch in actual vs expected delegates in fileDialogListView:
    expected: QList("/var/folders/gt/d6wqzmhj7yg3tsxjy7x94wkh0000gn/T/tst_qquickfiledialogimpl-roDrha/sub-dir", "/var/folders/gt/d6wqzmhj7yg3tsxjy7x94wkh0000gn/T/tst_qquickfiledialogimpl-roDrha/file1.txt", "/var/folders/gt/d6wqzmhj7yg3tsxjy7x94wkh0000gn/T/tst_qquickfiledialogimpl-roDrha/file2.txt")
      actual: QList("/private/var/folders/gt/d6wqzmhj7yg3tsxjy7x94wkh0000gn/T/tst_qquickfiledialogimpl-roDrha/sub-dir", "/private/var/folders/gt/d6wqzmhj7yg3tsxjy7x94wkh0000gn/T/tst_qquickfiledialogimpl-roDrha/file1.txt", "/private/var/folders/gt/d6wqzmhj7yg3tsxjy7x94wkh0000gn/T/tst_qquickfiledialogimpl-roDrha/file2.txt"))
   Loc: [/Users/mitch/dev/qt-dev/qtdeclarative/tests/auto/quickdialogs/qquickfiledialogimpl/tst_qquickfiledialogimpl.cpp(490)]
PASS   : tst_QQuickFileDialogImpl::changeFolderViaStandardButtons()
FAIL!  : tst_QQuickFileDialogImpl::changeFolderViaDoubleClick(showDirsFirst=true) 'verifyFileDialogDelegates(dialogHelper.fileDialogListView, expectedVisibleFiles, failureMessage)' returned FALSE. (Mismatch in actual vs expected delegates in fileDialogListView:
    expected: QList("/var/folders/gt/d6wqzmhj7yg3tsxjy7x94wkh0000gn/T/tst_qquickfiledialogimpl-roDrha/sub-dir/sub-sub-dir", "/var/folders/gt/d6wqzmhj7yg3tsxjy7x94wkh0000gn/T/tst_qquickfiledialogimpl-roDrha/sub-dir/sub-file1.txt", "/var/folders/gt/d6wqzmhj7yg3tsxjy7x94wkh0000gn/T/tst_qquickfiledialogimpl-roDrha/sub-dir/sub-file2.txt")
      actual: QList("/private/var/folders/gt/d6wqzmhj7yg3tsxjy7x94wkh0000gn/T/tst_qquickfiledialogimpl-roDrha/sub-dir/sub-sub-dir", "/private/var/folders/gt/d6wqzmhj7yg3tsxjy7x94wkh0000gn/T/tst_qquickfiledialogimpl-roDrha/sub-dir/sub-file1.txt", "/private/var/folders/gt/d6wqzmhj7yg3tsxjy7x94wkh0000gn/T/tst_qquickfiledialogimpl-roDrha/sub-dir/sub-file2.txt"))
   Loc: [/Users/mitch/dev/qt-dev/qtdeclarative/tests/auto/quickdialogs/qquickfiledialogimpl/tst_qquickfiledialogimpl.cpp(555)]
FAIL!  : tst_QQuickFileDialogImpl::changeFolderViaDoubleClick(showDirsFirst=false) 'verifyFileDialogDelegates(dialogHelper.fileDialogListView, expectedVisibleFiles, failureMessage)' returned FALSE. (Mismatch in actual vs expected delegates in fileDialogListView:
    expected: QList("/var/folders/gt/d6wqzmhj7yg3tsxjy7x94wkh0000gn/T/tst_qquickfiledialogimpl-roDrha/sub-dir/sub-file1.txt", "/var/folders/gt/d6wqzmhj7yg3tsxjy7x94wkh0000gn/T/tst_qquickfiledialogimpl-roDrha/sub-dir/sub-file2.txt", "/var/folders/gt/d6wqzmhj7yg3tsxjy7x94wkh0000gn/T/tst_qquickfiledialogimpl-roDrha/sub-dir/sub-sub-dir")
      actual: QList("/private/var/folders/gt/d6wqzmhj7yg3tsxjy7x94wkh0000gn/T/tst_qquickfiledialogimpl-roDrha/sub-dir/sub-file1.txt", "/private/var/folders/gt/d6wqzmhj7yg3tsxjy7x94wkh0000gn/T/tst_qquickfiledialogimpl-roDrha/sub-dir/sub-file2.txt", "/private/var/folders/gt/d6wqzmhj7yg3tsxjy7x94wkh0000gn/T/tst_qquickfiledialogimpl-roDrha/sub-dir/sub-sub-dir"))
   Loc: [/Users/mitch/dev/qt-dev/qtdeclarative/tests/auto/quickdialogs/qquickfiledialogimpl/tst_qquickfiledialogimpl.cpp(555)]
PASS   : tst_QQuickFileDialogImpl::chooseFolderViaTextEdit()
FAIL!  : tst_QQuickFileDialogImpl::chooseFolderViaEnter() 'verifyFileDialogDelegates(dialogHelper.fileDialogListView, tempSubDirExpectedVisibleFiles(ShowDirectoriesFirst), failureMessage)' returned FALSE. (Mismatch in actual vs expected delegates in fileDialogListView:
    expected: QList("/var/folders/gt/d6wqzmhj7yg3tsxjy7x94wkh0000gn/T/tst_qquickfiledialogimpl-roDrha/sub-dir/sub-sub-dir", "/var/folders/gt/d6wqzmhj7yg3tsxjy7x94wkh0000gn/T/tst_qquickfiledialogimpl-roDrha/sub-dir/sub-file1.txt", "/var/folders/gt/d6wqzmhj7yg3tsxjy7x94wkh0000gn/T/tst_qquickfiledialogimpl-roDrha/sub-dir/sub-file2.txt")
      actual: QList("/private/var/folders/gt/d6wqzmhj7yg3tsxjy7x94wkh0000gn/T/tst_qquickfiledialogimpl-roDrha/sub-dir/sub-sub-dir", "/private/var/folders/gt/d6wqzmhj7yg3tsxjy7x94wkh0000gn/T/tst_qquickfiledialogimpl-roDrha/sub-dir/sub-file1.txt", "/private/var/folders/gt/d6wqzmhj7yg3tsxjy7x94wkh0000gn/T/tst_qquickfiledialogimpl-roDrha/sub-dir/sub-file2.txt"))
   Loc: [/Users/mitch/dev/qt-dev/qtdeclarative/tests/auto/quickdialogs/qquickfiledialogimpl/tst_qquickfiledialogimpl.cpp(622)]
PASS   : tst_QQuickFileDialogImpl::chooseFileAndThenFolderViaTextEdit()
PASS   : tst_QQuickFileDialogImpl::cancelDialogWhileTextEditHasFocus()
PASS   : tst_QQuickFileDialogImpl::closingDialogCancels()
PASS   : tst_QQuickFileDialogImpl::goUp(showDirsFirst=true)
PASS   : tst_QQuickFileDialogImpl::goUp(showDirsFirst=false)
PASS   : tst_QQuickFileDialogImpl::goUpWhileTextEditHasFocus()
PASS   : tst_QQuickFileDialogImpl::goIntoLargeFolder()
PASS   : tst_QQuickFileDialogImpl::goUpIntoLargeFolder()
PASS   : tst_QQuickFileDialogImpl::keyAndShortcutHandling()
FAIL!  : tst_QQuickFileDialogImpl::bindNameFilters() 'verifyFileDialogDelegates(dialogHelper.fileDialogListView, tempDirExpectedVisibleFiles(ShowDirectoriesFirst), failureMessage)' returned FALSE. (Mismatch in actual vs expected delegates in fileDialogListView:
    expected: QList("/var/folders/gt/d6wqzmhj7yg3tsxjy7x94wkh0000gn/T/tst_qquickfiledialogimpl-roDrha/sub-dir", "/var/folders/gt/d6wqzmhj7yg3tsxjy7x94wkh0000gn/T/tst_qquickfiledialogimpl-roDrha/file1.txt", "/var/folders/gt/d6wqzmhj7yg3tsxjy7x94wkh0000gn/T/tst_qquickfiledialogimpl-roDrha/file2.txt")
      actual: QList("/private/var/folders/gt/d6wqzmhj7yg3tsxjy7x94wkh0000gn/T/tst_qquickfiledialogimpl-roDrha/sub-dir", "/private/var/folders/gt/d6wqzmhj7yg3tsxjy7x94wkh0000gn/T/tst_qquickfiledialogimpl-roDrha/file1.txt", "/private/var/folders/gt/d6wqzmhj7yg3tsxjy7x94wkh0000gn/T/tst_qquickfiledialogimpl-roDrha/file2.txt"))
   Loc: [/Users/mitch/dev/qt-dev/qtdeclarative/tests/auto/quickdialogs/qquickfiledialogimpl/tst_qquickfiledialogimpl.cpp(963)]
FAIL!  : tst_QQuickFileDialogImpl::changeNameFilters() 'verifyFileDialogDelegates(dialogHelper.fileDialogListView, tempDirExpectedVisibleFiles(ShowDirectoriesFirst), failureMessage)' returned FALSE. (Mismatch in actual vs expected delegates in fileDialogListView:
    expected: QList("/var/folders/gt/d6wqzmhj7yg3tsxjy7x94wkh0000gn/T/tst_qquickfiledialogimpl-roDrha/sub-dir", "/var/folders/gt/d6wqzmhj7yg3tsxjy7x94wkh0000gn/T/tst_qquickfiledialogimpl-roDrha/file1.txt", "/var/folders/gt/d6wqzmhj7yg3tsxjy7x94wkh0000gn/T/tst_qquickfiledialogimpl-roDrha/file2.txt")
      actual: QList("/private/var/folders/gt/d6wqzmhj7yg3tsxjy7x94wkh0000gn/T/tst_qquickfiledialogimpl-roDrha/sub-dir", "/private/var/folders/gt/d6wqzmhj7yg3tsxjy7x94wkh0000gn/T/tst_qquickfiledialogimpl-roDrha/file1.txt", "/private/var/folders/gt/d6wqzmhj7yg3tsxjy7x94wkh0000gn/T/tst_qquickfiledialogimpl-roDrha/file2.txt"))
   Loc: [/Users/mitch/dev/qt-dev/qtdeclarative/tests/auto/quickdialogs/qquickfiledialogimpl/tst_qquickfiledialogimpl.cpp(1017)]
FAIL!  : tst_QQuickFileDialogImpl::changeNameFiltersAfterChangingFolder() 'verifyFileDialogDelegates(dialogHelper.fileDialogListView, tempDirExpectedVisibleFiles(ShowDirectoriesFirst), failureMessage)' returned FALSE. (Mismatch in actual vs expected delegates in fileDialogListView:
    expected: QList("/var/folders/gt/d6wqzmhj7yg3tsxjy7x94wkh0000gn/T/tst_qquickfiledialogimpl-roDrha/sub-dir", "/var/folders/gt/d6wqzmhj7yg3tsxjy7x94wkh0000gn/T/tst_qquickfiledialogimpl-roDrha/file1.txt", "/var/folders/gt/d6wqzmhj7yg3tsxjy7x94wkh0000gn/T/tst_qquickfiledialogimpl-roDrha/file2.txt")
      actual: QList("/private/var/folders/gt/d6wqzmhj7yg3tsxjy7x94wkh0000gn/T/tst_qquickfiledialogimpl-roDrha/sub-dir", "/private/var/folders/gt/d6wqzmhj7yg3tsxjy7x94wkh0000gn/T/tst_qquickfiledialogimpl-roDrha/file1.txt", "/private/var/folders/gt/d6wqzmhj7yg3tsxjy7x94wkh0000gn/T/tst_qquickfiledialogimpl-roDrha/file2.txt"))
   Loc: [/Users/mitch/dev/qt-dev/qtdeclarative/tests/auto/quickdialogs/qquickfiledialogimpl/tst_qquickfiledialogimpl.cpp(1071)]
BFAIL  : tst_QQuickFileDialogImpl::tabFocusNavigation() 'dialogHelper.popupWindow()->activeFocusItem() == expectedFocusItem' returned FALSE. (
   Actual:   Button_QMLTYPE_821(0x611001926740, parent=0x6030005afc70, geometry=112,0 100x40, z=1)
   Expected: Button_QMLTYPE_821(0x611000a81280, parent=0x6030005afc70, geometry=0,0 100x40, z=1))
   Loc: [/Users/mitch/dev/qt-dev/qtdeclarative/tests/auto/quickdialogs/qquickfiledialogimpl/tst_qquickfiledialogimpl.cpp(1148)]
PASS   : tst_QQuickFileDialogImpl::acceptRejectLabel()
PASS   : tst_QQuickFileDialogImpl::bindTitle()
PASS   : tst_QQuickFileDialogImpl::itemsDisabledWhenNecessary()
FAIL!  : tst_QQuickFileDialogImpl::fileMode(OpenFile) Compared values are not the same
   Actual   (breadcrumbBar->textField()->text()): "/private/var/folders/gt/d6wqzmhj7yg3tsxjy7x94wkh0000gn/T/tst_qquickfiledialogimpl-roDrha"
   Expected (tempDir.path())                    : "/var/folders/gt/d6wqzmhj7yg3tsxjy7x94wkh0000gn/T/tst_qquickfiledialogimpl-roDrha"
   Loc: [/Users/mitch/dev/qt-dev/qtdeclarative/tests/auto/quickdialogs/qquickfiledialogimpl/tst_qquickfiledialogimpl.cpp(1346)]
BFAIL  : tst_QQuickFileDialogImpl::fileMode(OpenFiles) Compared values are not the same
   Actual   (QFileInfo(dialogHelper.dialog->currentFile().toLocalFile()).canonicalFilePath())        : "/private/var/folders/gt/d6wqzmhj7yg3tsxjy7x94wkh0000gn/T/tst_qquickfiledialogimpl-roDrha/file2.txt"
   Expected (QFileInfo(QUrl::fromLocalFile(tempFile1->fileName()).toLocalFile()).canonicalFilePath()): "/private/var/folders/gt/d6wqzmhj7yg3tsxjy7x94wkh0000gn/T/tst_qquickfiledialogimpl-roDrha/file1.txt"
   Loc: [/Users/mitch/dev/qt-dev/qtdeclarative/tests/auto/quickdialogs/qquickfiledialogimpl/tst_qquickfiledialogimpl.cpp(1312)]
PASS   : tst_QQuickFileDialogImpl::fileMode(SaveFile)
PASS   : tst_QQuickFileDialogImpl::defaultSuffix(txt)
PASS   : tst_QQuickFileDialogImpl::defaultSuffix(.txt)
PASS   : tst_QQuickFileDialogImpl::done(Accepted)
PASS   : tst_QQuickFileDialogImpl::done(Rejected)
FAIL!  : tst_QQuickFileDialogImpl::setSelectedFile(OpenFile) Compared values are not the same
   Actual   (dialogHelper.fileDialogListView->currentIndex()): -1
   Expected (1)                                              : 1
   Loc: [/Users/mitch/dev/qt-dev/qtdeclarative/tests/auto/quickdialogs/qquickfiledialogimpl/tst_qquickfiledialogimpl.cpp(1455)]
FAIL!  : tst_QQuickFileDialogImpl::setSelectedFile(OpenFiles) Compared values are not the same
   Actual   (dialogHelper.fileDialogListView->currentIndex()): -1
   Expected (1)                                              : 1
   Loc: [/Users/mitch/dev/qt-dev/qtdeclarative/tests/auto/quickdialogs/qquickfiledialogimpl/tst_qquickfiledialogimpl.cpp(1455)]
FAIL!  : tst_QQuickFileDialogImpl::setSelectedFile(SaveFile) Compared values are not the same
   Actual   (dialogHelper.fileDialogListView->currentIndex()): -1
   Expected (1)                                              : 1
   Loc: [/Users/mitch/dev/qt-dev/qtdeclarative/tests/auto/quickdialogs/qquickfiledialogimpl/tst_qquickfiledialogimpl.cpp(1455)]
PASS   : tst_QQuickFileDialogImpl::selectNewFileViaTextField(OpenFile)
PASS   : tst_QQuickFileDialogImpl::selectNewFileViaTextField(OpenFiles)
PASS   : tst_QQuickFileDialogImpl::selectNewFileViaTextField(SaveFile)
PASS   : tst_QQuickFileDialogImpl::selectExistingFileShouldWarnUserWhenFileModeEqualsSaveFile()
PASS   : tst_QQuickFileDialogImpl::fileNameTextFieldOnlyChangesWhenSelectingFiles()
PASS   : tst_QQuickFileDialogImpl::setSchemeForSelectedFile()
PASS   : tst_QQuickFileDialogImpl::reopenAfterHideEvent()
PASS   : tst_QQuickFileDialogImpl::sidebarStandardPaths()
PASS   : tst_QQuickFileDialogImpl::popupType()
PASS   : tst_QQuickFileDialogImpl::cleanupTestCase()
Totals: 36 passed, 11 failed, 0 skipped, 2 blacklisted, 181790ms
********* Finished testing of tst_QQuickFileDialogImpl *********
09:19:51: /Users/mitch/dev/qt-dev-debug-non-fw/qtdeclarative/tests/auto/quickdialogs/qquickfiledialogimpl/tst_qquickfiledialogimpl exited with code 11

      The full test without failures usually takes ~40 seconds:

      11:42:43: Starting /Users/mitch/dev/qt-dev-debug-non-fw/qtdeclarative/tests/auto/quickdialogs/qquickfiledialogimpl/tst_qquickfiledialogimpl...
********* Start testing of tst_QQuickFileDialogImpl *********
Config: Using QtTest library 6.10.0, Qt 6.10.0 (arm64-little_endian-lp64 shared (dynamic) debug build; by Apple LLVM 17.0.0 (clang-1700.0.13.5)), macos 15.5
PASS   : tst_QQuickFileDialogImpl::initTestCase()
[...]
PASS   : tst_QQuickFileDialogImpl::cleanupTestCase()
Totals: 47 passed, 0 failed, 0 skipped, 2 blacklisted, 42366ms
********* Finished testing of tst_QQuickFileDialogImpl *********
11:43:26: /Users/mitch/dev/qt-dev-debug-non-fw/qtdeclarative/tests/auto/quickdialogs/qquickfiledialogimpl/tst_qquickfiledialogimpl exited with code 0

      I would like to be able to configure the default value once (e.g. a Qt configure argument) and special-case the longer waits manually with QTRY_VERIFY_WITH_TIMEOUT, so that my code isn't full of QTRY_VERIFY_WITH_TIMEOUT calls, as QTRY_VERIFY_WITH_TIMEOUT is a very long name. Note that an application argument would be kinda annoying, as it would have to be set every time someone sets their project up in Creator. Another possibility would be a static setter in e.g. QTest.

      Attachments

        Issue Links

          is cloned by

          Suggestion - Public suggestions QTBUG-110314 Add QTestRandomNumber that tracks the seed used for reproducability

          • P3: Somewhat important - Should be fixed, but doesn't affect the release in any way.
          • Open
          relates to

          Task - A task that needs to be done. QTBUG-95421 need the ability to delay (increment timestamps of) touch events, as with mouse events

          • Not Evaluated - Not yet evaluated/prioritized
          • Reported

          Suggestion - Public suggestions QTBUG-104087 qtestlib watchdog timeout should be runtime-configurable

          • P3: Somewhat important - Should be fixed, but doesn't affect the release in any way.
          • Reported
          For Gerrit Dashboard: QTBUG-81979
          # Subject Branch Project Status CR V
          634604,18 Allow configuring of a global default for QTRY_* timeouts in tests dev qt/qtbase Status: NEW 0 0
          648230,3 Reduce default QTest QTRY_* timeout to 1000ms dev qt/qtbase Status: NEW 0 -1
          648330,9 Make QTest::defaultTryTimeout public dev qt/qtbase Status: NEW 0 0
          656607,1 QQmlDataTest: drop default try timeout from 5 seconds to 1 dev qt/qtdeclarative Status: NEW 0 0
          636223,5 Refactor default QTRY_* timeout value into one place dev qt/qtbase Status: MERGED +2 0

          Activity

            People

              mitch_curtis Mitch Curtis
              mitch_curtis Mitch Curtis
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

                Created:
                Updated:

                Gerrit Reviews

                  There are 4 open Gerrit changes