Uploaded image for project: 'Qt Quality Assurance Infrastructure'
  1. Qt Quality Assurance Infrastructure
  2. QTQAINFRA-5226

When a test times out the test runner only reruns the single affected test and skips all following tests

    XMLWordPrintable

Details

    • Task
    • Resolution: Fixed
    • P2: Important
    • None
    • None
    • Test scripts
    • None
    • 655bed656 (dev)

    Description

      Given a test, Test, with the tests 'a', 'b', 'c' and 'd'.
      If 'c' times out then it is labelled FAIL and the test returns.
      The test-runner sees this and reruns 'c'. If 'c' passes then all is good.
      Except 'd' never ran.

      Real example below:
      ctrl-f "FAIL! : tst_QTimer::singleShotToFunctors"

      https://testresults.qt.io/coin/api/results/qt/qtbase/7ad41ed53dfde061cd6417f248d8eb0fe4b88c2f/LinuxUbuntu_22_04x86_64LinuxQEMUarm64GCCqtci-linux-Ubuntu-22.04-x86_64-51-ba4346Sccache_UseConfigure/77308ddc55587491415901ea852c4ef0316f040c/test_1678858006/log.txt.gz

      agent:2022/10/14 08:08:03 build.go:394:         Start  19: tst_qtimer
      agent:2022/10/14 08:08:03 build.go:394: 19: Test command: /opt/cmake-3.21.3/bin/cmake "-P" "/home/qt/work/qt/qtbase_standalone_tests/tests/auto/corelib/kernel/qtimer/tst_qtimerWrapperRelWithDebInfo.cmake"
      agent:2022/10/14 08:08:03 build.go:394: 19: Test timeout computed to be: 1500
      agent:2022/10/14 08:08:03 build.go:394: 19: qt-testrunner.py     INFO: Will write XML test logs to directory COIN_CTEST_RESULTSDIR=/home/qt/work/testresults
      agent:2022/10/14 08:08:03 build.go:394: 19: qt-testrunner.py    DEBUG: Running test command line: ['/home/qt/work/qt/qtbase_standalone_tests/tests/auto/corelib/kernel/qtimer/tst_qtimer', '-o', '/home/qt/work/testresults/tst_qtimer-1665734883650.xml,xml', '-o', '/home/qt/work/testresults/tst_qtimer-1665734883650.junit.xml,junitxml', '-o', '-,txt']
      agent:2022/10/14 08:08:03 build.go:394: 19: ********* Start testing of tst_QTimer *********
      agent:2022/10/14 08:08:03 build.go:394: 19: Config: Using QtTest library 6.5.0, Qt 6.5.0 (arm64-little_endian-lp64 shared (dynamic) release build; by GCC 10.2.0), b2qt 3.2
      agent:2022/10/14 08:08:03 build.go:394: 19: PASS   : tst_QTimer::initTestCase()
      agent:2022/10/14 08:08:03 build.go:394: 19: PASS   : tst_QTimer::zeroTimer()
      agent:2022/10/14 08:08:04 build.go:394: 19: PASS   : tst_QTimer::singleShotTimeout()
      agent:2022/10/14 08:08:04 build.go:394: 19: PASS   : tst_QTimer::timeout()
      agent:2022/10/14 08:08:05 build.go:394: 19: PASS   : tst_QTimer::remainingTime()
      agent:2022/10/14 08:08:05 build.go:394: 19: PASS   : tst_QTimer::remainingTimeInitial(precise time 0ms)
      agent:2022/10/14 08:08:05 build.go:394: 19: PASS   : tst_QTimer::remainingTimeInitial(precise time 1ms)
      agent:2022/10/14 08:08:05 build.go:394: 19: PASS   : tst_QTimer::remainingTimeInitial(precise time 10ms)
      agent:2022/10/14 08:08:05 build.go:394: 19: PASS   : tst_QTimer::remainingTimeInitial(coarse time 0ms)
      agent:2022/10/14 08:08:05 build.go:394: 19: PASS   : tst_QTimer::remainingTimeInitial(coarse time 1ms)
      agent:2022/10/14 08:08:05 build.go:394: 19: PASS   : tst_QTimer::remainingTimeInitial(coarse time 10ms)
      agent:2022/10/14 08:08:05 build.go:394: 19: PASS   : tst_QTimer::remainingTimeDuringActivation(repeating)
      agent:2022/10/14 08:08:05 build.go:394: 19: PASS   : tst_QTimer::remainingTimeDuringActivation(single-shot)
      agent:2022/10/14 08:08:05 build.go:394: 19: PASS   : tst_QTimer::basic_chrono()
      agent:2022/10/14 08:08:06 build.go:394: 19: PASS   : tst_QTimer::livelock(zero timer)
      agent:2022/10/14 08:08:06 build.go:394: 19: PASS   : tst_QTimer::livelock(non-zero timer)
      agent:2022/10/14 08:08:06 build.go:394: 19: PASS   : tst_QTimer::livelock(longer than sleep)
      agent:2022/10/14 08:08:06 build.go:394: 19: PASS   : tst_QTimer::timerInfiniteRecursion(zero timer)
      agent:2022/10/14 08:08:06 build.go:394: 19: PASS   : tst_QTimer::timerInfiniteRecursion(non-zero timer)
      agent:2022/10/14 08:08:06 build.go:394: 19: PASS   : tst_QTimer::timerInfiniteRecursion(10ms timer)
      agent:2022/10/14 08:08:06 build.go:394: 19: PASS   : tst_QTimer::timerInfiniteRecursion(11ms timer)
      agent:2022/10/14 08:08:07 build.go:394: 19: PASS   : tst_QTimer::timerInfiniteRecursion(100ms timer)
      agent:2022/10/14 08:08:10 build.go:394: 19: PASS   : tst_QTimer::timerInfiniteRecursion(1s timer)
      agent:2022/10/14 08:08:10 build.go:394: 19: PASS   : tst_QTimer::recurringTimer(zero timer, don't recurse)
      agent:2022/10/14 08:08:10 build.go:394: 19: PASS   : tst_QTimer::recurringTimer(zero timer, recurse)
      agent:2022/10/14 08:08:10 build.go:394: 19: PASS   : tst_QTimer::recurringTimer(non-zero timer, don't recurse)
      agent:2022/10/14 08:08:11 build.go:394: 19: PASS   : tst_QTimer::recurringTimer(non-zero timer, recurse)
      agent:2022/10/14 08:08:11 build.go:394: 19: PASS   : tst_QTimer::deleteLaterOnQTimer()
      agent:2022/10/14 08:08:11 build.go:394: 19: QWARN  : tst_QTimer::moveToThread() QObject::killTimer: Timers cannot be stopped from another thread
      agent:2022/10/14 08:08:11 build.go:394: 19: QWARN  : tst_QTimer::moveToThread() QObject::~QObject: Timers cannot be stopped from another thread
      agent:2022/10/14 08:08:11 build.go:394: 19: PASS   : tst_QTimer::moveToThread()
      agent:2022/10/14 08:08:22 build.go:394: 19: PASS   : tst_QTimer::restartedTimerFiresTooSoon()
      agent:2022/10/14 08:08:22 build.go:394: 19: PASS   : tst_QTimer::timerFiresOnlyOncePerProcessEvents(zero timer)
      agent:2022/10/14 08:08:23 build.go:394: 19: PASS   : tst_QTimer::timerFiresOnlyOncePerProcessEvents(non-zero timer)
      agent:2022/10/14 08:08:23 build.go:394: 19: QWARN  : tst_QTimer::timerIdPersistsAfterThreadExit() QObject::killTimer: Timers cannot be stopped from another thread
      agent:2022/10/14 08:08:23 build.go:394: 19: QWARN  : tst_QTimer::timerIdPersistsAfterThreadExit() QObject::~QObject: Timers cannot be stopped from another thread
      agent:2022/10/14 08:08:23 build.go:394: 19: PASS   : tst_QTimer::timerIdPersistsAfterThreadExit()
      agent:2022/10/14 08:08:23 build.go:394: 19: PASS   : tst_QTimer::cancelLongTimer()
      agent:2022/10/14 08:08:24 build.go:394: 19: PASS   : tst_QTimer::singleShotStaticFunctionZeroTimeout()
      agent:2022/10/14 08:08:25 build.go:394: 19: PASS   : tst_QTimer::recurseOnTimeoutAndStopTimer()
      agent:2022/10/14 08:13:25 build.go:394: 19: 
      agent:2022/10/14 08:13:25 build.go:394: 19:          singleShotToFunctors function time: 300002ms, total time: 321365ms
      agent:2022/10/14 08:13:25 build.go:394: 19: 
      agent:2022/10/14 08:13:25 build.go:394: 19: === Stack trace ===
      agent:2022/10/14 08:13:25 build.go:394: 19: Could not attach to process.  If your uid matches the uid of the target
      agent:2022/10/14 08:13:25 build.go:394: 19: process, check the setting of /proc/sys/kernel/yama/ptrace_scope, or try
      agent:2022/10/14 08:13:25 build.go:394: 19: again as the root user.  For more details, see /etc/sysctl.d/10-ptrace.conf
      agent:2022/10/14 08:13:25 build.go:394: 19: ptrace: Inappropriate ioctl for device.
      agent:2022/10/14 08:13:25 build.go:394: 19: === End of stack trace ===
      agent:2022/10/14 08:13:25 build.go:394: 19: QFATAL : tst_QTimer::singleShotToFunctors() Test function timed out
      agent:2022/10/14 08:13:25 build.go:394: 19: QObject: Cannot create children for a parent that is in a different thread.
      agent:2022/10/14 08:13:25 build.go:394: 19: (Parent is QCoreApplication(0x5501833ad8), parent's thread is QThread(0x550004d9a0), current thread is QThread(0x5500055390)
      agent:2022/10/14 08:13:25 build.go:394: 19: FAIL!  : tst_QTimer::singleShotToFunctors() Received a fatal error.
      agent:2022/10/14 08:13:25 build.go:394: 19: Totals: 36 passed, 1 failed, 0 skipped, 0 blacklisted, 322019ms
      agent:2022/10/14 08:13:25 build.go:394: 19: ********* Finished testing of tst_QTimer *********
      agent:2022/10/14 08:13:25 build.go:394: 19: Received signal 6 (SIGABRT)
      agent:2022/10/14 08:13:25 build.go:394: 19:          singleShotToFunctors function time: 300659ms, total time: 322022ms
      agent:2022/10/14 08:13:25 build.go:394: 19: 
      agent:2022/10/14 08:13:25 build.go:394: 19: === Stack trace ===
      agent:2022/10/14 08:13:25 build.go:394: 19: Could not attach to process.  If your uid matches the uid of the target
      agent:2022/10/14 08:13:25 build.go:394: 19: process, check the setting of /proc/sys/kernel/yama/ptrace_scope, or try
      agent:2022/10/14 08:13:25 build.go:394: 19: again as the root user.  For more details, see /etc/sysctl.d/10-ptrace.conf
      agent:2022/10/14 08:13:25 build.go:394: 19: ptrace: Inappropriate ioctl for device.
      agent:2022/10/14 08:13:25 build.go:394: 19: === End of stack trace ===
      agent:2022/10/14 08:13:25 build.go:394: 19: qemu: uncaught target signal 6 (Aborted) - core dumped
      agent:2022/10/14 08:13:25 build.go:394: 19: qt-testrunner.py     INFO: Test process exited with code: -6
      agent:2022/10/14 08:13:25 build.go:394: 19: qt-testrunner.py     INFO: Parsed XML file /home/qt/work/testresults/tst_qtimer-1665734883650.xml in 0.001 seconds
      agent:2022/10/14 08:13:25 build.go:394: 19: qt-testrunner.py     INFO: Found 36 passes and 1 failures
      agent:2022/10/14 08:13:25 build.go:394: 19: qt-testrunner.py     INFO: Some tests failed, will re-run at most 5 times.
      agent:2022/10/14 08:13:25 build.go:394: 19: 
      agent:2022/10/14 08:13:25 build.go:394: 19: qt-testrunner.py     INFO: Re-running testcase: singleShotToFunctors
      agent:2022/10/14 08:13:25 build.go:394: 19: qt-testrunner.py    DEBUG: Running test command line: ['/home/qt/work/qt/qtbase_standalone_tests/tests/auto/corelib/kernel/qtimer/tst_qtimer', '-o', '/home/qt/work/testresults/tst_qtimer-1665735205929.xml,xml', '-o', '/home/qt/work/testresults/tst_qtimer-1665735205929.junit.xml,junitxml', '-o', '-,txt', 'singleShotToFunctors']
      agent:2022/10/14 08:13:26 build.go:394: 19: ********* Start testing of tst_QTimer *********
      agent:2022/10/14 08:13:26 build.go:394: 19: Config: Using QtTest library 6.5.0, Qt 6.5.0 (arm64-little_endian-lp64 shared (dynamic) release build; by GCC 10.2.0), b2qt 3.2
      agent:2022/10/14 08:13:26 build.go:394: 19: PASS   : tst_QTimer::initTestCase()
      agent:2022/10/14 08:13:26 build.go:394: 19: PASS   : tst_QTimer::singleShotToFunctors()
      agent:2022/10/14 08:13:26 build.go:394: 19: PASS   : tst_QTimer::cleanupTestCase()
      agent:2022/10/14 08:13:26 build.go:394: 19: Totals: 3 passed, 0 failed, 0 skipped, 0 blacklisted, 937ms
      agent:2022/10/14 08:13:26 build.go:394: 19: ********* Finished testing of tst_QTimer *********
      agent:2022/10/14 08:13:26 build.go:394: 19: qt-testrunner.py     INFO: Test process exited with code: 0
      agent:2022/10/14 08:13:26 build.go:394: 19: qt-testrunner.py     INFO: Test has PASSed as FLAKY after re-runs:1, passes:1, failures:0
      agent:2022/10/14 08:13:27 build.go:394:  19/591 Test  #19: tst_qtimer ..............................................   Passed  323.47 sec
      

      Attachments

        For Gerrit Dashboard: QTQAINFRA-5226
        # Subject Branch Project Status CR V

        Activity

          People

            jimis Dimitrios Apostolou
            manordheim MÃ¥rten Nordheim
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Gerrit Reviews

                There is 1 open Gerrit change