Uploaded image for project: 'Qt'
  1. Qt
  2. QTBUG-74611

Flaky tests/auto/webchannel crashes on multiple platforms in CI

    XMLWordPrintable

Details

    • Bug
    • Resolution: Done
    • P1: Critical
    • 5.14.1
    • 5.12, 5.13
    • WebChannel
    • None

    Description

      Crashing TestWebChannel is causing almost all failures of qtwebchannel component in CI:

      https://testresults.qt.io/grafana/d/000000007/coin-flaky-tests?orgId=1&var-project=qt%2Fqtwebchannel&var-branch=All&var-inter=1d

       

      Windows10 crash:

      http://testresults.qt.io/coin/integration/qt/qtwebchannel/tasks/1553172265

       

      agent:2019/03/21 12:53:24 build.go:449: Executing instruction 31 of 36 - RunQtUnitTest
agent:2019/03/21 12:53:24 runner.go:72: Generating test results with TESTARGS=-o C:\Users\qt\work\testresults\tests\auto\webchannel251589919,xml -o -,txt
agent:2019/03/21 12:53:24 build.go:314: [c:\users\qt\MSVC2017.bat jom check] 900 900
agent:2019/03/21 12:53:24 build.go:192: 
agent:2019/03/21 12:53:24 build.go:192: C:\Users\qt\work\qt\qtwebchannel\tests\auto\webchannel>call "C:\Program Files (x86)\Microsoft Visual Studio\2017\Professional\VC\Auxiliary\Build\vcvarsall.bat" amd64_x86  
agent:2019/03/21 12:53:25 build.go:192: **********************************************************************
agent:2019/03/21 12:53:25 build.go:192: ** Visual Studio 2017 Developer Command Prompt v15.4.0
agent:2019/03/21 12:53:25 build.go:192: ** Copyright (c) 2017 Microsoft Corporation
agent:2019/03/21 12:53:25 build.go:192: **********************************************************************
agent:2019/03/21 12:53:26 build.go:192: [vcvarsall.bat] Environment initialized for: 'x64_x86'
agent:2019/03/21 12:53:26 build.go:192: 
agent:2019/03/21 12:53:26 build.go:192: jom 1.1.2 - empower your cores
agent:2019/03/21 12:53:26 build.go:192: 
agent:2019/03/21 12:53:26 build.go:192: 	C:\Utils\Jom\jom.exe -f Makefile.Release check
agent:2019/03/21 12:53:27 build.go:192: 	call C:\Users\qt\work\qt\qtwebchannel\tests\auto\webchannel\target_wrapper.bat  release\tst_webchannel.exe -o C:\Users\qt\work\testresults\tests\auto\webchannel251589919,xml -o -,txt
agent:2019/03/21 12:53:27 build.go:192: ********* Start testing of TestWebChannel *********
agent:2019/03/21 12:53:27 build.go:192: Config: Using QtTest library 5.14.0, Qt 5.14.0 (i386-little_endian-ilp32 shared (dynamic) release build; by MSVC 2017)
agent:2019/03/21 12:53:27 build.go:192: PASS   : TestWebChannel::initTestCase()
agent:2019/03/21 12:53:27 build.go:192: PASS   : TestWebChannel::testRegisterObjects()
agent:2019/03/21 12:53:27 build.go:192: PASS   : TestWebChannel::testDeregisterObjects()
agent:2019/03/21 12:53:27 build.go:192: PASS   : TestWebChannel::testDeregisterObjectAtStart()
agent:2019/03/21 12:53:27 build.go:192: PASS   : TestWebChannel::testInfoForObject()
agent:2019/03/21 12:53:27 build.go:192: PASS   : TestWebChannel::testInvokeMethodConversion()
agent:2019/03/21 12:53:27 build.go:192: PASS   : TestWebChannel::testSetPropertyConversion()
agent:2019/03/21 12:53:27 build.go:192: PASS   : TestWebChannel::testDisconnect()
agent:2019/03/21 12:53:27 build.go:192: PASS   : TestWebChannel::testWrapRegisteredObject()
agent:2019/03/21 12:53:27 build.go:192: PASS   : TestWebChannel::testUnwrapObject()
agent:2019/03/21 12:53:27 build.go:192: PASS   : TestWebChannel::testRemoveUnusedTransports()
agent:2019/03/21 12:53:27 build.go:192: PASS   : TestWebChannel::testPassWrappedObjectBack()
agent:2019/03/21 12:53:27 build.go:192: PASS   : TestWebChannel::testWrapValues()
agent:2019/03/21 12:53:27 build.go:192: PASS   : TestWebChannel::testWrapObjectWithMultipleTransports()
agent:2019/03/21 12:53:27 build.go:192: PASS   : TestWebChannel::testJsonToVariant()
agent:2019/03/21 12:53:27 build.go:192: PASS   : TestWebChannel::testInfiniteRecursion()
agent:2019/03/21 12:53:28 build.go:192: A crash occurred in C:\Users\qt\work\qt\qtwebchannel\tests\auto\webchannel\release\tst_webchannel.exe.
agent:2019/03/21 12:53:28 build.go:192: Function time: 76ms Total time: 96ms
agent:2019/03/21 12:53:28 build.go:192: 
agent:2019/03/21 12:53:28 build.go:192: Exception address: 0x696953BD
agent:2019/03/21 12:53:28 build.go:192: Exception code   : 0xc0000005
agent:2019/03/21 12:53:28 build.go:192: Nearby symbol    : QMetaObjectPublisher::deleteWrappedObject
agent:2019/03/21 12:53:28 build.go:192: 
agent:2019/03/21 12:53:28 build.go:192: Stack:
agent:2019/03/21 12:53:28 build.go:192: #  1: QTest::toString() - 0x69659B40
agent:2019/03/21 12:53:28 build.go:192: 
agent:2019/03/21 12:53:28 build.go:192: jom: C:\Users\qt\work\qt\qtwebchannel\tests\auto\webchannel\Makefile.Release [check] Error 2
agent:2019/03/21 12:53:28 build.go:192: jom: C:\Users\qt\work\qt\qtwebchannel\tests\auto\webchannel\Makefile [release-check] Error 2
agent:2019/03/21 12:53:28 build.go:236: Process finished with error: exit status 2

       

      MacOS crash:

      https://testresults.qt.io/coin/integration/qt/qtwebchannel/tasks/1552645324

      agent:2019/03/15 11:01:16 runner.go:153: Re-running tests/auto/webchannel with TESTARGS=testAsyncObject
agent:2019/03/15 11:01:16 build.go:319: [make check -j1] 900 900
agent:2019/03/15 11:01:16 build.go:197: /Users/qt/work/qt/qtwebchannel/tests/auto/webchannel/target_wrapper.sh  ./tst_webchannel testAsyncObject -maxwarnings 0
agent:2019/03/15 11:01:16 build.go:197: ********* Start testing of TestWebChannel *********
agent:2019/03/15 11:01:16 build.go:197: Config: Using QtTest library 5.12.2, Qt 5.12.2 (x86_64-little_endian-lp64 shared (dynamic) release build; by Clang 9.0.0 (clang-900.0.39.2) (Apple))
agent:2019/03/15 11:01:16 build.go:197: PASS   : TestWebChannel::initTestCase()
agent:2019/03/15 11:01:16 build.go:197: 
agent:2019/03/15 11:01:16 build.go:197: === Received signal at function time: 54ms, total time: 56ms, dumping stack ===
agent:2019/03/15 11:01:17 build.go:197: (lldb) process attach --pid 950
agent:2019/03/15 11:01:32 build.go:197: Process 950 stopped
agent:2019/03/15 11:01:32 build.go:197: * thread #1, queue = 'com.apple.main-thread', stop reason = signal SIGSTOP
agent:2019/03/15 11:01:32 build.go:197:     frame #0: 0x00007fffb339a406 libsystem_kernel.dylib`__wait4_nocancel + 10
agent:2019/03/15 11:01:32 build.go:197: libsystem_kernel.dylib`__wait4_nocancel:
agent:2019/03/15 11:01:32 build.go:197: ->  0x7fffb339a406 <+10>: jae    0x7fffb339a410            ; <+20>
agent:2019/03/15 11:01:32 build.go:197:     0x7fffb339a408 <+12>: movq   %rax, %rdi
agent:2019/03/15 11:01:32 build.go:197:     0x7fffb339a40b <+15>: jmp    0x7fffb3392caf            ; cerror_nocancel
agent:2019/03/15 11:01:32 build.go:197:     0x7fffb339a410 <+20>: retq   
agent:2019/03/15 11:01:32 build.go:197: Target 0: (tst_webchannel) stopped.
agent:2019/03/15 11:01:32 build.go:197: 
agent:2019/03/15 11:01:32 build.go:197: Executable module set to "/Users/qt/work/qt/qtwebchannel/tests/auto/webchannel/tst_webchannel".
agent:2019/03/15 11:01:32 build.go:197: Architecture set to: x86_64-apple-macosx.
agent:2019/03/15 11:01:32 build.go:197: (lldb) bt all
agent:2019/03/15 11:01:32 build.go:197: * thread #1, queue = 'com.apple.main-thread', stop reason = signal SIGSTOP
agent:2019/03/15 11:01:32 build.go:197:   * frame #0: 0x00007fffb339a406 libsystem_kernel.dylib`__wait4_nocancel + 10
agent:2019/03/15 11:01:32 build.go:197:     frame #1: 0x00007fffb3320e3d libsystem_c.dylib`system + 463
agent:2019/03/15 11:01:32 build.go:197:     frame #2: 0x000000010daf3ba5 libQt5Test.5.dylib`___lldb_unnamed_symbol20$$libQt5Test.5.dylib + 325
agent:2019/03/15 11:01:32 build.go:197:     frame #3: 0x000000010daf39cf libQt5Test.5.dylib`___lldb_unnamed_symbol19$$libQt5Test.5.dylib + 207
agent:2019/03/15 11:01:32 build.go:197:     frame #4: 0x00007fffb347ab3a libsystem_platform.dylib`_sigtramp + 26
agent:2019/03/15 11:01:32 build.go:197:     frame #5: 0x000000010d514744 libQt5WebChannel.5.dylib`___lldb_unnamed_symbol48$$libQt5WebChannel.5.dylib + 36
agent:2019/03/15 11:01:32 build.go:197:     frame #6: 0x000000010d5145ad libQt5WebChannel.5.dylib`___lldb_unnamed_symbol42$$libQt5WebChannel.5.dylib + 61
agent:2019/03/15 11:01:32 build.go:197:     frame #7: 0x000000010dd4a761 libQt5Core.5.dylib`TestNamespace::QObject::event(TestNamespace::QEvent*) + 769
agent:2019/03/15 11:01:32 build.go:197:     frame #8: 0x000000010dd21008 libQt5Core.5.dylib`TestNamespace::QCoreApplicationPrivate::notify_helper(TestNamespace::QObject*, TestNamespace::QEvent*) + 88
agent:2019/03/15 11:01:32 build.go:197:     frame #9: 0x000000010dd20c58 libQt5Core.5.dylib`TestNamespace::QCoreApplication::notifyInternal2(TestNamespace::QObject*, TestNamespace::QEvent*) + 168
agent:2019/03/15 11:01:32 build.go:197:     frame #10: 0x000000010dd21e38 libQt5Core.5.dylib`TestNamespace::QCoreApplicationPrivate::sendPostedEvents(TestNamespace::QObject*, int, TestNamespace::QThreadData*) + 840
agent:2019/03/15 11:01:32 build.go:197:     frame #11: 0x000000010dd792b8 libQt5Core.5.dylib`TestNamespace::QEventDispatcherUNIX::processEvents(TestNamespace::QFlags<TestNamespace::QEventLoop::ProcessEventsFlag>) + 72
agent:2019/03/15 11:01:32 build.go:197:     frame #12: 0x000000010dd211bd libQt5Core.5.dylib`TestNamespace::QCoreApplication::processEvents(TestNamespace::QFlags<TestNamespace::QEventLoop::ProcessEventsFlag>, int) + 93
agent:2019/03/15 11:01:32 build.go:197:     frame #13: 0x000000010dd74ae9 libQt5Core.5.dylib`TestNamespace::QTest::qWait(int) + 57
agent:2019/03/15 11:01:32 build.go:197:     frame #14: 0x000000010d4dd966 tst_webchannel`TestNamespace::TestWebChannel::testAsyncObject() + 3846
agent:2019/03/15 11:01:32 build.go:197:     frame #15: 0x000000010dd2adfc libQt5Core.5.dylib`TestNamespace::QMetaMethod::invoke(TestNamespace::QObject*, TestNamespace::Qt::ConnectionType, TestNamespace::QGenericReturnArgument, TestNamespace::QGenericArgument, TestNamespace::QGenericArgument, TestNamespace::QGenericArgument, TestNamespace::QGenericArgument, TestNamespace::QGenericArgument, TestNamespace::QGenericArgument, TestNamespace::QGenericArgument, TestNamespace::QGenericArgument, TestNamespace::QGenericArgument, TestNamespace::QGenericArgument) const + 1308
agent:2019/03/15 11:01:32 build.go:197:     frame #16: 0x000000010daf1751 libQt5Test.5.dylib`___lldb_unnamed_symbol11$$libQt5Test.5.dylib + 1169
agent:2019/03/15 11:01:32 build.go:197:     frame #17: 0x000000010daf2433 libQt5Test.5.dylib`___lldb_unnamed_symbol13$$libQt5Test.5.dylib + 787
agent:2019/03/15 11:01:32 build.go:197:     frame #18: 0x000000010daf3557 libQt5Test.5.dylib`___lldb_unnamed_symbol17$$libQt5Test.5.dylib + 1031
agent:2019/03/15 11:01:32 build.go:197:     frame #19: 0x000000010daf4157 libQt5Test.5.dylib`TestNamespace::QTest::qRun() + 247
agent:2019/03/15 11:01:32 build.go:197:     frame #20: 0x000000010daf3e10 libQt5Test.5.dylib`TestNamespace::QTest::qExec(TestNamespace::QObject*, int, char**) + 16
agent:2019/03/15 11:01:32 build.go:197:     frame #21: 0x000000010d4e1389 tst_webchannel`main + 105
agent:2019/03/15 11:01:32 build.go:197:     frame #22: 0x00007fffb326b235 libdyld.dylib`start + 1
agent:2019/03/15 11:01:32 build.go:197:     frame #23: 0x00007fffb326b235 libdyld.dylib`start + 1
agent:2019/03/15 11:01:32 build.go:197:   thread #2
agent:2019/03/15 11:01:32 build.go:197:     frame #0: 0x00007fffb339a44e libsystem_kernel.dylib`__workq_kernreturn + 10
agent:2019/03/15 11:01:32 build.go:197:     frame #1: 0x00007fffb3484621 libsystem_pthread.dylib`_pthread_wqthread + 1426
agent:2019/03/15 11:01:32 build.go:197:     frame #2: 0x00007fffb348407d libsystem_pthread.dylib`start_wqthread + 13
agent:2019/03/15 11:01:32 build.go:197:   thread #3
agent:2019/03/15 11:01:32 build.go:197:     frame #0: 0x00007fffb339a44e libsystem_kernel.dylib`__workq_kernreturn + 10
agent:2019/03/15 11:01:32 build.go:197:     frame #1: 0x00007fffb348448e libsystem_pthread.dylib`_pthread_wqthread + 1023
agent:2019/03/15 11:01:32 build.go:197:     frame #2: 0x00007fffb348407d libsystem_pthread.dylib`start_wqthread + 13
agent:2019/03/15 11:01:32 build.go:197:   thread #4
agent:2019/03/15 11:01:32 build.go:197:     frame #0: 0x00007fffb339a44e libsystem_kernel.dylib`__workq_kernreturn + 10
agent:2019/03/15 11:01:32 build.go:197:     frame #1: 0x00007fffb3484621 libsystem_pthread.dylib`_pthread_wqthread + 1426
agent:2019/03/15 11:01:32 build.go:197:     frame #2: 0x00007fffb348407d libsystem_pthread.dylib`start_wqthread + 13
agent:2019/03/15 11:01:32 build.go:197:   thread #5, name = 'QThread'
agent:2019/03/15 11:01:32 build.go:197:     frame #0: 0x00007fffb3399bf2 libsystem_kernel.dylib`__psynch_cvwait + 10
agent:2019/03/15 11:01:32 build.go:197:     frame #1: 0x00007fffb34857fa libsystem_pthread.dylib`_pthread_cond_wait + 712
agent:2019/03/15 11:01:32 build.go:197:     frame #2: 0x000000010db6930d libQt5Core.5.dylib`___lldb_unnamed_symbol286$$libQt5Core.5.dylib + 301
agent:2019/03/15 11:01:32 build.go:197:     frame #3: 0x000000010db68ffe libQt5Core.5.dylib`___lldb_unnamed_symbol285$$libQt5Core.5.dylib + 94
agent:2019/03/15 11:01:32 build.go:197:     frame #4: 0x000000010db68f77 libQt5Core.5.dylib`TestNamespace::QWaitCondition::wait(TestNamespace::QMutex*, TestNamespace::QDeadlineTimer) + 167
agent:2019/03/15 11:01:32 build.go:197:     frame #5: 0x000000010db68e9e libQt5Core.5.dylib`TestNamespace::QWaitCondition::wait(TestNamespace::QMutex*, unsigned long) + 62
agent:2019/03/15 11:01:32 build.go:197:     frame #6: 0x000000010daf91d6 libQt5Test.5.dylib`___lldb_unnamed_symbol34$$libQt5Test.5.dylib + 70
agent:2019/03/15 11:01:32 build.go:197:     frame #7: 0x000000010db607dc libQt5Core.5.dylib`___lldb_unnamed_symbol227$$libQt5Core.5.dylib + 332
agent:2019/03/15 11:01:32 build.go:197:     frame #8: 0x00007fffb348493b libsystem_pthread.dylib`_pthread_body + 180
agent:2019/03/15 11:01:32 build.go:197:     frame #9: 0x00007fffb3484887 libsystem_pthread.dylib`_pthread_start + 286
agent:2019/03/15 11:01:32 build.go:197:     frame #10: 0x00007fffb348408d libsystem_pthread.dylib`thread_start + 13
agent:2019/03/15 11:01:32 build.go:197:   thread #6, name = 'QThread'
agent:2019/03/15 11:01:32 build.go:197:     frame #0: 0x00007fffb339b19e libsystem_kernel.dylib`poll + 10
agent:2019/03/15 11:01:32 build.go:197:     frame #1: 0x000000010dd77f80 libQt5Core.5.dylib`TestNamespace::qt_safe_poll(pollfd*, unsigned int, timespec const*) + 528
agent:2019/03/15 11:01:32 build.go:197:     frame #2: 0x000000010dd795c9 libQt5Core.5.dylib`TestNamespace::QEventDispatcherUNIX::processEvents(TestNamespace::QFlags<TestNamespace::QEventLoop::ProcessEventsFlag>) + 857
agent:2019/03/15 11:01:32 build.go:197:     frame #3: 0x000000010dd1c83e libQt5Core.5.dylib`TestNamespace::QEventLoop::exec(TestNamespace::QFlags<TestNamespace::QEventLoop::ProcessEventsFlag>) + 398
agent:2019/03/15 11:01:32 build.go:197:     frame #4: 0x000000010db5f862 libQt5Core.5.dylib`TestNamespace::QThread::exec() + 114
agent:2019/03/15 11:01:32 build.go:197:     frame #5: 0x000000010db607dc libQt5Core.5.dylib`___lldb_unnamed_symbol227$$libQt5Core.5.dylib + 332
agent:2019/03/15 11:01:32 build.go:197:     frame #6: 0x00007fffb348493b libsystem_pthread.dylib`_pthread_body + 180
agent:2019/03/15 11:01:32 build.go:197:     frame #7: 0x00007fffb3484887 libsystem_pthread.dylib`_pthread_start + 286
agent:2019/03/15 11:01:32 build.go:197:     frame #8: 0x00007fffb348408d libsystem_pthread.dylib`thread_start + 13
agent:2019/03/15 11:01:32 build.go:197: (lldb) quit
agent:2019/03/15 11:01:32 build.go:197: === End of stack trace ===
agent:2019/03/15 11:01:32 build.go:197: QFATAL : TestWebChannel::testAsyncObject() [     0.000 F] default unknown - Received signal 11
agent:2019/03/15 11:01:32 build.go:197:          Function time: 54ms Total time: 56ms
agent:2019/03/15 11:01:32 build.go:197: FAIL!  : TestWebChannel::testAsyncObject() Received a fatal error.
agent:2019/03/15 11:01:32 build.go:197:    Loc: [Unknown file(0)]
agent:2019/03/15 11:01:32 build.go:197: Totals: 1 passed, 1 failed, 0 skipped, 0 blacklisted, 15912ms
agent:2019/03/15 11:01:32 build.go:197: ********* Finished testing of TestWebChannel *********
agent:2019/03/15 11:01:32 build.go:197: make: *** [check] Abort trap: 6
agent:2019/03/15 11:01:32 build.go:241: Process finished with error: exit status 2
agent:2019/03/15 11:01:32 build.go:200: Error reading from stdout/err: exit status 2

       

      Flaky tests are slowing down Qt development so it is better to either fix or blacklist them.

       

      Attachments

        Issue Links

          is duplicated by

          Bug - A problem which impairs or prevents the functions of the product. QTBUG-77289 tst_webchannel.exe crash after TestWebChannel::testInfiniteRecursion()

          • P0: Blocker - MUST be fixed ASAP. This includes issues that prevent/block further Qt development; Legal issues & Serious data loss issues.
          • Closed

          Bug - A problem which impairs or prevents the functions of the product. QTBUG-76973 Auto test tests/auto/webchannel crashes a lot on Windows10

          • P1: Critical - Urgent and Important, will STOP the release if matched with set FixVersion field. This includes regressions from the last version that are not edge cases; Data loss; Build issues; All but the most unlikely crashes/lockups/hanging; Serious usability issues and embarrassing bugs & Bugs critical to a deliverable. This is the highest possible priority for requirements.
          • Closed

          Bug - A problem which impairs or prevents the functions of the product. QTBUG-80729 TestWebChannel::testAsyncObject() is flaky and crashes on many platforms

          • P1: Critical - Urgent and Important, will STOP the release if matched with set FixVersion field. This includes regressions from the last version that are not edge cases; Data loss; Build issues; All but the most unlikely crashes/lockups/hanging; Serious usability issues and embarrassing bugs & Bugs critical to a deliverable. This is the highest possible priority for requirements.
          • Closed
          is replaced by

          Bug - A problem which impairs or prevents the functions of the product. QTBUG-51366 QWebChannel cannot emit signals from SIGNALS in different QThread

          • P1: Critical - Urgent and Important, will STOP the release if matched with set FixVersion field. This includes regressions from the last version that are not edge cases; Data loss; Build issues; All but the most unlikely crashes/lockups/hanging; Serious usability issues and embarrassing bugs & Bugs critical to a deliverable. This is the highest possible priority for requirements.
          • Closed
          No reviews matched the request. Check your Options in the drop-down menu of this sections header.

          Activity

            People

              kiburtse Kirill Burtsev
              aakeskimo Aapo Keskimölö
              Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Gerrit Reviews

                  There are no open Gerrit changes