Uploaded image for project: 'Qt Quality Assurance Infrastructure'
  1. Qt Quality Assurance Infrastructure
  2. QTQAINFRA-6580

Missing information about flaky test in PSQL

    XMLWordPrintable

Details

    • Bug
    • Resolution: Unresolved
    • P2: Important
    • None
    • master
    • Test scripts
    • None

    Description

      Extremely rarely in less than 0.02% of checked flaky tests - there was missing entry in posgreSQL flaky test database.

      Below list of those rare cases:

      1) 
      tst_integration
      registryAddedTest

      https://testresults.qt.io/coin/log/qt/qtremoteobjects/81732ec61d2fb48c7e3cfe5534c16ae1075b2257/WindowsWindows_11_23H2x86_64WindowsWindows_11_23H2x86_64MSVC2022qtci-windows-11_23H2-x86_64-53-c7d4afDebugAndRelease_GenerateSBOM_Packaging_Sccache_UseConfigure_VerifySBOM/7d1f4274821db874905535e737d66828a145e738/test_1725309243

       
                  SELECT
                      *
                  FROM
                      workitem_ids
                  
                  WHERE
                      workitem_id = 'qt/qtremoteobjects/81732ec61d2fb48c7e3cfe5534c16ae1075b2257/WindowsWindows_11_23H2x86_64WindowsWindows_11_23H2x86_64MSVC2022qtci-windows-11_23H2-x86_64-53-c7d4afDebugAndRelease_GenerateSBOM_Packaging_Sccache_UseConfigure_VerifySBOM/7d1f4274821db874905535e737d66828a145e738/test_1725309243'

      result: 15607139

                  SELECT
                      *
                  FROM
                      flaky_tests
                  
                  WHERE
                      workitem_n = '15607139'

      result: empty

      agent:2024/09/02 14:19:41 build.go:404: 24: PASS   : tst_Integration::registryAddedTest(tcp)

      agent:2024/09/02 14:24:41 build.go:404: 24:

      agent:2024/09/02 14:24:41 build.go:404: 24:          registryAddedTest function time: 300232ms, total time: 307637ms

      agent:2024/09/02 14:24:41 build.go:404: 24: QFATAL : tst_Integration::registryAddedTest(local) Test function timed out

      agent:2024/09/02 14:24:41 build.go:404: 24: FAIL!  : tst_Integration::registryAddedTest(local) Received a fatal error.

      agent:2024/09/02 14:24:41 build.go:404: 24: Totals: 25 passed, 1 failed, 1 skipped, 0 blacklisted, 307642ms

      agent:2024/09/02 14:24:41 build.go:404: 24: ********* Finished testing of tst_Integration *********

      agent:2024/09/02 14:24:41 build.go:404: 24: qt-testrunner.py     INFO: Test process exited with code: 3221226505

      agent:2024/09/02 14:24:41 build.go:404: 24: qt-testrunner.py     INFO: Parsed XML file \Users\qt\work\testresults\tst_integration-1725286774026.xml in 0.001 seconds

      agent:2024/09/02 14:24:41 build.go:404: 24: qt-testrunner.py     INFO: Found 26 passes and 1 failures

      agent:2024/09/02 14:24:41 build.go:404: 24: qt-testrunner.py     INFO: Some tests failed, will re-run at most 5 times.

      agent:2024/09/02 14:24:41 build.go:404: 24:

      agent:2024/09/02 14:24:41 build.go:404: 24: qt-testrunner.py     INFO: Re-running testcase: registryAddedTest:local

      agent:2024/09/02 14:24:41 build.go:404: 24: qt-testrunner.py    DEBUG: Running test command line: ['C:/Users/qt/work/qt/qtremoteobjects_standalone_tests/tests/auto/integration/tst_integration.exe', '-o', '\\Users\\qt\\work\\testresults\\tst_integration-1725287081986.xml,xml', '-o', '\\Users\\qt\\work\\testresults\\tst_integration-1725287081986.junit.xml,junitxml', '-o', '\\Users\\qt\\work\\testresults\\tst_integration-1725287081986.txt,txt', '-o', '-,txt', 'registryAddedTest:local']

      agent:2024/09/02 14:24:42 build.go:404: 24: ********* Start testing of tst_Integration *********

      agent:2024/09/02 14:24:42 build.go:404: 24: Config: Using QtTest library 6.9.0, Qt 6.9.0 (x86_64-little_endian-llp64 shared (dynamic) release build; by MSVC 2022), windows 11

      agent:2024/09/02 14:24:42 build.go:404: 24: PASS   : tst_Integration::initTestCase()

      agent:2024/09/02 14:24:42 build.go:404: 24: PASS   : tst_Integration::registryAddedTest(local)

      agent:2024/09/02 14:24:42 build.go:404: 24: PASS   : tst_Integration::cleanupTestCase()

      agent:2024/09/02 14:24:42 build.go:404: 24: Totals: 3 passed, 0 failed, 0 skipped, 0 blacklisted, 241ms

       

      2) 

      tst_touchinput

      touchTapAndHold

      https://testresults.qt.io/coin/log/qt/qtremoteobjects/81732ec61d2fb48c7e3cfe5534c16ae1075b2257/WindowsWindows_11_23H2x86_64WindowsWindows_11_23H2x86_64MSVC2022qtci-windows-11_23H2-x86_64-53-c7d4afDebugAndRelease_GenerateSBOM_Packaging_Sccache_UseConfigure_VerifySBOM/7d1f4274821db874905535e737d66828a145e738/test_1725309243 

       
       

      agent:2024/09/02 17:44:01 build.go:404: 40: FAIL! : TouchInputTest::touchTapAndHold() Compared values are not the same
      agent:2024/09/02 17:44:01 build.go:404: 40: Actual (view.selectedText()) : ""
      agent:2024/09/02 17:44:01 build.go:404: 40: Expected ((QString(QtPrivate::qMakeStringPrivate(u"" "Company")))): "Company"
      agent:2024/09/02 17:44:01 build.go:404: 40: C:\Users\qt\work\qt\qtwebengine\tests\auto\widgets\touchinput\tst_touchinput.cpp(216) : failure location
      agent:2024/09/02 17:44:02 build.go:404: 40: XFAIL : TouchInputTest::touchTapAndHoldCancelled() Incorrect Chromium selection behavior when cancelling tap-and-hold on text

      agent:2024/09/02 17:44:02 build.go:404: 40: C:\Users\qt\work\qt\qtwebengine\tests\auto\widgets\touchinput\tst_touchinput.cpp(250) : failure location
      agent:2024/09/02 17:44:04 build.go:404: 40: XFAIL : TouchInputTest::touchTapAndHoldCancelled() Incorrect Chromium selection behavior when cancelling tap-and-hold on input field
      agent:2024/09/02 17:44:04 build.go:404: 40: C:\Users\qt\work\qt\qtwebengine\tests\auto\widgets\touchinput\tst_touchinput.cpp(255) : failure location
      agent:2024/09/02 17:44:04 build.go:404: 40: XFAIL : TouchInputTest::touchTapAndHoldCancelled() Incorrect Chromium focus behavior when cancelling tap-and-hold on input field
      agent:2024/09/02 17:44:04 build.go:404: 40: C:\Users\qt\work\qt\qtwebengine\tests\auto\widgets\touchinput\tst_touchinput.cpp(257) : failure location
      agent:2024/09/02 17:44:05 build.go:404: 40: PASS : TouchInputTest::touchTapAndHoldCancelled()
      agent:2024/09/02 17:44:08 build.go:404: 40: PASS : TouchInputTest::scrolling()
      agent:2024/09/02 17:44:10 build.go:404: 40: PASS : TouchInputTest::pinchZoom(sequential)
      agent:2024/09/02 17:44:12 build.go:404: 40: PASS : TouchInputTest::pinchZoom(simultaneous)
      agent:2024/09/02 17:44:15 build.go:404: 40: PASS : TouchInputTest::complexSequence()
      agent:2024/09/02 17:44:17 build.go:404: 40: XFAIL : TouchInputTest::buttonClickHandler() Shouldn't trigger twice due to synthesized mouse events for touch
      agent:2024/09/02 17:44:17 build.go:404: 40: C:\Users\qt\work\qt\qtwebengine\tests\auto\widgets\touchinput\tst_touchinput.cpp(351) : failure location
      agent:2024/09/02 17:44:17 build.go:404: 40: PASS : TouchInputTest::buttonClickHandler()
      agent:2024/09/02 17:44:19 build.go:404: 40: XFAIL : TouchInputTest::htmlSelectPopup() Shouldn't trigger twice due to synthesized mouse events for touch
      agent:2024/09/02 17:44:19 build.go:404: 40: C:\Users\qt\work\qt\qtwebengine\tests\auto\widgets\touchinput\tst_touchinput.cpp(368) : failure location
      agent:2024/09/02 17:44:19 build.go:404: 40: PASS : TouchInputTest::htmlSelectPopup()
      agent:2024/09/02 17:44:19 build.go:404: 40: PASS : TouchInputTest::cleanupTestCase()
      agent:2024/09/02 17:44:19 build.go:404: 40: Totals: 10 passed, 1 failed, 0 skipped, 0 blacklisted, 39545ms
      agent:2024/09/02 17:44:19 build.go:404: 40: ********* Finished testing of TouchInputTest *********
      agent:2024/09/02 17:44:19 build.go:404: 40: qt-testrunner.py INFO: Test process exited with code: 1
      agent:2024/09/02 17:44:19 build.go:404: 40: qt-testrunner.py INFO: Parsed XML file \Users\qt\work\testresults\tst_touchinput-1725299019705.xml in 0.001 seconds
      agent:2024/09/02 17:44:19 build.go:404: 40: qt-testrunner.py INFO: Found 15 passes and 1 failures
      agent:2024/09/02 17:44:19 build.go:404: 40: qt-testrunner.py INFO: Some tests failed, will re-run at most 5 times.
      agent:2024/09/02 17:44:19 build.go:404: 40:
      agent:2024/09/02 17:44:19 build.go:404: 40: qt-testrunner.py INFO: Re-running testcase: touchTapAndHold
      agent:2024/09/02 17:44:19 build.go:404: 40: qt-testrunner.py DEBUG: Running test command line: ['C:/Users/qt/work/qt/qtwebengine_standalone_tests/tests/auto/widgets/touchinput/tst_touchinput.exe', '-o', '\\Users\\qt\\work\\testresults\\tst_touchinput-1725299059980.xml,xml', '-o', '\\Users\\qt\\work\\testresults\\tst_touchinput-1725299059980.junit.xml,junitxml', '-o', '\\Users\\qt\\work\\testresults\\tst_touchinput-1725299059980.txt,txt', '-o', '-,txt', 'touchTapAndHold']
      agent:2024/09/02 17:44:20 build.go:404: 40: ********* Start testing of TouchInputTest *********
      agent:2024/09/02 17:44:20 build.go:404: 40: Config: Using QtTest library 6.9.0, Qt 6.9.0 (x86_64-little_endian-llp64 shared (dynamic) release build; by MSVC 2022), windows 11
      agent:2024/09/02 17:44:20 build.go:404: 40: [2520:4908:0902/174420.561:ERROR:gpu_channel_manager.cc(959)] Failed to create GLES3 context, fallback to GLES2.
      agent:2024/09/02 17:44:20 build.go:404: 40: [2520:4908:0902/174420.561:ERROR:gpu_channel_manager.cc(970)] ContextResult::kFatalFailure: Failed to create shared context for virtualization.
      agent:2024/09/02 17:44:20 build.go:404: 40: PASS : TouchInputTest::initTestCase()
      agent:2024/09/02 17:44:24 build.go:404: 40: PASS : TouchInputTest::touchTapAndHold()  
       
      3)
      tst_qwebengineview
      deferredDelete
       
      https://testresults.qt.io/coin/log/qt/qtwebengine/030827e5ca59c10764ac4b8a87fc90324c308f11/WindowsWindows_11_23H2x86_64WindowsWindows_11_23H2x86_64MSVC2022qtci-windows-11_23H2-x86_64-53-c7d4afDebugAndRelease_GenerateSBOM_Packaging_Sccache_UseConfigure_VerifySBOM/3e04c7304cbb365a7ce907af0c71855f8417020e/test_1725308976
       
      agent:2024/09/02 17:40:54 build.go:404: 28: FAIL! : tst_QWebEngineView::deferredDelete() Compared values are not the same
      agent:2024/09/02 17:40:54 build.go:404: 28: Actual (QApplication::allWidgets().size()): 3
      agent:2024/09/02 17:40:54 build.go:404: 28: Expected (2) : 2
      agent:2024/09/02 17:40:54 build.go:404: 28: C:\Users\qt\work\qt\qtwebengine\tests\auto\widgets\qwebengineview\tst_qwebengineview.cpp(4021) : failure location
      agent:2024/09/02 17:40:54 build.go:404: 28: PASS : tst_QWebEngineView::setCursorOnEmbeddedView()
      agent:2024/09/02 17:40:54 build.go:404: 28: PASS : tst_QWebEngineView::cleanupTestCase()
      agent:2024/09/02 17:40:54 build.go:404: 28: Totals: 164 passed, 1 failed, 8 skipped, 3 blacklisted, 60036ms
      agent:2024/09/02 17:40:54 build.go:404: 28: ********* Finished testing of tst_QWebEngineView *********
      agent:2024/09/02 17:40:55 build.go:404: 28: qt-testrunner.py INFO: Test process exited with code: 1
      agent:2024/09/02 17:40:55 build.go:404: 28: qt-testrunner.py INFO: Parsed XML file \Users\qt\work\testresults\tst_qwebengineview-1725298794636.xml in 0.003 seconds
      agent:2024/09/02 17:40:55 build.go:404: 28: qt-testrunner.py INFO: Found 178 passes and 1 failures
      agent:2024/09/02 17:40:55 build.go:404: 28: qt-testrunner.py INFO: Some tests failed, will re-run at most 5 times.
      agent:2024/09/02 17:40:55 build.go:404: 28:
      agent:2024/09/02 17:40:55 build.go:404: 28: qt-testrunner.py INFO: Re-running testcase: deferredDelete
      agent:2024/09/02 17:40:55 build.go:404: 28: qt-testrunner.py DEBUG: Running test command line: ['C:/Users/qt/work/qt/qtwebengine_standalone_tests/tests/auto/widgets/qwebengineview/tst_qwebengineview.exe', '-o', '\\Users\\qt\\work\\testresults\\tst_qwebengineview-1725298855134.xml,xml', '-o', '\\Users\\qt\\work\\testresults\\tst_qwebengineview-1725298855134.junit.xml,junitxml', '-o', '\\Users\\qt\\work\\testresults\\tst_qwebengineview-1725298855134.txt,txt', '-o', '-,txt', 'deferredDelete']
      agent:2024/09/02 17:40:55 build.go:404: 28: ********* Start testing of tst_QWebEngineView *********
      agent:2024/09/02 17:40:55 build.go:404: 28: Config: Using QtTest library 6.9.0, Qt 6.9.0 (x86_64-little_endian-llp64 shared (dynamic) release build; by MSVC 2022), windows 11
      agent:2024/09/02 17:40:55 build.go:404: 28: PASS : tst_QWebEngineView::initTestCase()
      agent:2024/09/02 17:40:55 build.go:404: 28: [5092:8168:0902/174055.662:ERROR:gpu_channel_manager.cc(959)] Failed to create GLES3 context, fallback to GLES2.
      agent:2024/09/02 17:40:55 build.go:404: 28: [5092:8168:0902/174055.662:ERROR:gpu_channel_manager.cc(970)] ContextResult::kFatalFailure: Failed to create shared context for virtualization.
      agent:2024/09/02 17:40:55 build.go:404: 28: PASS : tst_QWebEngineView::deferredDelete()
       
      4) 
      tst_qtexttospeech synthesize https://testresults.qt.io/coin/log/qt/qtspeech/d7e8657be6e0567e77048119fefea0ede68a82e6/WindowsWindows_11_23H2x86_64WindowsWindows_11_23H2x86_64MSVC2022qtci-windows-11_23H2-x86_64-53-c7d4afDebugAndRelease_GenerateSBOM_Packaging_Sccache_UseConfigure_VerifySBOM/7c5ca023676c353707b341d0ff31cefa092dd5ee/test_1725309173
       
       
      agent:2024/09/02 15:04:34 build.go:404: 1: FAIL! : tst_QTextToSpeech::synthesize(sapi:text) 'running' returned FALSE. ()
      agent:2024/09/02 15:04:34 build.go:404: 1: C:\Users\qt\work\qt\qtspeech\tests\auto\qtexttospeech\tst_qtexttospeech.cpp(956) : failure location
      agent:2024/09/02 15:04:34 build.go:404: 1: SKIP : tst_QTextToSpeech::synthesizeCallback(winrt:one) Only testing with mock engine
      agent:2024/09/02 15:04:35 build.go:404: 1: C:\Users\qt\work\qt\qtspeech\tests\auto\qtexttospeech\tst_qtexttospeech.cpp(993) : failure location
      agent:2024/09/02 15:04:35 build.go:404: 1: SKIP : tst_QTextToSpeech::synthesizeCallback(winrt:several) Only testing with mock engine
      agent:2024/09/02 15:04:35 build.go:404: 1: C:\Users\qt\work\qt\qtspeech\tests\auto\qtexttospeech\tst_qtexttospeech.cpp(993) : failure location
      agent:2024/09/02 15:04:36 build.go:404: 1: PASS : tst_QTextToSpeech::synthesizeCallback(mock:one)
      agent:2024/09/02 15:04:41 build.go:404: 1: PASS : tst_QTextToSpeech::synthesizeCallback(mock:several)
      agent:2024/09/02 15:04:41 build.go:404: 1: SKIP : tst_QTextToSpeech::synthesizeCallback(sapi:one) Only testing with mock engine
      agent:2024/09/02 15:04:41 build.go:404: 1: C:\Users\qt\work\qt\qtspeech\tests\auto\qtexttospeech\tst_qtexttospeech.cpp(993) : failure location
      agent:2024/09/02 15:04:41 build.go:404: 1: SKIP : tst_QTextToSpeech::synthesizeCallback(sapi:several) Only testing with mock engine
      agent:2024/09/02 15:04:41 build.go:404: 1: C:\Users\qt\work\qt\qtspeech\tests\auto\qtexttospeech\tst_qtexttospeech.cpp(993) : failure location
      agent:2024/09/02 15:04:41 build.go:404: 1: PASS : tst_QTextToSpeech::cleanupTestCase()
      agent:2024/09/02 15:04:41 build.go:404: 1: Totals: 84 passed, 1 failed, 34 skipped, 0 blacklisted, 38795ms
      agent:2024/09/02 15:04:41 build.go:404: 1: ********* Finished testing of tst_QTextToSpeech *********
      agent:2024/09/02 15:04:41 build.go:404: 1: qt-testrunner.py INFO: Test process exited with code: 1
      agent:2024/09/02 15:04:41 build.go:404: 1: qt-testrunner.py INFO: Parsed XML file \Users\qt\work\testresults\tst_qtexttospeech-1725289442290.xml in 0.003 seconds
      agent:2024/09/02 15:04:41 build.go:404: 1: qt-testrunner.py INFO: Found 118 passes and 1 failures
      agent:2024/09/02 15:04:41 build.go:404: 1: qt-testrunner.py INFO: Some tests failed, will re-run at most 5 times.
      agent:2024/09/02 15:04:41 build.go:404: 1:
      agent:2024/09/02 15:04:41 build.go:404: 1: qt-testrunner.py INFO: Re-running testcase: synthesize:sapi:text
      agent:2024/09/02 15:04:41 build.go:404: 1: qt-testrunner.py DEBUG: Running test command line: ['C:/Users/qt/work/qt/qtspeech_standalone_tests/tests/auto/qtexttospeech/tst_qtexttospeech.exe', '-o', '\\Users\\qt\\work\\testresults\\tst_qtexttospeech-1725289481331.xml,xml', '-o', '\\Users\\qt\\work\\testresults\\tst_qtexttospeech-1725289481331.junit.xml,junitxml', '-o', '\\Users\\qt\\work\\testresults\\tst_qtexttospeech-1725289481331.txt,txt', '-o', '-,txt', 'synthesize:sapi:text']
      agent:2024/09/02 15:04:41 build.go:404: 1: ********* Start testing of tst_QTextToSpeech *********
      agent:2024/09/02 15:04:41 build.go:404: 1: Config: Using QtTest library 6.9.0, Qt 6.9.0 (x86_64-little_endian-llp64 shared (dynamic) release build; by MSVC 2022), windows 11
      agent:2024/09/02 15:04:41 build.go:404: 1: QINFO : tst_QTextToSpeech::initTestCase() Available text-to-speech engines:
      agent:2024/09/02 15:04:41 build.go:404: 1: QINFO : tst_QTextToSpeech::initTestCase() - winrt
      agent:2024/09/02 15:04:41 build.go:404: 1: QINFO : tst_QTextToSpeech::initTestCase() - mock
      agent:2024/09/02 15:04:41 build.go:404: 1: QINFO : tst_QTextToSpeech::initTestCase() - sapi
      agent:2024/09/02 15:04:41 build.go:404: 1: PASS : tst_QTextToSpeech::initTestCase()
      agent:2024/09/02 15:04:41 build.go:404: 1: PASS : tst_QTextToSpeech::synthesize(sapi:text)
       
       
       

      Attachments

        Issue Links

          No reviews matched the request. Check your Options in the drop-down menu of this sections header.

          Activity

            People

              jimis Dimitrios Apostolou
              anwojcie Anna Wojciechowska
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

                Created:
                Updated:

                Gerrit Reviews

                  There are no open Gerrit changes