Details
-
Bug
-
Resolution: Incomplete
-
P2: Important
-
None
-
production
-
None
-
e4fe162f0 (dev), 741981ff3 (dev), 5fafd262a (dev), 48aedc5c9 (dev), 51f283f00 (dev), 8b17904d6 (dev), 2d9daf0b9 (dev), bc9c99021 (dev), cb9589860 (dev), 074e3a90a (dev), d08f85f4a (dev), 75400d7c5 (dev)
Description
In preparation for Bugfix week, H2 2024, we want to blacklist the worst offending flaky tests in an effort to improve overall CI smoothness. Since a release is not immediately pending the bugfix week, this should be a relatively safe operation, though measures will be taken immedately after bugfix week to verify that the blacklisted tests are not hard-failing before un-blacklisting them again.
Axel Sporel has taken a cursory look at the proposed blacklist list below, and found no obvious customer-reported bugs which would relate to the tests being blacklisted.
| test_case | test_function | flaky_instances | fail_instances |
| inputpanel | tst_plugin::test_hangulInputMethod | 1 | 28 |
| tst_QWidget_window | mouseMoveWithPopup | 5 | 18 |
| tst_lupdate | good | 16 | 17 |
| Tst_QMqttSubscription | qtbug_106203 | 16 | 12 |
| tst_QHostInfo | lookupConnectToLambda | 56 | 9 |
| tst_QQuickMultiPointTouchArea | inFlickable | 51 | 9 |
| tst_Http2 | duplicateRequestsWithAborts | 31 | 8 |
| tst_QNetworkReply | authenticationCacheAfterCancel | 101 | 5 |
| tst_clientSSL | testRun | 61 | 5 |
| tst_QWidget | multipleToplevelFocusCheck | 9 | 5 |
| tst_QNetworkReply | ioGetFromHttpWithProxyAuth | 1 | 5 |
| tst_QWidget | hoverPosition | 60 | 4 |
| tst_QNetworkReply | httpReUsingConnectionSequential | 54 | 4 |
| tst_Printing | printFromPdfViewer | 35 | 4 |
| tst_QTcpSocket | waitForConnectedInHostLookupSlot | 24 | 4 |
| tst_QNetworkReply | ioHttpRedirectPostPut | 11 | 4 |
| tst_QNetworkReply | ioHttpRedirectFromLocalToRemote | 7 | 4 |
| tst_QHostInfo | reverseLookup | 3 | 4 |
| tst_qml | itemAndWindowGeometry | 1 | 4 |
| declarative_ui | MapItems::test_no_items_on_map | 238 | 3 |
Attachments
| For Gerrit Dashboard: QTQAINFRA-6482 | ||||||
|---|---|---|---|---|---|---|
| # | Subject | Branch | Project | Status | CR | V |
| 581639,5 | Blacklist flaky tests for Bugfix week 2024/H2 | dev | qt/qtbase | Status: MERGED | +2 | 0 |
| 581640,4 | Blacklist flaky tests for Bugfix week 2024/H2 | dev | qt/qtdeclarative | Status: MERGED | +2 | 0 |
| 581641,3 | Blacklist flaky tests for Bugfix week 2024/H2 | dev | qt/qtmqtt | Status: ABANDONED | -1 | 0 |
| 581642,4 | Blacklist flaky tests for Bugfix week 2024/H2 | dev | qt/qttools | Status: MERGED | +2 | 0 |
| 581643,4 | Blacklist flaky tests for Bugfix week 2024/H2 | dev | qt/qtwebengine | Status: MERGED | +2 | 0 |
| 581653,3 | Blacklist flaky tests for Bugfix week 2024/H2 | dev | qt/qtvirtualkeyboard | Status: MERGED | +2 | 0 |
| 581654,2 | Blacklist flaky tests for Bugfix week 2024/H2 | dev | qt/qtlocation | Status: MERGED | +2 | 0 |