Details
-
Bug
-
Resolution: Done
-
P4: Low
-
5.12, 5.13, 6.0
-
-
558eb844d59185d944c233348d2fef649e90b643 (qt/qtquickcontrols/5.12)
Description
Many of the tests in qtquickcontrols::Tests_TreeView have been very flaky according to the testresults grafana dashboard. Especially test_indexAt which is flaking maybe tens of times per week on MacOS_10_12, causing integrations to fail.
EDIT: the flakiness was mitigated using delays, see the patch in gerrit. This is not optimal, and most likely a very subtle issue is hidden underneath. But the API is deprecated, and a solution is not obvious, while adding the delays vastly improves the reliability of our CI.
Having said that here are some notes regarding the most flaky test in Tests_TreeView, which is test_indexAt(). I modified the code to print extra debug info, and I also replaced compare with tryCompare, here it is:
function test_indexAt() { var component = Qt.createComponent("treeview/treeview_1.qml") compare(component.status, Component.Ready) var tree = component.createObject(container); verify(tree !== null, "tree created is null") verify(waitForRendering(tree), "TreeView did not render on time") var model = tree.model wait(50) // This resolves the flakiness! // Sample each row and test for (var row = 0; row < tree.__listView.count; row++) { for (var x = 1; x < tree.getColumn(0).width; x += 10) { var treeIndex = tree.indexAt(x, 50 * (row + 1) + 1) // offset by header height console.log("treeIndex: ", treeIndex) var modelIndex = model.index(row, 0) if (treeIndex.row !== modelIndex.row || treeIndex.column !== modelIndex.column || treeIndex.internalId !== modelIndex.internalId) { console.log("Test about to fail: row = " + row + ", __listView.count =" + tree.__listView.count) console.log(" . . . . . . . . . x =" + x + ", getColumn(0).width =" + tree.getColumn(0).width) console.log("pairs of should-be-equal values: ", treeIndex.row, modelIndex.row, treeIndex.column, modelIndex.column, treeIndex.internalId, modelIndex.internalId) } tryCompare(treeIndex, "row", modelIndex.row) tryCompare(treeIndex, "column", modelIndex.column) tryCompare(treeIndex, "internalId", modelIndex.internalId) } }
When the test fails, it is because indexAt() does not return a proper object. Success prints:
QDEBUG : qtquickcontrols::Tests_TreeView::test_indexAt() qml: treeIndex: QModelIndex(9,0,0x7fb6205c05f0,TestModel(0x7fb6205ac6a0))
while failure prints
QDEBUG : qtquickcontrols::Tests_TreeView::test_indexAt() qml: treeIndex: QModelIndex() QDEBUG : qtquickcontrols::Tests_TreeView::test_indexAt() qml: Test about to fail: row = 0, __listView.count =10 QDEBUG : qtquickcontrols::Tests_TreeView::test_indexAt() qml: . . . . . . . . . x =1, getColumn(0).width =200 QDEBUG : qtquickcontrols::Tests_TreeView::test_indexAt() qml: pairs of should-be-equal values: -1 0 -1 0 0 140303447317520 QWARN : qtquickcontrols::Tests_TreeView::test_indexAt() Model size of -3 is less than 0 // waits 5 seconds for tryCompare to timeout ... FAIL! : qtquickcontrols::Tests_TreeView::test_indexAt() property row Actual (): -1 Expected (): 0 Loc: [/Users/qt/work/qt/qtquickcontrols/tests/auto/controls/data/tst_treeview.qml(777)]
To reproduce the flakiness it is enough to run many tests from Tests_Treeview together, on a MacOS_10_12, with -mousedelay 10 as parameter to the test executable. This is my command-line:
while ./target_wrapper.sh ./tst_controls -mousedelay 10 -import "/Users/qt/work/qt/qtquickcontrols/tests/auto/controls/../testplugin" Tests_TreeView::test_clicked_signals Tests_TreeView::test_headerHidden Tests_TreeView::test_indexAt; do :; done
Attachments
Issue Links
- relates to
-
QTBUG-53084 qtquickcontrols - tst_controls - TreeView auto tests are flaky
-
- Closed
-