-
Bug
-
Resolution: Done
-
P4: Low
-
5.12, 5.13, 6.0
-
-
558eb844d59185d944c233348d2fef649e90b643 (qt/qtquickcontrols/5.12)
Many of the tests in qtquickcontrols::Tests_TreeView have been very flaky according to the testresults grafana dashboard. Especially test_indexAt which is flaking maybe tens of times per week on MacOS_10_12, causing integrations to fail.
EDIT: the flakiness was mitigated using delays, see the patch in gerrit. This is not optimal, and most likely a very subtle issue is hidden underneath. But the API is deprecated, and a solution is not obvious, while adding the delays vastly improves the reliability of our CI.
Having said that here are some notes regarding the most flaky test in Tests_TreeView, which is test_indexAt(). I modified the code to print extra debug info, and I also replaced compare with tryCompare, here it is:
function test_indexAt() {
var component = Qt.createComponent("treeview/treeview_1.qml")
compare(component.status, Component.Ready)
var tree = component.createObject(container);
verify(tree !== null, "tree created is null")
verify(waitForRendering(tree), "TreeView did not render on time")
var model = tree.model
wait(50) // This resolves the flakiness!
// Sample each row and test
for (var row = 0; row < tree.__listView.count; row++) {
for (var x = 1; x < tree.getColumn(0).width; x += 10) {
var treeIndex = tree.indexAt(x, 50 * (row + 1) + 1) // offset by header height
console.log("treeIndex: ", treeIndex)
var modelIndex = model.index(row, 0)
if (treeIndex.row !== modelIndex.row
|| treeIndex.column !== modelIndex.column
|| treeIndex.internalId !== modelIndex.internalId) {
console.log("Test about to fail: row = " + row + ", __listView.count =" + tree.__listView.count)
console.log(" . . . . . . . . . x =" + x + ", getColumn(0).width =" + tree.getColumn(0).width)
console.log("pairs of should-be-equal values: ",
treeIndex.row, modelIndex.row,
treeIndex.column, modelIndex.column,
treeIndex.internalId, modelIndex.internalId)
}
tryCompare(treeIndex, "row", modelIndex.row)
tryCompare(treeIndex, "column", modelIndex.column)
tryCompare(treeIndex, "internalId", modelIndex.internalId)
}
}
When the test fails, it is because indexAt() does not return a proper object. Success prints:
QDEBUG : qtquickcontrols::Tests_TreeView::test_indexAt() qml: treeIndex: QModelIndex(9,0,0x7fb6205c05f0,TestModel(0x7fb6205ac6a0))
while failure prints
QDEBUG : qtquickcontrols::Tests_TreeView::test_indexAt() qml: treeIndex: QModelIndex() QDEBUG : qtquickcontrols::Tests_TreeView::test_indexAt() qml: Test about to fail: row = 0, __listView.count =10 QDEBUG : qtquickcontrols::Tests_TreeView::test_indexAt() qml: . . . . . . . . . x =1, getColumn(0).width =200 QDEBUG : qtquickcontrols::Tests_TreeView::test_indexAt() qml: pairs of should-be-equal values: -1 0 -1 0 0 140303447317520 QWARN : qtquickcontrols::Tests_TreeView::test_indexAt() Model size of -3 is less than 0 // waits 5 seconds for tryCompare to timeout ... FAIL! : qtquickcontrols::Tests_TreeView::test_indexAt() property row Actual (): -1 Expected (): 0 Loc: [/Users/qt/work/qt/qtquickcontrols/tests/auto/controls/data/tst_treeview.qml(777)]
To reproduce the flakiness it is enough to run many tests from Tests_Treeview together, on a MacOS_10_12, with -mousedelay 10 as parameter to the test executable. This is my command-line:
while ./target_wrapper.sh ./tst_controls -mousedelay 10 -import "/Users/qt/work/qt/qtquickcontrols/tests/auto/controls/../testplugin" Tests_TreeView::test_clicked_signals Tests_TreeView::test_headerHidden Tests_TreeView::test_indexAt; do :; done
- relates to
-
QTBUG-53084 qtquickcontrols - tst_controls - TreeView auto tests are flaky
-
- Closed
-