Uploaded image for project: 'Qt'
  1. Qt
  2. QTBUG-76570

Flaky tests in qtquickcontrols::Tests_TreeView

    XMLWordPrintable

Details

    • macOS
    • 558eb844d59185d944c233348d2fef649e90b643 (qt/qtquickcontrols/5.12)

    Description

      Many of the tests in qtquickcontrols::Tests_TreeView have been very flaky according to the testresults grafana dashboard. Especially test_indexAt which is flaking maybe tens of times per week on MacOS_10_12, causing integrations to fail.

      EDIT: the flakiness was mitigated using delays, see the patch in gerrit. This is not optimal, and most likely a very subtle issue is hidden underneath. But the API is deprecated, and a solution is not obvious, while adding the delays vastly improves the reliability of our CI.

      Having said that here are some notes regarding the most flaky test in Tests_TreeView, which is test_indexAt(). I modified the code to print extra debug info, and I also replaced compare with tryCompare, here it is:

              function test_indexAt() {
                  var component = Qt.createComponent("treeview/treeview_1.qml")
                  compare(component.status, Component.Ready)
                  var tree = component.createObject(container);
                  verify(tree !== null, "tree created is null")
                  verify(waitForRendering(tree), "TreeView did not render on time")
                  var model = tree.model
      
                  wait(50)    // This resolves the flakiness!
      
                  // Sample each row and test
                  for (var row = 0; row < tree.__listView.count; row++) {
                      for (var x = 1; x < tree.getColumn(0).width; x += 10) {
                          var treeIndex = tree.indexAt(x, 50 * (row + 1) + 1) // offset by header height
                          console.log("treeIndex: ", treeIndex)
                          var modelIndex = model.index(row, 0)
                          if (treeIndex.row !== modelIndex.row
                              || treeIndex.column !== modelIndex.column
                              || treeIndex.internalId !== modelIndex.internalId) {
                              console.log("Test about to fail: row = " + row + ", __listView.count =" + tree.__listView.count)
                              console.log(" . . . . . . . . .  x =" + x + ", getColumn(0).width =" + tree.getColumn(0).width)
                              console.log("pairs of should-be-equal values: ",
                                          treeIndex.row, modelIndex.row,
                                          treeIndex.column, modelIndex.column,
                                          treeIndex.internalId, modelIndex.internalId)
                          }
      
                          tryCompare(treeIndex, "row", modelIndex.row)
                          tryCompare(treeIndex, "column", modelIndex.column)
                          tryCompare(treeIndex, "internalId", modelIndex.internalId)
                      }
                  }
      

      When the test fails, it is because indexAt() does not return a proper object. Success prints:

      QDEBUG : qtquickcontrols::Tests_TreeView::test_indexAt() qml: treeIndex:  QModelIndex(9,0,0x7fb6205c05f0,TestModel(0x7fb6205ac6a0))
      

      while failure prints

      QDEBUG : qtquickcontrols::Tests_TreeView::test_indexAt() qml: treeIndex:  QModelIndex()
      QDEBUG : qtquickcontrols::Tests_TreeView::test_indexAt() qml: Test about to fail: row = 0, __listView.count =10
      QDEBUG : qtquickcontrols::Tests_TreeView::test_indexAt() qml:  . . . . . . . . .  x =1, getColumn(0).width =200
      QDEBUG : qtquickcontrols::Tests_TreeView::test_indexAt() qml: pairs of should-be-equal values:  -1 0 -1 0 0 140303447317520
      QWARN  : qtquickcontrols::Tests_TreeView::test_indexAt() Model size of -3 is less than 0
      // waits 5 seconds for tryCompare to timeout ...
      FAIL!  : qtquickcontrols::Tests_TreeView::test_indexAt() property row
         Actual   (): -1
         Expected (): 0
         Loc: [/Users/qt/work/qt/qtquickcontrols/tests/auto/controls/data/tst_treeview.qml(777)]
      

      To reproduce the flakiness it is enough to run many tests from Tests_Treeview together, on a MacOS_10_12, with -mousedelay 10 as parameter to the test executable. This is my command-line:

      while ./target_wrapper.sh ./tst_controls -mousedelay 10  -import "/Users/qt/work/qt/qtquickcontrols/tests/auto/controls/../testplugin"   Tests_TreeView::test_clicked_signals Tests_TreeView::test_headerHidden  Tests_TreeView::test_indexAt; do :; done
      

      Attachments

        Issue Links

          No reviews matched the request. Check your Options in the drop-down menu of this sections header.

          Activity

            People

              qt.team.quick.subscriptions Qt Quick and Widgets Team
              jimis Dimitrios Apostolou
              Votes:
              0 Vote for this issue
              Watchers:
              7 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Gerrit Reviews

                  There are no open Gerrit changes