Uploaded image for project: 'Qt'
  1. Qt
  2. QTBUG-88816

Export test results for consumption by Coin's test result parser

    XMLWordPrintable

Details

    Description

      Background

      When Qt is built with qmake, Coin's legacy instructions has a bunch of hardcoded behaviour to create a test plan, run the test, extract the test results from xml files, and feed that data to Influx / Grafana for flaky test inspection.

      Check https://testresults.qt.io/coin/logview/qt/qtsvg/09dd827c922c06e6d8d995e039b9de61df32ca3e/LinuxCentOS_8_1x86_64LinuxCentOS_8_1x86_64GCCqtci-linux-CentOS-8.1-x86_64-50-64ffe9UseLegacyInstructions/ed6289f7b0615a85920b2dcf137f65842e4fe1c7/build_1606211670 for example
      and search for TestPlan and UploadTestPlan keywords, and the instructions between them.

      Also check https://testresults.qt.io/coin/logview/qt/qtsvg/09dd827c922c06e6d8d995e039b9de61df32ca3e/LinuxCentOS_8_1x86_64LinuxCentOS_8_1x86_64GCCqtci-linux-CentOS-8.1-x86_64-50-64ffe9UseLegacyInstructions/ed6289f7b0615a85920b2dcf137f65842e4fe1c7/test_1606211671

      and see how each test is ran as a separate instruction (RunQtUnitTest) and the qmake built test outputs the test result to an xml file

      Generating test results with TESTARGS=-o /home/qt/work/testresults/tests/auto/qsvggenerator378963611,xml -o /home/qt/work/testresults/tests/auto/qsvggenerator378963611-junit,xunitxml -o -,txt
      

      These xml files are then collected and uploaded by the Upload test results instruction for further processing in Coin, and feeding InfluxDB.

      CMake

      When Qt is built with CMake, all of the above currently does not run.

      Qt + CMake is not built using hardcoded Legacy instructions, but rather yaml files, and we can't really hardcode running each test as a separate RunQtUnitTest instruction.

      Ideally we don't need to create a testplan, and can just run ctest directly to run all tests.

      The remaining question for the build system is how to create the xml files.
      There's an open feature request to add junit xml output to ctest, which is not yet implemented https://gitlab.kitware.com/cmake/cmake/-/issues/18654 . I spoke with Tor Arne, and he says it's not worth fixing it upstream, and we should instead use the QtTestLib provided xml generation capabilities.

      So we will most likely have to create a shell script wrapper for each test (maybe using CMake as the script language) like qmake does, so that TESTARGS and TESTRUNNER environment variables (like the ones above) are taken into account, and thus let the QtTestLib executables create the xml files.

      An open question is what kind of parameters they should be, if Coin expects anything in particular, can we replicate the same ones as done by Coin for qmake integrations.

      Coin will also have to be adapted to use the new ctest approach (if it works), bypassing the testplan creation.

      Necessary coin changes tracked in COIN-643

      Attachments

        Issue Links

          For Gerrit Dashboard: QTBUG-88816
          # Subject Branch Project Status CR V

          Activity

            People

              semlanik Alexey Edelev
              alexandru.croitor Alexandru Croitor
              Votes:
              0 Vote for this issue
              Watchers:
              7 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Gerrit Reviews

                  There are no open Gerrit changes