Uploaded image for project: 'Qt'
  1. Qt
  2. QTBUG-116771

qmltestrunner runs identically named tests that are different

    XMLWordPrintable

Details

    • Bug
    • Resolution: Unresolved
    • P3: Somewhat important
    • Some future release
    • dev
    • QML: Tooling
    • None

    Description

       
      Can't tell apart tests running under qmltestrunner.
      Example copied from this log but this is not the only one:

      agent:2023/08/27 23:26:26 build.go:405: 107: PASS   : qmltestrunner::A::test_do()
agent:2023/08/27 23:26:26 build.go:405: 107: PASS   : qmltestrunner::A::cleanupTestCase()
agent:2023/08/27 23:26:26 build.go:405: 107: PASS   : qmltestrunner::A::initTestCase()
agent:2023/08/27 23:26:26 build.go:405: 107: PASS   : qmltestrunner::A::test_do()
agent:2023/08/27 23:26:26 build.go:405: 107: PASS   : qmltestrunner::A::cleanupTestCase()
agent:2023/08/27 23:26:26 build.go:405: 107: PASS   : qmltestrunner::A::initTestCase()
agent:2023/08/27 23:26:26 build.go:405: 107: PASS   : qmltestrunner::A::test_do()
agent:2023/08/27 23:26:26 build.go:405: 107: PASS   : qmltestrunner::A::cleanupTestCase() 
[ ... seemingly same test runs 100 times ... ]

      Here is part of the XML file generated by qtestlib:

      <?xml version="1.0" encoding="UTF-8"?>
<TestCase name="qmltestrunner">
  <Environment>
    <QtVersion>6.7.0</QtVersion>
    <QtBuild>Qt 6.7.0 (x86_64&#x002D;little_endian&#x002D;lp64 shared (dynamic) release build; by GCC 11.4.0)</QtBuild>
    <QTestVersion>6.7.0</QTestVersion>
  </Environment>
[...]
  <TestFunction name="A::initTestCase">
    <Incident type="pass" file="" line="0" />
    <Duration msecs="0.16281"/>
  </TestFunction>
  <TestFunction name="A::test_do">
    <Incident type="pass" file="" line="0" />
    <Duration msecs="1.27274"/>
  </TestFunction>
  <TestFunction name="A::cleanupTestCase">
    <Incident type="pass" file="" line="0" />
    <Duration msecs="0.060903"/>
  </TestFunction>
  <TestFunction name="A::initTestCase">
    <Incident type="pass" file="" line="0" />
    <Duration msecs="0.1256"/>
  </TestFunction>
  <TestFunction name="A::test_do">
    <Incident type="pass" file="" line="0" />
    <Duration msecs="1.65724"/>
  </TestFunction>
  <TestFunction name="A::cleanupTestCase">
    <Incident type="pass" file="" line="0" />
    <Duration msecs="0.053843"/>
  </TestFunction>
  <TestFunction name="A::initTestCase">
    <Incident type="pass" file="" line="0" />
    <Duration msecs="0.064591"/>
  </TestFunction>
[...]
  <Duration msecs="3253.55"/>
</TestCase>
      • The executable is always qmltestrunner without any parameters. AFAICT all the QML files from the CWD (current working directory) are executed. For the example snippets I attach here, the CWD is qtdeclarative/tests/auto/qmltest/selftests but the file executed is tst_multiTestCase.qml.
        • It would be better to explicitly pass only one QML file as argument to qmltestrunner, and execute that one.
      • The testcase is always qmltestrunner (in common tests this is tst_whatever)  when qmltestrunner is involved. I believe this is a bug. In the snippet example here, what is printed in the log is qmltestrunner::A::test_do(). Either "tst_multiTestCase.qml" from the filename, or this "A" from the testfunction should be in the testCase.
      • Many A::test_do "functions" are being called, and I can't tell them apart. Is it the same test executing again and again? IMO qtestlib should not allow this, but I'll file a separate bug for it. (EDIT: filed QTBUG-116772)

      Attaching the file tst_multiTestCase.qml as an example, but I believe the problem is more generic.

      Attachments

        Issue Links

          relates to

          Bug - A problem which impairs or prevents the functions of the product. QTBUG-116772 qtestlib shouldn't allow the same test+testdata to be invoked multiple times in one run

          • P3: Somewhat important - Should be fixed, but doesn't affect the release in any way.
          • Reported
          For Gerrit Dashboard: QTBUG-116771
          # Subject Branch Project Status CR V
          579367,1 Differentiate testcases dev qt/qtdeclarative Status: ABANDONED 0 -1

          Activity

            People

              qtqmlteam Qt Qml Team User
              jimis Dimitrios Apostolou
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

                Created:
                Updated:

                Gerrit Reviews

                  There are no open Gerrit changes