Uploaded image for project: 'Qt'
  1. Qt
  2. QTBUG-116772

qtestlib shouldn't allow the same test+testdata to be invoked multiple times in one run

    XMLWordPrintable

Details

    • Bug
    • Resolution: Unresolved
    • P3: Somewhat important
    • None
    • dev
    • Testing: qtestlib
    • None
    • 8
    • Foundation PM Staging

    Description

      From QTBUG-116771 I notice that qmltestrunner invokes the same test function again and again. By reading the XML logs one can't see a difference. QTest should exit with error, or at least warn.

      Attaching tst_multiTestCase.qml. Example logs copied from this log:

      agent:2023/08/27 23:26:26 build.go:405: 107: PASS   : qmltestrunner::A::test_do()
agent:2023/08/27 23:26:26 build.go:405: 107: PASS   : qmltestrunner::A::cleanupTestCase()
agent:2023/08/27 23:26:26 build.go:405: 107: PASS   : qmltestrunner::A::initTestCase()
agent:2023/08/27 23:26:26 build.go:405: 107: PASS   : qmltestrunner::A::test_do()
agent:2023/08/27 23:26:26 build.go:405: 107: PASS   : qmltestrunner::A::cleanupTestCase()
agent:2023/08/27 23:26:26 build.go:405: 107: PASS   : qmltestrunner::A::initTestCase()
agent:2023/08/27 23:26:26 build.go:405: 107: PASS   : qmltestrunner::A::test_do()
agent:2023/08/27 23:26:26 build.go:405: 107: PASS   : qmltestrunner::A::cleanupTestCase() 
[ ... seemingly same test runs 100 times ... ]

      Here is part of the XML file generated by qtestlib:

      <?xml version="1.0" encoding="UTF-8"?>
<TestCase name="qmltestrunner">
  <Environment>
    <QtVersion>6.7.0</QtVersion>
    <QtBuild>Qt 6.7.0 (x86_64&#x002D;little_endian&#x002D;lp64 shared (dynamic) release build; by GCC 11.4.0)</QtBuild>
    <QTestVersion>6.7.0</QTestVersion>
  </Environment>
[...]
  <TestFunction name="A::initTestCase">
    <Incident type="pass" file="" line="0" />
    <Duration msecs="0.16281"/>
  </TestFunction>
  <TestFunction name="A::test_do">
    <Incident type="pass" file="" line="0" />
    <Duration msecs="1.27274"/>
  </TestFunction>
  <TestFunction name="A::cleanupTestCase">
    <Incident type="pass" file="" line="0" />
    <Duration msecs="0.060903"/>
  </TestFunction>
  <TestFunction name="A::initTestCase">
    <Incident type="pass" file="" line="0" />
    <Duration msecs="0.1256"/>
  </TestFunction>
  <TestFunction name="A::test_do">
    <Incident type="pass" file="" line="0" />
    <Duration msecs="1.65724"/>
  </TestFunction>
  <TestFunction name="A::cleanupTestCase">
    <Incident type="pass" file="" line="0" />
    <Duration msecs="0.053843"/>
  </TestFunction>
  <TestFunction name="A::initTestCase">
    <Incident type="pass" file="" line="0" />
    <Duration msecs="0.064591"/>
  </TestFunction>
[...]
  <Duration msecs="3253.55"/>
</TestCase>

      Attachments

        Issue Links

          relates to

          Bug - A problem which impairs or prevents the functions of the product. QTBUG-116771 qmltestrunner runs identically named tests that are different

          • P3: Somewhat important - Should be fixed, but doesn't affect the release in any way.
          • Reported
          No reviews matched the request. Check your Options in the drop-down menu of this sections header.

          Activity

            People

              Eddy Edward Welbourne
              jimis Dimitrios Apostolou
              Vladimir Minenko Vladimir Minenko
              Alex Blasche Alex Blasche
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

                Created:
                Updated:

                Gerrit Reviews

                  There are no open Gerrit changes