Loading...

Type: Bug
Resolution: Unresolved
Priority: P3: Somewhat important
Fix Version/s: None
Affects Version/s: 6.11
Component/s: Testing: qtestlib, Testing: QuickTest
Labels:
None

Some test functions run in different variations, aside from datatags.

For example, we run the following command-line to execute only one function+datatag:

$ tst_qquickpopup fadeDimmer:modal

********* Start testing of tst_QQuickPopup *********
Config: Using QtTest library 6.11.0, Qt 6.11.0 (x86_64-little_endian-lp64 shared (dynamic) release build; by Android (12470979, +pgo, +bolt, +lto, +mlgo, based on r522817c) Clang 18.0.3 (https://android.googlesource.com/toolchain/llvm-project d8003a456d14a3deb8054cdaa529ffbf02d9b262)), android 15.0
PASS   : tst_QQuickPopup::Basic::initTestCase()
PASS   : tst_QQuickPopup::Basic::fadeDimmer(modal)
PASS   : tst_QQuickPopup::Basic::cleanupTestCase()
PASS   : tst_QQuickPopup::Fusion::initTestCase()
PASS   : tst_QQuickPopup::Fusion::fadeDimmer(modal)
PASS   : tst_QQuickPopup::Fusion::cleanupTestCase()
PASS   : tst_QQuickPopup::FluentWinUI3::initTestCase()
PASS   : tst_QQuickPopup::FluentWinUI3::fadeDimmer(modal)
PASS   : tst_QQuickPopup::FluentWinUI3::cleanupTestCase()
PASS   : tst_QQuickPopup::Imagine::initTestCase()
PASS   : tst_QQuickPopup::Imagine::fadeDimmer(modal)
PASS   : tst_QQuickPopup::Imagine::cleanupTestCase()
PASS   : tst_QQuickPopup::Material::initTestCase()
PASS   : tst_QQuickPopup::Material::fadeDimmer(modal)
PASS   : tst_QQuickPopup::Material::cleanupTestCase()
PASS   : tst_QQuickPopup::Universal::initTestCase()
PASS   : tst_QQuickPopup::Universal::fadeDimmer(modal)
PASS   : tst_QQuickPopup::Universal::cleanupTestCase()
Totals: 18 passed, 0 failed, 0 skipped, 0 blacklisted, 6218ms
********* Finished testing of tst_QQuickPopup *********

What are the variations of Basic, Fusion etc and how should they be recorded in the testresults database?
For now they are all recorded as re-runs of the same test_function+test_datatag, and it's not possible to differentiate. They appear as duplicate datatags when examining the data in the database.

Here is the XML log of the above testcase. It is apparently missing the "theme" info.

<?xml version="1.0" encoding="UTF-8"?>
<TestCase name="tst_QQuickPopup">
  <Environment>
    <QtVersion>6.11.0</QtVersion>
    <QtBuild>Qt 6.11.0 (x86_64&#x002D;little_endian&#x002D;lp64 shared (dynamic) release build; by Android (12470979, +pgo, +bolt, +lto, +mlgo, based on r522817c) Clang 18.0.3 (https://android.googlesource.com/toolchain/llvm&#x002D;project d8003a456d14a3deb8054cdaa529ffbf02d9b262))</QtBuild>
    <QTestVersion>6.11.0</QTestVersion>
  </Environment>
  <TestFunction name="initTestCase">
    <Incident type="pass" file="" line="0" />
    <Duration msecs="0.189858"/>
  </TestFunction>
  <TestFunction name="fadeDimmer">
    <Incident type="pass" file="" line="0">
      <DataTag><![CDATA[modal]]></DataTag>
    </Incident>
    <Duration msecs="1087.79"/>
  </TestFunction>
  <TestFunction name="cleanupTestCase">
    <Incident type="pass" file="" line="0" />
    <Duration msecs="0.285954"/>
  </TestFunction>
  <TestFunction name="initTestCase">
    <Incident type="pass" file="" line="0" />
    <Duration msecs="0.142962"/>
  </TestFunction>
  <TestFunction name="fadeDimmer">
    <Incident type="pass" file="" line="0">
      <DataTag><![CDATA[modal]]></DataTag>
    </Incident>
    <Duration msecs="987.538"/>
  </TestFunction>
  <TestFunction name="cleanupTestCase">
    <Incident type="pass" file="" line="0" />
    <Duration msecs="0.075116"/>
  </TestFunction>
  <TestFunction name="initTestCase">
    <Incident type="pass" file="" line="0" />
    <Duration msecs="0.173241"/>
  </TestFunction>
  <TestFunction name="fadeDimmer">
    <Incident type="pass" file="" line="0">
      <DataTag><![CDATA[modal]]></DataTag>
    </Incident>
    <Duration msecs="1067.86"/>
  </TestFunction>
  <TestFunction name="cleanupTestCase">
    <Incident type="pass" file="" line="0" />
    <Duration msecs="0.05846"/>
  </TestFunction>
  <TestFunction name="initTestCase">
    <Incident type="pass" file="" line="0" />
    <Duration msecs="0.761143"/>
  </TestFunction>
  <TestFunction name="fadeDimmer">
    <Incident type="pass" file="" line="0">
      <DataTag><![CDATA[modal]]></DataTag>
    </Incident>
    <Duration msecs="930.94"/>
  </TestFunction>
  <TestFunction name="cleanupTestCase">
    <Incident type="pass" file="" line="0" />
    <Duration msecs="0.040437"/>
  </TestFunction>
  <TestFunction name="initTestCase">
    <Incident type="pass" file="" line="0" />
    <Duration msecs="0.149435"/>
  </TestFunction>
  <TestFunction name="fadeDimmer">
    <Incident type="pass" file="" line="0">
      <DataTag><![CDATA[modal]]></DataTag>
    </Incident>
    <Duration msecs="1208.05"/>
  </TestFunction>
  <TestFunction name="cleanupTestCase">
    <Incident type="pass" file="" line="0" />
    <Duration msecs="0.053628"/>
  </TestFunction>
  <TestFunction name="initTestCase">
    <Incident type="pass" file="" line="0" />
    <Duration msecs="0.199507"/>
  </TestFunction>
  <TestFunction name="fadeDimmer">
    <Incident type="pass" file="" line="0">
      <DataTag><![CDATA[modal]]></DataTag>
    </Incident>
    <Duration msecs="917.525"/>
  </TestFunction>
  <TestFunction name="cleanupTestCase">
    <Incident type="pass" file="" line="0" />
    <Duration msecs="0.030343"/>
  </TestFunction>
  <Duration msecs="6218.03"/>
</TestCase>

EDIT: And here is the JUnit XML logfile, which apparently records the information missing above:

<?xml version="1.0" encoding="UTF-8" ?>
<testsuite name="tst_QQuickPopup" timestamp="2025&#x002D;08&#x002D;20T07:55:50" hostname="localhost" tests="23" failures="0" errors="0" skipped="0" time="6.218">
  <properties>
    <property name="QTestVersion" value="6.11.0"/>
    <property name="QtVersion" value="6.11.0"/>
    <property name="QtBuild" value="Qt 6.11.0 (x86_64&#x002D;little_endian&#x002D;lp64 shared (dynamic) release build; by Android (12470979, +pgo, +bolt, +lto, +mlgo, based on r522817c) Clang 18.0.3 (https://android.googlesource.com/toolchain/llvm&#x002D;project d8003a456d14a3deb8054cdaa529ffbf02d9b262))"/>
  </properties>
  <testcase name="initTestCase" classname="tst_QQuickPopup::Basic" time="0.000"/>
  <testcase name="fadeDimmer(modal)" classname="tst_QQuickPopup::Basic" time="1.088"/>
  <testcase name="cleanupTestCase" classname="tst_QQuickPopup::Basic" time="0.000"/>
  <testcase name="initTestCase" classname="tst_QQuickPopup::Fusion" time="0.000"/>
  <testcase name="fadeDimmer" classname="tst_QQuickPopup::Fusion" time="0.000"/>
  <testcase name="fadeDimmer(modal)" classname="tst_QQuickPopup::Fusion" time="0.988"/>
  <testcase name="cleanupTestCase" classname="tst_QQuickPopup::Fusion" time="0.000"/>
  <testcase name="initTestCase" classname="tst_QQuickPopup::FluentWinUI3" time="0.000"/>
  <testcase name="fadeDimmer" classname="tst_QQuickPopup::FluentWinUI3" time="0.000"/>
  <testcase name="fadeDimmer(modal)" classname="tst_QQuickPopup::FluentWinUI3" time="1.068"/>
  <testcase name="cleanupTestCase" classname="tst_QQuickPopup::FluentWinUI3" time="0.000"/>
  <testcase name="initTestCase" classname="tst_QQuickPopup::Imagine" time="0.001"/>
  <testcase name="fadeDimmer" classname="tst_QQuickPopup::Imagine" time="0.000"/>
  <testcase name="fadeDimmer(modal)" classname="tst_QQuickPopup::Imagine" time="0.931"/>
  <testcase name="cleanupTestCase" classname="tst_QQuickPopup::Imagine" time="0.000"/>
  <testcase name="initTestCase" classname="tst_QQuickPopup::Material" time="0.000"/>
  <testcase name="fadeDimmer" classname="tst_QQuickPopup::Material" time="0.000"/>
  <testcase name="fadeDimmer(modal)" classname="tst_QQuickPopup::Material" time="1.208"/>
  <testcase name="cleanupTestCase" classname="tst_QQuickPopup::Material" time="0.000"/>
  <testcase name="initTestCase" classname="tst_QQuickPopup::Universal" time="0.000"/>
  <testcase name="fadeDimmer" classname="tst_QQuickPopup::Universal" time="0.000"/>
  <testcase name="fadeDimmer(modal)" classname="tst_QQuickPopup::Universal" time="0.917"/>
  <testcase name="cleanupTestCase" classname="tst_QQuickPopup::Universal" time="0.000"/>
</testsuite>

All the above logs come from this workitem:
https://testresults.qt.io/coin/log/qt/qtdeclarative/d3eb0dd3f259593317cbf6250d38cd082df1f0bc/LinuxRHEL_8_10x86_64AndroidAndroid_ANYx86_64GCCqtci-linux-RHEL-8.10-x86_64-51-5c5fabAndroidTestRun_GenerateSBOM_Packaging_Sccache_UseConfigure_VerifySBOM_WarningsAreErrors/8e0e07b5bf5a84716b07f22690ac5bb015a0b91f/test_1755696174

resulted from

QTQAINFRA-7349 Wrong handling of failed tests re-run on Android

In Progress

QTQAINFRA-7351 qt-testrunner.py does not record the test executable properly on Android

In Review

- Issue Only
- Show All Reviews
- Show Open Reviews

For Gerrit Dashboard: QTBUG-139332
#	Subject	Branch	Project	Status	CR	V
670570,1	Convert e.g. tst_controls::Basic to tst_controls_Basic	dev	qt/qtdeclarative	Status: NEW	+1	0

Details

Description

Here is the XML log of the above testcase. It is apparently missing the "theme" info.

EDIT: And here is the JUnit XML logfile, which apparently records the information missing above:

Attachments

Issue Links

Gerrit Reviews

Activity

People

Dates

Gerrit Reviews