Details
-
Bug
-
Resolution: Invalid
-
Not Evaluated
-
None
-
production
-
None
Description
QtWebChannel runs QML tests next to the usual C++ tests. However, the CI does not treat failures of the QML tests as failures of integration.
These is pretty severe and has lead to shipping defective code in QtWebChannel 6.2.
Example:
https://testresults.qt.io/coin/integration/qt/qtwebchannel/tasks/1638795460
Log of Ubuntu 20.04 host test execution: https://testresults.qt.io/coin/api/results/qt/qtwebchannel/91c68dd0d221a68b935117a9bfca8b76019a67c7/LinuxUbuntu_20_04x86_64LinuxUbuntu_20_04x86_64GCCqtci-linux-Ubuntu-20.04-x86_64-50-f028e3Sccache/fe490a12fe996ed543ede2f4eb5acf776eaac9ae/test_1638795470/log.txt.gz
Excerpt:
<snip> agent:2021/12/06 12:57:21 build.go:394: 2/2 Test #2: tst_qmlwebchannel ................***Failed 3.43 sec agent:2021/12/06 12:57:21 build.go:394: 50% tests passed, 1 tests failed out of 2 agent:2021/12/06 12:57:21 build.go:394: Label Time Summary: agent:2021/12/06 12:57:21 build.go:394: tests/auto/qml/tst_qmlwebchannel = 3.43 sec*proc (1 test) agent:2021/12/06 12:57:21 build.go:394: tests/auto/webchannel/tst_webchannel = 18.64 sec*proc (1 test) agent:2021/12/06 12:57:21 build.go:394: Total Test time (real) = 22.08 sec agent:2021/12/06 12:57:21 build.go:394: The following tests FAILED: agent:2021/12/06 12:57:21 build.go:394: 2 - tst_qmlwebchannel (Failed) agent:2021/12/06 12:57:21 build.go:394: Errors while running CTest agent:2021/12/06 12:57:21 build.go:394: Output from these tests are in: /home/qt/work/qt/qtwebchannel_standalone_tests/Testing/Temporary/LastTest.log agent:2021/12/06 12:57:21 build.go:394: Use "--rerun-failed --output-on-failure" to re-run the failed cases verbosely. agent:2021/12/06 12:57:21 build.go:442: Process finished with error exit status 8 but the result is ignored as instructed by Coin.
That last part is puzzling; it should definitely not be like this.