Details
-
Task
-
Resolution: Done
-
P2: Important
-
6.3
-
None
-
-
13
-
qtbase/2c2cdb959f0d69cc56036ae17bc60009df8c7b4b qtbase/326d94e94b513a7d5be17493d57d31cf3329cb72 qtbase/427be739ff63eac32cae4f44260f52ccece14be7 qtbase/09fd39895913fddcd5909d217c2718976a27f6ce qtbase/b9f7add531ceab2590e2b24dec7b495950f9870a qtbase/77a93e6
-
Team 1 Foundation_Sprint 43, Team 1 Foundation_Sprint 44, Team 1 Foundation_Sprint 45, Team 1 Foundation_Sprint 46, Team 1 Foundation_Sprint 47, Team 1 Foundation_Sprint 48, Team One Foundation Sprint 49, Team One Foundation Sprint 50, Team One Foundation Sprint 51, Team One Foundation Sprint 52, Team A Foundation Sprint 53, Team A Foundation Sprint 54
Description
At present the APIs for logging in QTest are a bit incoherent:
- the QTestLog methods addIncident() and addMessage() split things to be reported a bit eccentrically – B?XFAIL is never the decisive result of a test, while SKIP is, but the former is reported via addIncident() with the things that are decisive, while the latter is reported by addMessage(), along with others that aren't (although QCritical and QFatal typically accompany a FAIL that is); and addIncident() lacks a parameter from the caller that could tell it when the incident isn't decisive (as happens if the decisive incident didn't return from the test, e.g. a B?XPASS/Continue or a SKIP or B?FAIL in a helper function, such as an event handler in a test that uses an event loop).
- in addition, addMessage() rolls together internal logging from QTest, such as the Info output from -v1 and -v2 command-line options, with messages originating in the usual Qt logging infrastructure; while the enum does provide a way to separate these, it might make more sense to have separate functions, to separate test-framework messages from messages originating in the actual test or the code-under-test.
the counting is out of whack and It's all undocumentedOK, I've just fixed that ;^)
We need to do a design review and at least come up with a plan of action for how to move forward. vestbo has begun doing that (and I've helped a bit); this ticket is really a post hoc way to track our progress with work already begun ! It can also serve as a place to document what we decide and why, along with work we do to move forward with it.
Attachments
Issue Links
- depends on
-
QTBUG-95661 Test result counts are incorrect
- Closed
- relates to
-
QTBUG-101291 After a QFAIL in a _data() function the test associated with it is still executed
- Reported
-
QTBUG-67702 QTest::ignoreMessage() fails to match regex against warning string
- Closed
-
QTBUG-93438 Android unit tests capture C++ logs, not Java logs
- Closed
-
QTBUG-96353 Test run procedure and flaky detection
- Closed