Uploaded image for project: 'Qt'
  1. Qt
  2. QTBUG-103386

A test can FAIL then PASS and no failure will be recorded in the log

    XMLWordPrintable

Details

    • Bug
    • Resolution: Unresolved
    • P2: Important
    • None
    • 6.4
    • Testing: qtestlib
    • None
    • macOS

    Description

      In this nightly build and this test log of qtpositioning:

      test 4
          Start 4: tst_opcua
      4: Test command: /Applications/CMake.app/Contents/bin/cmake "-P" "/Users/qt/work/qt/qtopcua_standalone_tests/tests/auto/declarative/tst_opcuaWrapperRelWithDebInfo.cmake"
      4: Test timeout computed to be: 1500
      4: qt-testrunner.py     INFO: Will write XML test logs to directory COIN_CTEST_RESULTSDIR=/Users/qt/work/testresults
      4: qt-testrunner.py    DEBUG: Running test command line: ['/Users/qt/work/qt/qtopcua_standalone_tests/tests/auto/declarative/tst_opcua', '-import', '/Users/qt/work/qt/qtopcua/tests/auto/declarative/../../../src/plugins/declarative', '-o', '/Users/qt/work/testresults/tst_opcua-1651870032738.xml,xml', '-o', '-,txt']
      4: Server Path: "/Users/qt/work/qt/qtopcua_standalone_tests/tests/auto/declarative/../../open62541-testserver/open62541-testserver.app/Contents/MacOS/open62541-testserver"
      4: Starting test server
      4: qml: Setting up AbsoluteNodeTest for open62541
      4: qt.opcua.plugins.qml: Client instance is invalid
      4: qt.opcua.plugins.qml: Client instance is invalid
      4: qt.opcua.plugins.qml: Client instance is invalid
      4: qt.opcua.plugins.qml: Client instance is invalid
      4: qt.opcua.plugins.qml: Client instance is invalid
      4: qt.opcua.plugins.qml: Client instance is invalid
      4: qt.opcua.plugins.qml: Client instance is invalid
      4: qt.opcua.plugins.qml: Client instance is invalid
      4: qt.opcua.plugins.qml: Client instance is invalid
      4: qt.opcua.plugins.qml: Client instance is invalid
      4: qt.opcua.plugins.qml: Client instance is invalid
      4: qt.opcua.plugins.qml: Client instance is invalid
      4: qt.opcua.plugins.qml: Client instance is invalid
      4: qt.opcua.plugins.qml: Client instance is invalid
      4: qt.opcua.plugins.qml: Client instance is invalid
      4: qt.opcua.plugins.qml: Client instance is invalid
      4: qt.opcua.plugins.qml: Client instance is invalid
      4: qt.opcua.plugins.qml: Client instance is invalid
      4: qt.opcua.plugins.qml: Client instance is invalid
      4: qt.opcua.plugins.qml: Client instance is invalid
      4: qt.opcua.plugins.qml: Client instance is invalid
      4: qt.opcua.plugins.qml: Client instance is invalid
      4: qt.opcua.plugins.qml: Client instance is invalid
      4: qt.opcua.plugins.qml: Client instance is invalid
      4: qt.opcua.plugins.open62541.sdk.userland: "AcceptAll Certificate Verification. Any remote certificate will be accepted."
      4: qml: Running AbsoluteNodeTest with open62541
      4: qt.opcua.plugins.open62541.sdk.securechannel: "Connection 19 | SecureChannel 1 | Opened SecureChannel with SecurityPolicy http://opcfoundation.org/UA/SecurityPolicy#None"
      4: qt.opcua.plugins.open62541.sdk.client: "Client Status: ChannelState: Open, SessionState: Closed, ConnectStatus: Good"
      4: qt.opcua.plugins.open62541.sdk.client: "Client Status: ChannelState: Closed, SessionState: Closed, ConnectStatus: Good"
      4: qt.opcua.plugins.open62541.sdk.userland: "AcceptAll Certificate Verification. Any remote certificate will be accepted."
      4: qt.opcua.plugins.open62541.sdk.securechannel: "Connection 24 | SecureChannel 2 | Opened SecureChannel with SecurityPolicy http://opcfoundation.org/UA/SecurityPolicy#None"
      4: qt.opcua.plugins.open62541.sdk.client: "Client Status: ChannelState: Open, SessionState: Closed, ConnectStatus: Good"
      4: qt.opcua.plugins.open62541.sdk.client: "Client Status: ChannelState: Closed, SessionState: Closed, ConnectStatus: Good"
      4: qt.opcua.plugins.open62541.sdk.userland: "AcceptAll Certificate Verification. Any remote certificate will be accepted."
      4: qt.opcua.plugins.open62541.sdk.securechannel: "Connection 24 | SecureChannel 3 | Opened SecureChannel with SecurityPolicy http://opcfoundation.org/UA/SecurityPolicy#None"
      4: qt.opcua.plugins.open62541.sdk.client: "Client Status: ChannelState: Open, SessionState: Closed, ConnectStatus: Good"
      4: qt.opcua.plugins.open62541.sdk.client: "Selected Endpoint opc.tcp://localhost:43344/ with SecurityMode None and SecurityPolicy http://opcfoundation.org/UA/SecurityPolicy#None"
      4: qt.opcua.plugins.open62541.sdk.client: "Selected UserTokenPolicy open62541-anonymous-policy with UserTokenType Anonymous and SecurityPolicy http://opcfoundation.org/UA/SecurityPolicy#None"
      4: qt.opcua.plugins.open62541.sdk.client: "Client Status: ChannelState: Open, SessionState: Created, ConnectStatus: Good"
      4: qt.opcua.plugins.open62541.sdk.client: "Client Status: ChannelState: Open, SessionState: Activated, ConnectStatus: Good"
      4: ********* Start testing of opcua *********
      4: Config: Using QtTest library 6.4.0, Qt 6.4.0 (arm64-little_endian-lp64 shared (dynamic) release build; by Clang 13.0.0 (clang-1300.0.29.30) (Apple)), macos 11.6
      4: PASS   : opcua::AbsoluteNodeTest: open62541: Create String Node Id::initTestCase()
      4: PASS   : opcua::AbsoluteNodeTest: open62541: Create GUID Node Id::initTestCase()
      4: PASS   : opcua::AbsoluteNodeTest: open62541: Create GUID Node Id::test_nodeTest()
      4: PASS   : opcua::AbsoluteNodeTest: open62541: Create Opaque Node Id::initTestCase()
      4: PASS   : opcua::AbsoluteNodeTest: open62541: Create Opaque Node Id::test_nodeTest()
      4: PASS   : opcua::AbsoluteNodeTest: open62541: Create Integer Node Id::initTestCase()
      4: PASS   : opcua::AbsoluteNodeTest: open62541: Create Integer Node Id::test_nodeTest()
      4: PASS   : opcua::AbsoluteNodeTest: open62541: Write Value Node with Multiple Listeners::initTestCase()
      4: PASS   : opcua::AbsoluteNodeTest: open62541: Emitting signals on node changes::initTestCase()
      4: PASS   : opcua::AbsoluteNodeTest: open62541: Node with namespace in Id::initTestCase()
      4: PASS   : opcua::AbsoluteNodeTest: open62541: Node with namespace in Id::test_nodeTest()
      4: PASS   : opcua::AbsoluteNodeTest: open62541: Standard attributes on variable node::initTestCase()
      4: PASS   : opcua::AbsoluteNodeTest: open62541: Node timestamps::initTestCase()
      4: PASS   : opcua::AbsoluteNodeTest: open62541: Node timestamps::test_nodeTest()
      4: PASS   : opcua::AbsoluteNodeTest: open62541: Node timestamps::cleanupTestCase()
      4: PASS   : opcua::AbsoluteNodeTest: open62541: Assign int value to double node::initTestCase()
      4: FAIL!  : opcua::AbsoluteNodeTest: open62541: Assign int value to double node::test_nodeTest() Compared values are not the same
      4:    Actual   (): 2
      4:    Expected (): 1
      4:    Loc: [/Users/qt/work/qt/qtopcua/tests/auto/declarative/AbsoluteNodeTest.qml(440)]
      4: PASS   : opcua::AbsoluteNodeTest: open62541: Assign int value to double node::cleanupTestCase()
      4: PASS   : opcua::UnknownTestFunc()
      [...]
      

      This subtest prints FAIL then PASS. At the end execution, qt-testrunner reports that exit code was 1 (correctly) but the XML did not contain any failure:

      4: qt-testrunner.py     INFO: Test process exited with code: 1
      4: qt-testrunner.py     INFO: Parsed XML file /Users/qt/work/testresults/tst_opcua-1651870032738.xml in 0.003 seconds
      4: qt-testrunner.py     INFO: Found 36 passes and 0 failures
      4: qt-testrunner.py  WARNING: No failures listed in the XML test log! Did the test CRASH right after all its testcases PASSed?
      4: qt-testrunner.py     INFO: Will re-run the full test executable
      

      Attachments

        Issue Links

          No reviews matched the request. Check your Options in the drop-down menu of this sections header.

          Activity

            People

              macadder Jason McDonald
              jimis Dimitrios Apostolou
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

                Created:
                Updated:

                Gerrit Reviews

                  There are no open Gerrit changes