Create detailed tests for quick3d/robotarm

Pre-requisites

• Have Squish 7.1.0 for Qt6.4 using MinGW installed (ci-files01-hki.intra.qt.io/input/squish/coin/64x/squish-7.1.0-qt64x-win64-mingw_gcc112_posix_seh.exe)
  • Note, no spaces in the installationpath! Squish installer adds spaces by default, remove those!
  • You might need to have several Squish installations later on, so I recommend to install to e.g. [home]/Squish/7.1/ -dir
• have tqtc-qt-rta repository cloned for example [home]/git -dir
• Launch Squish, if there is no directories listed in the Global Scripts (on the right), open command prompt and run

• [python3] tqtc-qt-rta\squish_config\use_system_python.py [path_to_your_squish_installation] 

• Restart Squish, now there should be multiple dirs in the Global Scripts

Prepare Squish env

• locate env_variables.py file in the Global Scripts (suites/shared/)
• hard code config/versions which are read from environment variables when running on Jenkins
  • get_job_name(), add line:

    return "RTA_installer_exampleDetail_highlighted/cfg=win-MinGW11.2.0-Windows10-22H2-x64"

• • get_version(), add line:

    return "6.5.0"

• • get_qt_version(), add line:

    return "6.5.0"

• • get_vm_branch()(), add line:

    return "6.5"

• Open test suite tqtc-qt-rta\suites\packaging\suite_highlighted_apps and run the first tests tst_install_qt. This will create the RTA dir to your home directory, download Qt online installer in there and run the installer so that Qt 6.5.0 MinGW pre-build binaries will be installed to RTA\Qt_6.5.0 for the test
• create new test case to the suite (Script Test Case) and name it tst_cli_quick3d_robotarm and use drag-n-drop to place it in alphabetical order among other test cases
• all test cases do same initializations in the beginning, so open tst_cli_bluetooth_heartrate_game test case, copy all and paste to the new test case, fix the example name and location and remove all after line "test.log("Application %s is visible" % test.example_name)" (those are heartrate example specific verifications), also comment the import line which imports the object map: "import heartrate_game_names as names", that will be uncommented and edited later
• run the test case, it will compile the example and add to Squish AUT list, you can verify that by checking the Squish AUT list (see Squish documentation)

Do planning on the verification

• Now that the example has been compiled, next step is to record the actual test steps, it's a good practice to do some initial planning outside of the actual test suite
• create new test suite to a location of your choice, it will not be pushed to gerrit, so it can be any location on your computer, make sure you select Qt as wrapper and python as script language
• add new test case, name how ever you like, it's just temporary practicing test
• run the AUT, no need to do any recordings yet, and get familiar what it does and do some planning what could be tested in there, for example
  • changing slider position changes the value in the text edit
  • changing dark mode and claw switches
  • note, animations can not be verified with Squish, only the final position/frame can be verified with screenshot, and then in test code, test execution needs to be snoozed (shared.snooze(add_seconds_here) for time it takes the animation to run to the end
• practice recording, you can make multiple new test cases, since this is only practice test suite
• also run the tests to see that recordings work, check that there is passes/fails in test, and in fail cases see how the logging prints are, in real life when the test fails, it's important what is printed in the log, clear messages stating what went wrong will ease the fixing the test or creating a bug report if it's a real bug in Qt code

Implement the test

• when it's clear to you what you will record, change back to the actual test suite (suite_highlighted_apps/tst_cli_quick3d_robotarm), add break point to the last line of code to stop execution there. Run the test to the breakpoint and add the test verifications by clicking the 'Record snippet' button in SquishIDE which will switch Squish to the recording mode, do recordings and click 'Stop recording' and 'Terminate' (red square) and now you should have recorded code in the test case
• re-run the test without stopping to the breakpoint and make sure it does the verifications you intended
• code that Squish generated can be changed manually if needed
• some practical tips:
  • test execution needs to be slowed down in many cases (Squish is sometimes too fast), so just add shared.snooze(seconds) where ever you need (you can see other test cases for reference)
  • typical Squish mouseClick is recorded as:
    • mouseClick(squish.waitForObjectItem(names_qttools.Assistant_content, item), 35, 8, Qt.NoModifier, Qt.LeftButton)
    • in some cases is better to delete the coordinates and mouse definitions, (we use the defaults 99% of the cases) so it would like as:
    • mouseClick(squish.waitForObjectItem(names_qttools.Assistant_content, item))
    • this will help when running same test in other platforms (mac/linux) because sometimes the objects are different size and removing the coordinates will make Squish to calculate the center of the object automatically and click in that position instead of hard-coded position (35,8 in this case)
  • when doing screenshot verifications, sometimes objects look different in win/linux/macOS so it might be useful to do partial screenshot verifications instead of the full view (see Squish documentation how to do that), also the threshold needs usually to be set below the default 100%, for example 90-95%. And sometimes even that wont help and separate screenshot recordings needs to do be done for each platform and then just create if-else branch to check which platform is used (there is helper functions for that, e.g. is_feature.is_win(), use search inside Squish to see how they are used)

Push to git

• Squish will create names.py file for each test suite and define used objects there (object map), in this test suite it is decided to have separate names.py files for all test cases, so when recordings are done and complete, locate the names.py file, create new file in left bottom corner of SquishIDE: Test Suite Resources -> Scripts -> New Test Recource. Name new file as robotarm_names.py, cut the python code from names.py and paste to robotarm_names.py, also add the header to it
• uncomment the import line from the test case, which was commented earlier and fix the name
• re-run test once to verify that correct object map is used, the names.py should not have new changes
• when the test is as good as it can get now, make new commit to git (see https://intranet.qt.io/display/QTRD/RTA%2C+where+to+start 'adding changes to git') add reviewers to the change, e.g. me or Milla Pohjanheimo
• when review is done, run the change in Jenkins with the ref before merging (contact Johanna or Milla for details)