Embedded Regression testing enhancements

Detection of Regression Items

The goal here is to enhance embedded regression testing. The intended approach is to have continuous testing of Qt dev branch with the following apps:

• QML Bench
• Performance evaluation app

Results are available in Grafana:

​

Results are checked regularly in every second embedded daily meeting by the embedded performance team (initially Pasi, Samuli, Juho, Jari) at least initially and Jira ticket is auto-generated.

Separate test sets in nightly mode similar to what desktop testing is doing (Eskil and Daniel Smith) to easily see what has changed in Qt:

1. Choose one released Boot to Qt version with latest Qt version from dev branch.
2. Change the boot to qt version once a year to a newer version

Results may vary without modifications, hence we need some time to learn what is a significant finding. If there are findings, we run the bisected commits only rerun  and not the whole test suite.

Regression escalation

Suspicious results sent to the patch owner automatically to a performance Teams channel or before that manually. Follow-up to be separately discussed inside R&D.

​