It would be nice to have benchmark alerts. It is possible to check the dashboards but this takes time. The alert performs tests on the benchmark results to find improvements and regressions and notifies the relevant persons automatically. That's the idea.

It is necessary for the test to be somewhat reliable. It does not have to be perfect. We can probably survive with a small number of false positives and false negatives. This is feasible.

It might be a good idea to plot benchmark results in distributions to figure out how the test should work.

See QTQAINFRA-6873.

This ticket is about finding out what kind of test can be used.