We need to have coverage for removed tests. There have been a couple of instances now where a rather large set of tests have accidentally stopped being run. When the issue causing these not to be run is fixed, the tests start failing in CI, blocking integrations.

We need to have some kind of report or notification for this, so that we can check if it's actually intended or not, as it's quite easy not to notice it happening.