The testrunner was in most cases sufficient for our cases. Sometimes a test behaved flaky, and we hat to repeat the COIN procedure.
End of November 2017, we suddenly observed strange behavior of the RHEL 6.6 tests: It was almost impossible to check something in. The problem was circumvented by analysis of some specific errors and by black-listing others.
This change shall improve our situation:
Tests will be run multiple times in the COIN system, and failing once will not be reported as an error, but the behavior of such failing tests will be monitored.
The correctness of this change is crucial, since it modifies the whole test system.