Not long I implemented log validation for functional tests, i.e. we test that there are no WARN or ERROR messages in the logs and if there are we fail the test. Then we proceeded to add excludes for all the cases where we think we have false positives.
I think the idea was good in theory but in practice it’s taking too much of our time and leading to lots of test instability in our CI and I don’t believe this is going to be fixed anytime soon.
Since the ratio of cost of stabilizing the tests vs problems found is very low, I propose that we put this idea on hold for now and remove this check for functional tests (we should keep it for unit tests since instability there is very low or none).
PS: It’s now been several months since we implemented this and to this date there are still lots of tests failing in our CI caused by this, showing it’s too hard to stabilize and the strategy is not working. I think we learnt things in the process and it’s good to try new things but it’s also good to rollback when we find that a strategy is not working as well as we thought it could.