Better test handling for our CI

[vmassol]

@vmassol

UC1: List the slowest executing tests (so that we can work on the slowest and more generally monitor the speed of our test, by graphing their speed over time)

https://ge.xwiki.org/scans/tests?search.relativeStartTime=P90D&search.timeZoneId=Europe%2FParis&tests.sortField=MEAN_DURATION

@vmassol

UC2: List the most flickering tests (so that we can focus on them)

https://ge.xwiki.org/scans/tests?search.relativeStartTime=P90D&search.timeZoneId=Europe%2FParis&tests.sortField=FLAKY

@vmassol

UC3: List known flickering tests (those registered in JIRA and open)

https://jira.xwiki.org/issues/?jql=%20labels%20%3D%20flickering%20and%20resolution%20%3DUnresolved%20%20

@vmassol

UC4: List discovered flickering tests (3 changes of state in the past 20 executions for example)

https://ge.xwiki.org/scans/tests?search.relativeStartTime=P90D&search.timeZoneId=Europe%2FParis&tests.sortField=FLAKY

@vmassol

UC5: List the slowest unit tests (to be sure we don’t have unit tests that take too long)

Use GE and find conditions to identify functional tests or unit tests (e.g. not:CI stage=Build for IT* - not enough but a start).

@vmassol

UC6: Are there flickering tests which haven’t passed at least once during the current release timeframe (to help decide if we can release)?

Can be done using GE, using the timeframe of the release and sorted by most flickering, if you get one with 100% then you have it.

https://ge.xwiki.org/scans/tests?search.buildToolType=maven&search.timeZoneId=Europe%2FParis&tests.sortField=FLAKY

Conclusion:

I don’t think anymore that we need a new tool since ge.xwiki.org (Develocity) seems to globally do the job for us.