Better test handling for our CI

[surli]

@vmassol

Does it mean you’re ok with spending time to develop handling of our tests by ourselves (i.e. creating our database of test results in Elastic Search) instead of relying on Jenkins?

AFAIU Jenkins only allows to keep all info about a build or no info. And our build are very expensive in terms of logs. And your proposal is to only keep some logs/info in an ES instance so that we can request it to get information on the long term.
So yeah it looks like something we need. Now I haven’t performed any search on how to do it yet, so maybe there’s already existing stuff on Jenkins that we can reuse or not, I trust you on that part if you say that we need to do it ourselves.

@vmassol

We can’t easily have the time for containers to be created but we can probably have the time it takes to setup the test (the fixture time).

Still thinking on the long term: I really think that we’d need the info about the time taken for a container to be fully initialized. Maybe we need to request some APIs on TC for computing it properly, but I really think it would be helpful.

@vmassol

Regarding the whole scenario, why do you think it would be useful? We can’t compare scenario times between each others since some scenarios are small and other are large.

No but you could compare the evolution of a scenario when you added one test for example. And discover that a specific scenario is not long because of its tests but because of some repeated fixture. Things like that.

@vmassol

@surli

Is it something we want for our local builds? I’m tempted to say not really, so wondering if we shouldn’t have this log push at Jenkins code level?

I thought about it and my answer so far is: no. But it would be easy to do (easier than filtering only the CI builds). The reason I think it’s “no”, is because you need an internet connection (which is not always the case locally) and it would slow down the local execution a little bit and I’d prefer to have them as fast as possible.

I’m not sure you answered my question there. I’m a bit worried that we put stuff in our java code that is related to a specific CI need. And that should be in CI level. For example, I was about to say that we should be extra-careful to not make a build (or lots of builds) fail or error because of the introduction of this ES stuff. And adding it at Java level is a very good example of something that could break our build or make them fragile for something which is really not related with building those elements.
Hence my question, about having the mechanism to push the info in the CI only, and not in the build.

For example, we could have some Java code to compute data when executing tests and create JSON files, and on the CI an addon in charge of sending those info on the ES instance.