the current status of our CI since 6 months or more is to mix up in the same jobs different kinds of builds:
- standard builds which execute all tests and the docker test if the agent support docker (now all our agents support docker)
- docker-only builds that only execute docker tests on various configurations (we have even different kind of docker builds for the different set of config we want to test, but not the subject today)
If I’m right there was two main reasons for mixing different builds on the same job:
- to be able to see only one build failing
- to manage all branches with only one Jenkins file
Source: https://markmail.org/message/64mtprh5hshg2rtv and discuss on the chat.
Now after several months of using this setup, I don’t think we should continue with it, for two main reasons:
- mixing different kind of builds might lead to false positive: if a docker build is passing, then it will be “hidden” in the CI, even if the previous standard build was failing
- the history might be harder to follow: we should keep 30 builds, but in fact we are keeping 30 mixed builds of docker and standard builds. I don’t recall properly but I also think we had some issue at a point when following the history of a failing test because of mixed builds.
Another reason is that now we have docker agents we are executing the docker test by default in our standard build, so we won’t miss a failure on those tests even if we separate the docker-only builds. So IMO the first reason why we performed the separation is not good anymore.
Now the only problem that remains IMO is to separate the builds without having much work to do for the branches: AFAIR we cannot have two different maintained jobs with the same Jenkinsfile.
I don’t know Jenkins well enough to make good proposal there I think, but I guess we have some options, for example:
- working only with standard jenkins configurations (not related to a Jenkinsfile) that the RM would copy / edit to put the right version after a release
- put a dedicated Jenkinsfile at another place and improve our release script to perform the appropriate git actions on this place too
It would be indeed a bit more work for the RM than the automated setup we have, but I think it might be better on the long term to not mix up those builds. At least to not having each time to open the history to check the real status of the job. WDYT?