It doesn’t provide any way to gather junit test results OOB (See community · Discussions · GitHub) so we need to find an action for that or create one. I’ve found 2 that could work but they both have problems:
Scope: https://scope.dev/. Nice thing is that you get test results on the go + flaky test identification, test perf, etc. However 2 problems that I need to report (not done yet):
What’s the cost?
It got stuck when I tested it on commons and I had to kill the workflow after 1 hour.
Thoughts
It’s very interesting and very well integrated with GH
Since we have a docker build image for xwiki, we don’t need that much from the CI, which is good since that allows us to depend less on it.
The usage of remote servers for gathering tests and their history is both good and bad. Good because this allows us to go further with test analysis and handling in general (and be independent of the CI) and Bad because this ties us to another service which may not be available (fragility).
Next steps:
be able to gather test data and failing tests and see why they failed.
find out if we can prevent a build at each commit and instead stack them as we do in our ci (now it’s less an issue since we don’t operate the agents).
We can have it work even though our repos are on GitHub (it sets up a mirorring clone)
Seems a bit more advanced than GitHub Actions (GA)
We can use gitlab.com which is very nice for open source projects. However, note that the CPU/Memory for the agents are lower than GA and it could be not enough for our platform builds (GitLab.com settings | GitLab - 3.75GB of RAM). Also we get 10 parallel pipelines (GitLab.com settings | GitLab).
Same as GA it’s possible to setup our own runners (even for gitlab.com).
In order to not loose it, here’s the current .gitlab-ci.yml script I have:
variables:
# This will suppress any download for dependencies and plugins or upload messages which would clutter the console log.
# `showDateTime` will show the passed time in milliseconds. You need to specify `--batch-mode` to make this work.
MAVEN_OPTS: "-Xmx2048m -Xms512m -Dhttps.protocols=TLSv1.2 -Dmaven.repo.local=$CI_PROJECT_DIR/.m2/repository -Dorg.slf4j.simpleLogger.log.org.apache.maven.cli.transfer.Slf4jMavenTransferListener=WARN -Dorg.slf4j.simpleLogger.showDateTime=true -Djava.awt.headless=true"
# As of Maven 3.3.0 instead of this you may define these options in `.mvn/maven.config` so the same config is used
# when running from the command line.
# `installAtEnd` and `deployAtEnd` are only effective with recent version of the corresponding plugins.
MAVEN_CLI_OPTS: "--batch-mode --errors --fail-at-end --show-version -DinstallAtEnd=true -DdeployAtEnd=true -Dmaven.test.failure.ignore"
image: xwiki/build:latest
# Cache downloaded dependencies and build results across jobs
# Note: We don't cache the target directory, see
# https://blog.deniger.net/post/gitlab-maven-optimize-build/
cache:
paths:
- .m2/
# Note that Gitlab only caches /build and /cache ATM and since the XWiki build image
# puts the m2 repo in /root/.m2/repository we need to symlink it below.
# Current directory set by gitlab is /build/<github org name>/<github repo name>
# Variable name is $CI_PROJECT_DIR
before_script:
- ln -fs /root/.m2 $CI_PROJECT_DIR/.m2
build:
stage: build
script:
# Apparently gitlab runner doesn't run our bashrc, I think it's because it
# overrides the root user's home directory to be /builds, so we need to force
# execute it to have the mvn executable in the PATH.
- . /root/.bashrc
# Remove the xwiki dependencies from the cache repo so that we build them or
# download them and have reproducible builds
- rm -Rf $CI_PROJECT_DIR/.m2/repository/org/xwiki
- rm -Rf $CI_PROJECT_DIR/.m2/repository/com/xpn
# We run the clean goal because we cache target/ so that the test job won't have
# to rebuild everything and thus will be faster. However this seems to carry over
# across pipeline executions.
- mvn $MAVEN_CLI_OPTS clean install -DskipTests -Plegacy,snapshot
artifacts:
# When not specifyin expire_in the default is used whih is 30 days or forever, see
# https://docs.gitlab.com/ee/ci/yaml/#artifactsexpire_in
# Note: see https://blog.deniger.net/post/gitlab-maven-optimize-build/ for the syntax used.
paths:
- "*/target"
test:
stage: test
script:
- . /root/.bashrc
# Setup a display for functional tests requiring one (non-docker tests)
- vncserver :1 -geometry 1280x960 -localhost -nolisten tcp
- export DISPLAY=:1
# Make sure we don't recompile sources, see:
# https://blog.deniger.net/post/gitlab-maven-optimize-build/
- find . -name "*.class" -exec touch {} \+
- mvn $MAVEN_CLI_OPTS test -Plegacy,integration-tests,docker,snapshot
artifacts:
reports:
junit:
- target/surefire-reports/TEST-*.xml
- target/failsafe-reports/TEST-*.xml
I’ve also just read the doc from circleci.com and it’s really nice. It’s the best doc I’ve seen and I really like the features. The concepts are very clear and nice and they’ve paid a lot of attention to easily making the parts reusable.
