Making the XWiki build reproducible

Development
1

I propose to make the XWiki build reproducible, thereby making it possible to recompute bit-by-bit identical binaries out of XWiki sources. See https://reproducible-builds.org/ for some background information. This has several benefits:

  1. It allows verifying that XWiki releases actually correspond to the source code, and, e.g., no malicious code has been introduced during the release process.
  2. It should improve caching as it will be much easier to verify that the inputs to the build are the same. I don’t know if this could help Develocity, but I imagine it should help.
  3. We could even save some disk space in repositories as we could detect when artifacts are 100% identical and not store them twice. This will mainly concern snapshots, though.

I did some experiments today, and it seems that for xwiki-commons and xwiki-rendering, it should be enough to set the project.build.outputTimestamp property as explained in the guide for reproducible builds of Maven. We would need to verify this by re-building on an independent system, but at least locally, I got completely identical builds with this property. I suggest setting this property to a fixed date and to let the Maven release plugin update it during releases. To me, this seems enough, I don’t think we should use something like the Git commit time as it would negate the caching benefits. We should probably also configure <notimestamp>true</notimestamp> for the JavaDoc plugin.

For xwiki-platform, the situation is a bit different, unfortunately. It appears that the XAR plugin doesn’t create reproducible archives. The good news is, however, that it should be relatively straightforward to fix, we just need to take project.build.outputTimestamp into account to set the modification time of the files accordingly, and we need to make sure that the order of the files in the archive is stable. I don’t know if this is the only issue, though, we would need to see if everything else is okay after fixing the XAR plugin.

Do you agree that we should work on making the XWiki build reproducible and set these properties as proposed?

1 Like
2

+1 in general

Ideally, we would need a way to make sure we keep having a reproducible build, as otherwise I’m afraid we’ll break this quickly (once we fixed the current issues).

I doubt it. For example, I expect the XIP package to have a similar problem (but it should be similarly easy to fix).

1 Like
3

+1

Thanks,
Marius

1 Like
4

+1 since this is already our goal (it’s just that we were not aware - at least I wasn’t aware - of initiatives like https://reproducible-builds.org/ that provides a methodology and tools for it).

As Thomas mentioned, we should have some special job/build on the CI running mvn artifact:check-buildplan from time to time (like once every month) to verify we’re still obeying the rules.

Thanks, this looks great.

1 Like
5

For xwiki-commons and xwiki-rendering, we can probably add them to GitHub - jvm-repo-rebuild/reproducible-central: Reproducible Central: rebuild instructions for artifacts published to (Maven) Central Repository to get an independent reproducibility status. For the others, the problem is that by rebuilding them on our own CI, we only catch issues with timestamps and nothing else as the build environment will be pretty much identical I fear. But still, this might be better than nothing.

1 Like
6

+1
thanks

1 Like
7

+1
Thank you.