Automate the contributors list during the release

Development
1

Hello all,

Today is automation fixing day and in this context I’d like to work on the automation of the list of contributors in the release note.

This step is closely followed by other manual tasks such as adding the list of translated languages, or setting the release date. They are not in the direct scope of this proposal, but I’d like to aim for a general approach that would help to automate other aspects more easily.

Currently, as you can see in the release plan guide, this step is done by running the ~/xwiki-dev-tools/xwiki-release-scripts/list_contributors.sh script on the ~/releases/xwiki-trunks directory. This involves resolving two tags, the previous release tag (i.e., the value of the xwiki.compatibility.previous.version property in xwiki-commons pom) and the current release value.
Then, manually clean up the list (e.g., for bots or user displayed with several names), and finally edit the release page.

The goal of this proposal is to automate the resolution of the tags, the resolution of the list of contributors, and it’s insertion in the release notes. But, not the manual clean up that would still be required before checking the Ensure that the Release Notes are complete and nice-looking for version 18.5.0 step.

Option A - Extend the existing release script and push content to xwiki.org

Improve xwiki-dev-tools to automate:

  • The previous tag resolution (by reading xwiki.compatibility.previous.version in the root pom of xwiki-commons). To be confirmed that this is always the right value, of it we need a prompt in the release script to ask for an override if needed.
  • Optionally, have a mechanism to have list of known bots, and automatically clean them up (e.g., renovate).
  • Performing a REST request to add a newly defined XObject (e.g., ReleaseNotes.ContributorsList, with a single contributor text field) to the release note.

Update the release notes template to display the list of contributors only if a ReleaseNotes.ContributorsList XObject is found, and a warning otherwise.
Optionally, improve the template to allow editing the list of contributors in-place when manually fixing duplicates and bots.

Pros:

  • Can be integrated in the existing release scripts, either as a much more automated manual step or fully integrated in the build script

Cons:

  • Requires to give write access to the wiki on behalf of the release manager by asking for a token. This requires some configuration change in the authentication section of xwiki.org administration.

Option B - Improve the release note template and pull content from the tag

The idea is to inspire from what’s done for backwardCompatibilityReport134.
When the macro is executed, it checks for the presence of a given XObject on the page (ReleaseNotes.BackwardCompatibility). If absent, computes and add the XObject. Then, renders based on the content of the XObject.

For the backward compatibility report, the results can be resolved by a single GET query on the release tag on each repository, for a total of 3 query.
In the case of the contributors we need to fetch the full list of commits since the previous release. Even if paginated this can sum up to a significant amount of GET requests. This can be an acceptable slowness as this needs to be computed only once. This can also be mitigated by providing a token, since iirc authenticated users can query larger page size and less throttled.

The main con of this approach is that we’d need to re-implement the logic of the bash script we currently use in a language that can be executed on the wiki (e.g., Groovy or Java).

Option C - a github action pushed content to xwiki.org

Mentioning it for exhaustiveness. A GitHub action is trigger when a new release if added on xwiki-platform.
This is close to Option A, and the existing bash script can probably be reused.

The main con is GitHub vendor lock in, and I fail to see real benefits in comparison to Option A.

Conclusion

I’m +1 for option A as I believe is the more promising to move toward Continuous Delivery (CD).
By carefully splitting the work in small tasks, a lot of the work done to automate the listing of contributions can be reused for the list of translated languages.

WDYT? Thanks

2

+1 too for option A.

Note that we discussed with @mleduc about a strategy for also automating the list of translated language to use more or less the same kind of mechanism. The idea would be to use Weblate REST API to gather info about translated language and push that info in an xobject of the RN (or directly in the content).

And same credentials could be used for both list of contribs and list of translated languages, and we could probably reuse parts of the code too.

3

Note that Option A means improve the Release Notes app at GitHub - xwiki-contrib/application-releasenotes: Application to manager release notes · GitHub

We already have the need to move more RN data to structured data (like backward compatibility items, translations, etc), so that’s going in the right direction. The idea for adding more structured data is to be able to answer user questions such as: 'between v17.0 and v18.5, show me the full list of backward compatible items that I need to be careful about". Re contributors, that would allow answering questions such as: “who are the contributors for versions from 17.0.0 to 18.5.0”.

See See Loading... for existing jiras.

I’m not fully sure about that. For me, CD is about having all the release process inside our jenkins pipeline. Now I guess the idea for the future could be to move the release script inside a jenkins pipeline.

Option B removes the need to have the code in the pipeline but it’s probably a bit more fragile. BTW if we think it’s not the right approach, we should consider refactoring backwardCompatibilityReport134 into the CD/release script too and also introduce an XObject for it.

Re introducing new XObjects in the RN app, I vaguely remember having done a refactor to introduce a generic “type” for entries from the RN. That should probably be reused.

So indeed, I’m also leaning towards option A but maybe with a separate script in charge of automating the creation of the Release Notes page. The RN app provides the structure, then it can used manually to create a RN or filled automatically by a script.

Thanks

4

Yes that’s what I have in mind

+1 as a future improvement

That’s good to know, thanks!

My goal is to have many small scripts doing small tasks well.
Then we can discuss how to orchstrate them, as individual scripts to call manually, or as larger automated steps (or of course a mix of both).

5

+1 for option A

6

Thanks for your feedback.
Status update: I have opened RN-70: Move the list of contributors to a dedicated XObject by manuelleduc · Pull Request #5 · xwiki-contrib/application-releasenotes · GitHub for the improvement of the release not app.