When to drop the previous LTS branches when a new Cycle LTS is released?

Development
,
1

Hi devs,

We don’t have a very clear written rule ATM. All we have is what is on https://dev.xwiki.org/xwiki/bin/view/Community/VersioningAndReleasePractices/

There’s a section about What to do when a N.4.2 or N.10.2 version is released, but it doesn’t state clearly what to with regards to the previous LTS(s) branches.

All it says is related to this topic is:

  • (Cycle LTS) Check if there are important issues to release in the previous LTS branch and schedule a last release if need be.

IMO, the spirit of Introduce the concept of "Intermediate LTS" and what we had before was the following:

  • When N.10.2 is released, drop the intermediate LTS branch and the previous cycle LTS branch

However, we need to account that we also have a Manual Testing Strategy (which needs to be updated since we don’t have “recommended” extensions anymore, but that’s another story, cc @ilie.andriuta ), and that it takes time for the Manual QA team (currently @ilie.andriuta and @atarata) to perform tests for a LTS and that it would be good to have the Manual QA validation before we can call the LTS a LTS.

Now, I think the Manual QA strategy in place already covers this, since it says:

Some general rules:

  • The LTS version has higher testing priority than other versions;

And

According to release strategy on every end of cycle, a Long Term Support version is released.

From testing point of view this is a special stable release and should be tested more in-depth. The time-box indicated for this releases is 3 weeks/person.

So I think we need a few clarifications and agreements to be good:

  1. The Manual QA for LTS must start as soon as N.4.0 and N.10.0 are released, i.e. if tests are underway on some other versions of XWiki then pause them until all the priority testing is finished (or if there’s only 1 or 2 more days needed to finish them, then finish them)
  2. Have the following testing priorities:
    • First current Cycle LTS branch release (i.e. N.10.0) > current Cycle LTS branch bugfixes (i.e. N.10.x, x > 0) > previous Cycle LTS bugfixes (e.g. N-10.10.x, x > 0) > First intermediary LTS branch release (i.e. N.4.0) > intermediary LTS branch bufixes (i.e. N.4.x, x > 0) > Stable bugfix releases (i.e. N.x.y, x != 4, x != 10 and y > 0) > Stable releases (i.e. N.x.0, x != 4 and x != 10) > Extension releases
    • The idea is to validate the new Cycle LTS ASAP, then the previous Cycle LTS bugfixes, then the intermediate LTS, then the stable releases.
  3. Add a note on https://dev.xwiki.org/xwiki/bin/view/Community/VersioningAndReleasePractices/ that if Manual QA is late to validate the new Cycle LTS, then we won’t drop the previous LTS(s) branches until Manual QA has finished, which means we won’t drop the N-10.10.x and N.4.x branches and will continue to merge changes in them. And when Manual QA has validated, drop the previous LTS(s) branches.
  4. We should also probably note that N10.2 (and N.4.2) become LTSes only if there are no known blocker issues for them (discovered by Manual QA testing, see point 3)
  5. If Manual QA finds blocker issues during LTS validation, this will require a new LTS bugfix release, and the previous LTS(s) branches continue to be supported during that time.

If we do 1) then there’s 3 weeks of testing before the LTS can be validated. Since we have 4 weeks, that looks ok. However:

  1. It’s a period of the year (December) when there are holidays
  2. We will release N.10.1 and N.10.2 during the N.10.0 testing, and in theory each one adds 1 week/person of test.

Note that if N.10.0 is validated, then this means it’s of LTS quality and N.10.2 could be labelled as LTS (same idea for N.4.0 and N.4.2) since the likelihood of a regression is low and in any case that can happen later on, in N.10.7 for ex, so it’s not a real problem. The problem is when N.10.0 (and N.4.0) is not validated (it has blocker issues). Either it’s fixed in N.10.1 or N.10.2 already (and Manual QA will validate this, but 2 weeks later), or it’s not fixed and we will need to release N.10.3 and add one more week to test it.

All in all, depending on how many Manual QA testers we have and the quality of the releases, this means that the validation could slip after the end of the year for N.10.x, up to end of January. This is covered by point 3) and 4) above.

WDYT about the points 1) to 5) above?

Thanks

1 Like
2

The general idea of bounding the removal of branches to QA testing makes sense, but I’m wondering if we need to do it for both previous LTS branches: I mean I’m wondering if when N.10.2 is released, we shouldn’t perform the strategy you propose for N.4.x but perform a last release and drop N-1.10.x asap.

I don’t think it worth to keep supporting 2 LTS on top of the new LTS to be validated.

Otherwise your proposal sounds good to me but we probably need opinion from QA.

3

+1 for the dropping strategy.

I’m not sure, I wonder if some users will only jump from LTS to LTS (i.e., will skip the intermediate LTSs). In this case, they’ll need bug fixes on LTS for the whole cycle.

4

Yes I’m not proposing to drop this, just saying that in theory the new LTS should be validated quickly, technically the cycle is over already: it’s adding maintenance time on us to keep supporting the N-1 LTS while we are already working on validating (and probably fixing if it’s not yet validated) the new one. Following the strategy for N-1.4.x seems quite enough.

5

Same here, especially since we already switched the lts Debian repository and docker tag to 17.10.x. If we were to keep maintaining the 16.10.x branch until 17.10.x is validated, we should revert that switch.

6

What I proposed basically means that we wouldn’t label N.10.2 (or N.4.2) as LTSes automatically. They’d be labelled as LTS when N.10.0 (or N.4.0) have been validated by Manual QA. If that happens before N.10.2 (or N.4.2) are released then it’ll work as it works now but if Manual QA is late or if Manual QA found some blockers issues, then these blockers need to be fixed before we label them as LTSes.

Based on this, it feels normal to not close the current cycle LTS (N-1.10.x) and the intermediary LTS (N-1.4.x) before N.10.x is marked LTS.

Note that, above, I’ve mentioned validating N.10.0 (or N.4.0) with Manual QA before labelling the branch as LTS, and not N.10.2 (or N.4.2). The rationale is 1) validating N.10.0 (N.4.0) is what takes time since it’s a full Manual QA test (3 weeks) vs 1w testing for bugfix versions, and 2) if N.10.0 (N.4.0) is validated, there’s little risk that N.10.1/N.10.2 (N.4.1/N.4.2) make it worse and if that were the case, we would know it quickly and can quickly release a new bugfix version, and 3) to not take too much time, as waiting to test N.10.1/N.10.2 (N.4.1/N.4.2) would add at least 1w more. But if we wanted to be strict, we could wait for the Mnual QA of N.10.1/N.10.2 (N.4.1/N.4.2).

7

If QA is just late and there are no blockers then we’d have to add the LTS label to the already released N.10.2. Is the RM of N.10.2 responsible for this? i.e. should the QA notify us (the committers) on xwiki after the LTS is validated so that the RM can add the LTS label?

Thanks,
Marius

8

I have now discussed with @ilie.andriuta and @atarata and I’d like to propose an amendment to the proposal above as I don’t really like the fact that we wouldn’t be able to label a release as LTS before ManualQA has validated it and found no blockers in it. I feel this is too long and will take us towards mid or end of January before we can label the N.10.x branch as LTS.

This would be a pity since the reason we moved to doing 2 bugfix releases in December was to be able to have a LTS by end of the year.

So the proposed amendment is:

  • We keep our current process as written on https://dev.xwiki.org/xwiki/bin/view/Community/VersioningAndReleasePractices/ and mark the LTS when N.4.2 and N.10.2 are released
  • Manual QA does the testing as indicated above, in the priority order mentioned above
  • Once Manual QA has validated a version (and no blocker has been found in it), we indicate it in the Release Notes of that version + on the download page we add some QA badge next to the version (with some hint explaining that the version has been validated by Manual QA), or simply some text after the version content list.
  • We don’t close the previous Cycle LTS branche + last intermediary LTS and continue to backport important bug fixes on them, until Manual QA has approved the new Cycle LTS. At that time, we do one last release of the 2 branches and drop support for them.

WDYT?

Thx

9

+1 on my side.

Thanks,
Marius

10

+1

11

+1

I’m not super happy about that: I still think we should only keep last intermediary LTS and we should drop the cycle LTS. You said yourself previously that the reason to keep the cycle LTS was that the proposal was about not labelling new LTS, but you amended it here, to label the new LTS so that argument doesn’t stand anymore.

Also the intermediary LTS is supposed to have been tested and validated by QA so I don’t see the need of keeping the previous cycle LTS.

Finally, keeping the previous cycle LTS implies to maintain it, and also to have to support one more branch on the CI. Just as the reminder as of today it means supporting 5 branches:

  • stable-16.10.x
  • stable-17.4.x
  • stable-17.10.x
  • stable-18.0.x
  • master

That’s really a lot to support IMO, and so I’d really prefer to only keep supporting intermediary LTS. So I’m -0 to support both.