When to drop the previous LTS branches when a new Cycle LTS is released?

Development
,
1

Hi devs,

We don’t have a very clear written rule ATM. All we have is what is on https://dev.xwiki.org/xwiki/bin/view/Community/VersioningAndReleasePractices/

There’s a section about What to do when a N.4.2 or N.10.2 version is released, but it doesn’t state clearly what to with regards to the previous LTS(s) branches.

All it says is related to this topic is:

  • (Cycle LTS) Check if there are important issues to release in the previous LTS branch and schedule a last release if need be.

IMO, the spirit of Introduce the concept of "Intermediate LTS" and what we had before was the following:

  • When N.10.2 is released, drop the intermediate LTS branch and the previous cycle LTS branch

However, we need to account that we also have a Manual Testing Strategy (which needs to be updated since we don’t have “recommended” extensions anymore, but that’s another story, cc @ilie.andriuta ), and that it takes time for the Manual QA team (currently @ilie.andriuta and @atarata) to perform tests for a LTS and that it would be good to have the Manual QA validation before we can call the LTS a LTS.

Now, I think the Manual QA strategy in place already covers this, since it says:

Some general rules:

  • The LTS version has higher testing priority than other versions;

And

According to release strategy on every end of cycle, a Long Term Support version is released.

From testing point of view this is a special stable release and should be tested more in-depth. The time-box indicated for this releases is 3 weeks/person.

So I think we need a few clarifications and agreements to be good:

  1. The Manual QA for LTS must start as soon as N.4.0 and N.10.0 are released, i.e. if tests are underway on some other versions of XWiki then pause them until all the priority testing is finished (or if there’s only 1 or 2 more days needed to finish them, then finish them)
  2. Have the following testing priorities:
    • First current Cycle LTS branch release (i.e. N.10.0) > current Cycle LTS branch bugfixes (i.e. N.10.x, x > 0) > previous Cycle LTS bugfixes (e.g. N-10.10.x, x > 0) > First intermediary LTS branch release (i.e. N.4.0) > intermediary LTS branch bufixes (i.e. N.4.x, x > 0) > Stable bugfix releases (i.e. N.x.y, x != 4, x != 10 and y > 0) > Stable releases (i.e. N.x.0, x != 4 and x != 10) > Extension releases
    • The idea is to validate the new Cycle LTS ASAP, then the previous Cycle LTS bugfixes, then the intermediate LTS, then the stable releases.
  3. Add a note on https://dev.xwiki.org/xwiki/bin/view/Community/VersioningAndReleasePractices/ that if Manual QA is late to validate the new Cycle LTS, then we won’t drop the previous LTS(s) branches until Manual QA has finished, which means we won’t drop the N-10.10.x and N.4.x branches and will continue to merge changes in them. And when Manual QA has validated, drop the previous LTS(s) branches.
  4. We should also probably note that N10.2 (and N.4.2) become LTSes only if there are no known blocker issues for them (discovered by Manual QA testing, see point 3)
  5. If Manual QA finds blocker issues during LTS validation, this will require a new LTS bugfix release, and the previous LTS(s) branches continue to be supported during that time.

If we do 1) then there’s 3 weeks of testing before the LTS can be validated. Since we have 4 weeks, that looks ok. However:

  1. It’s a period of the year (December) when there are holidays
  2. We will release N.10.1 and N.10.2 during the N.10.0 testing, and in theory each one adds 1 week/person of test.

Note that if N.10.0 is validated, then this means it’s of LTS quality and N.10.2 could be labelled as LTS (same idea for N.4.0 and N.4.2) since the likelihood of a regression is low and in any case that can happen later on, in N.10.7 for ex, so it’s not a real problem. The problem is when N.10.0 (and N.4.0) is not validated (it has blocker issues). Either it’s fixed in N.10.1 or N.10.2 already (and Manual QA will validate this, but 2 weeks later), or it’s not fixed and we will need to release N.10.3 and add one more week to test it.

All in all, depending on how many Manual QA testers we have and the quality of the releases, this means that the validation could slip after the end of the year for N.10.x, up to end of January. This is covered by point 3) and 4) above.

WDYT about the points 1) to 5) above?

Thanks

1 Like
2

The general idea of bounding the removal of branches to QA testing makes sense, but I’m wondering if we need to do it for both previous LTS branches: I mean I’m wondering if when N.10.2 is released, we shouldn’t perform the strategy you propose for N.4.x but perform a last release and drop N-1.10.x asap.

I don’t think it worth to keep supporting 2 LTS on top of the new LTS to be validated.

Otherwise your proposal sounds good to me but we probably need opinion from QA.

3

+1 for the dropping strategy.

I’m not sure, I wonder if some users will only jump from LTS to LTS (i.e., will skip the intermediate LTSs). In this case, they’ll need bug fixes on LTS for the whole cycle.

4

Yes I’m not proposing to drop this, just saying that in theory the new LTS should be validated quickly, technically the cycle is over already: it’s adding maintenance time on us to keep supporting the N-1 LTS while we are already working on validating (and probably fixing if it’s not yet validated) the new one. Following the strategy for N-1.4.x seems quite enough.

5

Same here, especially since we already switched the lts Debian repository and docker tag to 17.10.x. If we were to keep maintaining the 16.10.x branch until 17.10.x is validated, we should revert that switch.

6

What I proposed basically means that we wouldn’t label N.10.2 (or N.4.2) as LTSes automatically. They’d be labelled as LTS when N.10.0 (or N.4.0) have been validated by Manual QA. If that happens before N.10.2 (or N.4.2) are released then it’ll work as it works now but if Manual QA is late or if Manual QA found some blockers issues, then these blockers need to be fixed before we label them as LTSes.

Based on this, it feels normal to not close the current cycle LTS (N-1.10.x) and the intermediary LTS (N-1.4.x) before N.10.x is marked LTS.

Note that, above, I’ve mentioned validating N.10.0 (or N.4.0) with Manual QA before labelling the branch as LTS, and not N.10.2 (or N.4.2). The rationale is 1) validating N.10.0 (N.4.0) is what takes time since it’s a full Manual QA test (3 weeks) vs 1w testing for bugfix versions, and 2) if N.10.0 (N.4.0) is validated, there’s little risk that N.10.1/N.10.2 (N.4.1/N.4.2) make it worse and if that were the case, we would know it quickly and can quickly release a new bugfix version, and 3) to not take too much time, as waiting to test N.10.1/N.10.2 (N.4.1/N.4.2) would add at least 1w more. But if we wanted to be strict, we could wait for the Mnual QA of N.10.1/N.10.2 (N.4.1/N.4.2).

7

If QA is just late and there are no blockers then we’d have to add the LTS label to the already released N.10.2. Is the RM of N.10.2 responsible for this? i.e. should the QA notify us (the committers) on xwiki after the LTS is validated so that the RM can add the LTS label?

Thanks,
Marius

8

I have now discussed with @ilie.andriuta and @atarata and I’d like to propose an amendment to the proposal above as I don’t really like the fact that we wouldn’t be able to label a release as LTS before ManualQA has validated it and found no blockers in it. I feel this is too long and will take us towards mid or end of January before we can label the N.10.x branch as LTS.

This would be a pity since the reason we moved to doing 2 bugfix releases in December was to be able to have a LTS by end of the year.

So the proposed amendment is:

  • We keep our current process as written on https://dev.xwiki.org/xwiki/bin/view/Community/VersioningAndReleasePractices/ and mark the LTS when N.4.2 and N.10.2 are released
  • Manual QA does the testing as indicated above, in the priority order mentioned above
  • Once Manual QA has validated a version (and no blocker has been found in it), we indicate it in the Release Notes of that version + on the download page we add some QA badge next to the version (with some hint explaining that the version has been validated by Manual QA), or simply some text after the version content list.
  • We don’t close the previous Cycle LTS branche + last intermediary LTS and continue to backport important bug fixes on them, until Manual QA has approved the new Cycle LTS. At that time, we do one last release of the 2 branches and drop support for them.

WDYT?

Thx

9

+1 on my side.

Thanks,
Marius

10

+1

11

+1

I’m not super happy about that: I still think we should only keep last intermediary LTS and we should drop the cycle LTS. You said yourself previously that the reason to keep the cycle LTS was that the proposal was about not labelling new LTS, but you amended it here, to label the new LTS so that argument doesn’t stand anymore.

Also the intermediary LTS is supposed to have been tested and validated by QA so I don’t see the need of keeping the previous cycle LTS.

Finally, keeping the previous cycle LTS implies to maintain it, and also to have to support one more branch on the CI. Just as the reminder as of today it means supporting 5 branches:

  • stable-16.10.x
  • stable-17.4.x
  • stable-17.10.x
  • stable-18.0.x
  • master

That’s really a lot to support IMO, and so I’d really prefer to only keep supporting intermediary LTS. So I’m -0 to support both.

12

The reason is that we need Manual QA to prove that the LTS version is effectively of good quality. Some users (like XWiki SAS) will wait until Manual QA has validated the LTS before proposing that version to its clients (for upgrades or new installs).

Yes but again, intermediary LTS is not the same as cycle LTS and a lot of users would wait for a cycle LTS before upgrading, moving from N-1.10.x to N.10.x. If it were the same, we would drop the last cycle LTS when the intermediary LTS is released and validated by Manual QA, and we don’t want to do that for the reason that the intermediary LTS is not as polished than the cycle LTS.

Yes but it’s only for a small period of time and it’s needed so that users upgrading from cycle LTS to cycle LTS and wanting Manual QA validation can upgrade. To be precise, the problem of dropping the last cycle LTS before Manual QA has validated the new cycle LTS is that in the meantime (about 1 month+), we may find security issues or blocker issues that would be needed for our users and we would need to backport and release a N-1.10.x and N.4.x so that they can get the fixes ASAP and not have to wait 1 more month.

Ofc we should only backport to N-1.10.x and N.4.x very important bugs (i.e. blocker issues and some critical ones probably).

Thanks

13

Ok but I’m failing to really see the benefit of keeping the branch: in practice, right now we are waiting Manual QA to finish on 17.10.x to perform a last release of 16.10.x. So even if we found important bug fixes, admins would need to wait for 17.10.x to be the official new LTS to actually be able to upgrade on latest 16.10.x. You’ll probably answer that in case of a very important bug fix we could release very quickly a 16.10.x, but that’s actually true even if we don’t drop the branch. So clearly for me it’s adding a burden without any benefits even on the admins of those LTS cycle users.

I don’t agree with your argument here: intermediary LTS is as polish as a cycle LTS. It’s a LTS, we do perform Manual QA testing on it etc. I don’t see why you make a distinction here the only difference is that Cycle LTS are receiving full year updates, allowing admins to only have to perform “big” upgrades once a year, instead of every 6 months. But they could do the same by following only Intermediary LTS in theory.

EDIT: “the only difference is that Cycle LTS are receiving full year updates” that’s not even true, the only difference is that it’s happening in december.

14

How do you do a last release of 16.10.x if you’ve removed the branch when 17.10.2 was released (since Manual QA will have validated it, at best only a month after)?

What you say is true in term of quality but not in term of polishness. Manual QA doesn’t test for polishness (meaning that the feature started in the cycle are usually not finished and possibly even not usable from a functional production POV when the intermediary LTS is released).

That’s the reason it’s called “intermediary LTS” and not just “LTS”. That’s what we discussed back then. That’s what we tried to express at https://www.xwiki.org/xwiki/bin/view/Main/Support#HCommunitySupport :

Thx

15

So my opinion on that didn’t really change but clearly I won’t block the decision for that as we need to move forward. We’ll see next year how it’s going and if it needs to be rediscussed. So I can go for a +0 here if it’s less blocking.

16

Cool, we now have an agreement.

I’m going to start documenting When to drop the previous LTS branches when a new Cycle LTS is released? - #8 by vmassol and defining the details.

@tkrieck could you help me design 2 badges for the proposal? :slight_smile: I’ve tried asking gemini but it’s not doing a good job: https://gemini.google.com/share/a45b955604ea

Feel free to be creative and propose some alternate way to indicate when manual QA has validated a release.

Thanks