Supported version strategy, take 3

Hi devs,

In https://forum.xwiki.org/t/new-database-and-servlet-container-support-strategy/6934 we agreed to support 2 versions (latest and LTS), with an exception for debian “stable”, see https://dev.xwiki.org/xwiki/bin/view/Community/SupportStrategy/LatestLTSDefinition/

There we say:

In addition and since we provide a Debian packaging, we also support the “stable” version of the product, as defined by Debian, when it doesn’t match the “latest” and “LTS” versions above.

I’d like to propose a change to this rule since I feel that the likelihood to find problems vs the cost of implementing it is too low (basically it costs too much for the results). It leads to too many configs tested and slows down our builds.

So I’d like to propose and idea that @tmortagne proposed on the chat:

New rule: Always test a maximum of 2 versions and when the debian “stable” version is different, take the highest and lowest versions.

For example, instead of having:

'mariadb' : [ 'latest' : '10.5', 'lts' : '10.4', 'debian' : '10.3' ],
'postgresql' : [ 'latest' : '12', 'lts' : '11', 'debian' : '11.7' ],

We would have:

'mariadb' : [ 'latest' : '10.5', 'lts' : '10.3'],
'postgresql' : [ 'latest' : '12', 'lts' : '11.7'],

(note that lts is 11.7 in this example because the latest of the 11 docker tag is currently 11.9 which is > 11.7)

WDYT?

Thanks

Hi,

globally +1 on the idea, but just to be sure does it mean we wouldn’t accept anymore specific version to test if it leads to having more than 2 version to test?
I can see for example that we currently have:

'tomcat' : [ 'latest' : '9-jdk11', 'lts' : '8.5-jdk8', 'special' : '9-jdk8' ],

I guess that if we introduced this “special” version, it’s because we specifically wanted to test it no?
Or does the rule “a maximum of 2 versions” only stands regarding Stable / LTS / Debian?

+1 but I’m wondering if we should not compare cycles (granularity to be defined product by product since it’s probably not the same for mariadb and postgresql here for example) instead of exact versions which would mean

'mariadb' : [ 'latest' : '10.5', 'lts' : '10.3'],
'postgresql' : [ 'latest' : '12', 'lts' : '11'],

Yep that’s the idea. We’ll always have special cases when we want to test something extra but that’s a case by case basis.

1 Like

Thomas, if you do this then you drop the idea of always supporting “stable” for Debian. Is that what you’re proposing?

EDIT: 11 means 11.9 today for ex while debian is on 11.7 for stable.

I’m not sure it’s really different actually, in the example you pasted there is only two different versions of Tomcat and two different version of Java so it’s not really about versions here but more about how many configurations we test.

In this case the reason for the special version is to verify we support the latest Tomcat on Java8 (ie even more recent than latest. The reason is that since we wanted to verify we support utf8 we might as well do some advance testing on the most recent tomcat:

        // Special case: verify we still support utf8 for MySQL and at the same time test with latest Tomcat on Java8
        // to potentially discover problem in advance (to add more value since we're doing another config test).

Anyway the point is that we’ll always have special versions and that’s outside the scope of this proposal.

Yes I understood that, my point here is that we should define the granularity of a cycle and it seems to me that in the case of Postgres the cycle is the major version which would make the version provided by Debian the “same” (cycle 11).

Could you explain what you mean by the same? Because for me 11.9 is not the same as 11.7 and if we use 11 then we don’t test 11.7 but 11.9.

I tough I did. What I’m wondering is if we can define a cycle as a group of versions which should not really be different from XWiki point of view. When looking at Postgres website and Debian package naming it feels like 11 is such a cycle (i.e. there is no enough differences between 11.9 and 11.7 to put an explicit version and we could keep the docker “11” tag).

For example I really doubt anybody wants to consider that Tomcat 8.5.39 (Debian stable) and 8.5.58 (docker 8.5-jdk8 label) are different enough to worth testing a more explicit 8.5.39 instead.

So if I try to summarize it (still not sure I get it), the rules would be:

  1. Always test a maximum of 2 versions and when the debian “stable” version is different, take the highest and lowest versions.
  2. Exception to the first rule: when the lowest version is part of the same “cycle” (we need to define that more precisely since it doesn’t mean much) then don’t use the lowest version but the latest one.

Have I understood your point?

If so then I think it’s way too complex and I don’t see what it gets us. It also means defining what is a “cycle” for each product which is a major pain. I’ve probably misunderstood somehting.

Thanks

Well I feel we already do that when we decide what is “latest” and what is “lts” cycles.

I now read again this thread and trying to take into account Thomas’s proposal, and at the same time trying to make something easy to do, I’m proposing this:

  • Always test a maximum of 2 versions
  • Always test the latest version of the current cycle (i.e. latest of latest)
  • Always test the latest version of the previous cycle (i.e. latest of lts)

Taking the original examples of:

'mariadb' : [ 'latest' : '10.5', 'lts' : '10.4', 'debian' : '10.3' ],
'postgresql' : [ 'latest' : '12', 'lts' : '11', 'debian' : '11.7' ],

This would mean:

'mariadb' : [ 'latest' : '10.5', 'lts' : '10.4'],
'postgresql' : [ 'latest' : '12', 'lts' : '11'],

In practice this means that we wouldn’t test 10.3 for mariadb which was the debian version, but so be it. And we test the latest of 10.4 instead. In other words, we stop caring too much about debian and hope that it’ll follow the previous cycle quite quickly.

I feel that if we need to make concessions, it makes sense to test the latest of the current + previous cycles.

WDYT?

Thanks

It’s a good starting point and we should probably apply that.

I do think we should only test two versions no matter what. Now of which versions exactly I just feel that we should not completely refuse the possibility that we might have different selection rules in the future depending on the tools since they all have a very different lifecycle and specific problems.