Testing priorities proposal


Following the discussion we had for the analysis of the manual testing strategy for XWiki product, we need to establish a set of tests that are indicated to be run on each test run session among other product (and/or recommended extensions tests) (full list available at https://test.xwiki.org/xwiki/bin/view/Main/).

We selected the following tests to be part of the list:

For P1.Product:

Administration tests

Nested Spaces

Security tests

Search tests

App Within Minutes Tests

CKEditor tests

Wiki tests

Web tests

User tests

Notifications tests

User Mention tests

Page Likes tests

For P2.Product:

Dashboard tests

Color Themes tests

Export tests

Macros tests

Index tests

For P3.Product:

Activity Stream tests

Annotations tests

Other tests

Tag tests

Message Stream Tests

It has to be taken into consideration that on test sessions (not for every XWiki version, though) there are tests run for Recommended extensions also.

These are the current priorities for product tests:




Any feedback regarding the minimal list of tests chosen above or about changing the prioritization hierarchy of tests is welcomed.


Hi @ilie.andriuta. I think you’re missing the explanation of how you came up with this list and the strategy used.

Also what do P1, P2 & P3 mean for testing? Why do we need categories?

Personally I’m fine to go with this list for now and to document it in https://dev.xwiki.org/xwiki/bin/view/Community/ManualTestStrategy. I see there’s already a list there about priorities. Is it different from this new list you’re proposing?

Also there are some outstanding questions regarding to manual testing:

  • Adopting a time-box testing strategy, as we discussed. Could you propose that?
  • Explain the strategy to use to handle new features: what is the process to add manual tests for them? Who does it? How do we decide which priority they are compared to the existing tests?
  • Also describe the strategy from reducing the number of manual tests vs automated test that we just started.

All of this should be proposed and https://dev.xwiki.org/xwiki/bin/view/Community/ManualTestStrategy be udpated.

What I suggest is that you start by updating https://dev.xwiki.org/xwiki/bin/view/Community/ManualTestStrategy so that it’s the best possible and includes answers to all the questions I’ve asked and then you propose it on this forum to make sure everyone agrees. WDYT?


Hi! I’ve restructured and updated https://dev.xwiki.org/xwiki/bin/view/Community/ManualTestStrategy to include more detailed explanations.

The list of tests proposed here is the same as the list from the updated page.

The main reason for categorizing tests in a list of testing priorities both for XWiki and for recommended extensions (P1, P2 and P3) is to systematize the process of manual testing and to make sure the important features are tested with increased priority. These priorities were established (also) based on the feedback received from teams/community.

About adopting a time-box strategy, I propose (as we discussed) the following:

  • RC and bugfix releases: 1 week/person
  • Stable releases: 2 weeks/person
  • LTS releases: 3 weeks/person

In order to reduce the number of manual tests, a strategy has been started to identify manual tests run for XWiki Platform for which there are already automated tests or to create tasks for automation of tests that don’t exist yet and can be automated. Briefly, this strategy consists in:

  • One manual test per week (usually, depending on the complexity of tests) is analyzed and discussed on #xwiki:matrix.xwiki.com. With the help of Developers team will be searched whether an automated test exists and has or not the same scenario as the manual test;
  • If an automated test doesn’t exist or the scenario is different, will be decided if a JIRA ticket should be raised for it (to be automated) or should be dropped and marked as a deprecated manual test.

Regarding the strategy to handle tests for new features, we need to define and detail it a little bit.
Following the brief discussion about it, we said that it should be asked if there are tests created already for the respective new features, this meaning that on every release it will be asked on #xwiki:matrix.xwiki.com about some scenarios, which I don’t know how it will work out.

On the other hand (from past experience) it proved to be useful for many having manual tests for the new features (with steps to reproduce), even if they should/will not be executed manually. There are even cases when these tests are used by other people to see how features work and helps them to reproduce issues for some use-cases.

Thanks for the feedback!