Testing priorities proposal

[ilie.andriuta]

Hi!

Following the discussion we had for the analysis of the manual testing strategy for XWiki product, we need to establish a set of tests that are indicated to be run on each test run session among other product (and/or recommended extensions tests) (full list available at https://test.xwiki.org/xwiki/bin/view/Main/).

We selected the following tests to be part of the list:

For P1.Product:

Administration tests

• CAPTCHA Image
• Change the language of an wiki
• Configure Mail Sending with Sender email address
• Create Page from Office
• Export and Import Partial Wiki
• Install an Extension
• Name strategies - preformatted names strategy
• Space Panel Wizard Drag And Drop Panel From Panel List To Side
• Recommended Templates when creating pages
• Set Language Settings
• Setting password security policy
• Switch the language
• Uninstall an Extension
• Upgrade an Extension

Nested Spaces

• Create a Nested Document using Add button
• Copy a WebHome page from one space to another
• Create a Nested Document without creating the parents
• Create a terminal page
• Delete a document
• Export Import a document with defined permissions
• Override a right at page level
• Warning conflict when editing a page
• Rename a Document

Security tests

• Deny the rights for a user at wiki level and give rights to a group at space level
• Subwiki - Deny the rights for a user at document level and give the rights to a group
• Grant Create Wiki Right For An User

Search tests

• Create Page And Search
• Search For Attachments

App Within Minutes Tests

• Create an AWM application in a nested document
• Delete an AWM Application
• In place edit for application entries
• Rename Move an AWM application

CKEditor tests

• Add a link using the CKEditor menu
• Add Emoji
• Automatic link creation when dropping a non-image file
• Copy paste text
• Edit a section of a page in place
• Edit source Save and View
• Import Office File
• In-line macro content editing
• Insert Display Macro
• Insert Office Document Viewer Macro
• Insert a Special Character
• Add a simple macro in CKEditor
• Link autocomplete
• Live macro rendering when editing in place
• Save and View Fullscreen
• Switch to source while editing in place

Wiki tests

• Add And Delete Users To A Wiki
• Create Wiki From A Template With Extensions Installed
• Translate a document in a different language
• Translate page while editing in place
• Login With Local User From a subwiki

Web tests

• Add a Comment from More Actions Menu
• Delete an attachment from More Actions Menu
• Attach Same File Twice
• Authentication security module CAPTCHA
• Authentication security module disable account
• Forgot your password
• Forgot your username
• Import Office Document
• Register an user using email verification
• Replace Author when Deleting Users
• Restore a Page
• View An Office Attachment

User tests

• Change Avatar Image Select Another Image
• Change Password
• Enable and disable user account
• Edit Profile
• User membership in the profile

Notifications tests

• Add exclusive filters
• Add inclusive filters
• Clear All notifications
• Live notifications on subwiki
• Live email notifications with filter
• Load older notifications
• Notifications count (below 21)
• Notifications for a page from a subwiki

User Mention tests

• Blog Mention - structured pages
• Mention an user from a subwiki
• Add inclusive filters

Page Likes tests

• Like and Unlike a page
• View Page Likes

For P2.Product:

Dashboard tests

• Create a Dashboard Page

Color Themes tests

• Base Colors Headings Color
• Base Colors Text Color
• Buttons Default Background
• Change Logo
• Typography Font Family Sans Serif

Export tests

• Export an Office page as ODT
• Export a page as Pdf

Macros tests

• Notifications Email Preferences Macro
• Notifications Filters Preferences Macro
• Notifications Macro on subwiki restrict displayed events
• Notifications Macro

Index tests

• Filter Nested Document
• Page Index Deleted Attachments
• Restore a deleted page

For P3.Product:

Activity Stream tests

• Create a page event

Annotations tests

• Delete an Annotation
• Reply To an Annotation

Other tests

• Homepage Tour

Tag tests

• Retrieve a Tag

Message Stream Tests

• Send Message To Followers
• Send Message To Group

It has to be taken into consideration that on test sessions (not for every XWiki version, though) there are tests run for Recommended extensions also.

These are the current priorities for product tests:

P1.Product:

• http://test.xwiki.org/xwiki/bin/view/Administration+Tests
• http://test.xwiki.org/xwiki/bin/view/Nested+Spaces/
• http://test.xwiki.org/xwiki/bin/view/Security+Tests/
• http://test.xwiki.org/xwiki/bin/view/Search+Tests/
• http://test.xwiki.org/xwiki/bin/view/Upgrade+tests/
• http://test.xwiki.org/xwiki/bin/view/Syntax/
• http://test.xwiki.org/xwiki/bin/view/AppWithinMinutes+Tests/
• http://test.xwiki.org/xwiki/bin/view/CKEditor/
• http://test.xwiki.org/xwiki/bin/view/Wiki+Tests/
• http://test.xwiki.org/xwiki/bin/view/Web+Tests/
• http://test.xwiki.org/xwiki/bin/view/User+Tests/
• http://test.xwiki.org/xwiki/bin/view/Notifications Tests/
• http://test.xwiki.org/xwiki/bin/view/User Mentions Tests/ [Since XWiki 12.5 RC1]
• http://test.xwiki.org/xwiki/bin/view/Page Likes Tests/ [Since XWiki 12.7 RC1]

P2.Product:

• http://test.xwiki.org/xwiki/bin/view/Panels+Tests/
• http://test.xwiki.org/xwiki/bin/view/Dashboard+Tests/
• http://test.xwiki.org/xwiki/bin/view/Color+Themes+Tests/
• http://test.xwiki.org/xwiki/bin/view/Export/
• http://test.xwiki.org/xwiki/bin/view/Macros/
• http://test.xwiki.org/xwiki/bin/view/Index+Tests/
• http://test.xwiki.org/xwiki/bin/view/Skin+Tests/

P3.Product:

• http://test.xwiki.org/xwiki/bin/view/Activity+Stream+Tests/
• http://test.xwiki.org/xwiki/bin/view/Annotations+Tests/
• http://test.xwiki.org/xwiki/bin/view/Other+Tests/
• http://test.xwiki.org/xwiki/bin/view/Tag+Tests/
• http://test.xwiki.org/xwiki/bin/view/Message+Stream+Tests/

Any feedback regarding the minimal list of tests chosen above or about changing the prioritization hierarchy of tests is welcomed.

Thanks!

[vmassol]

@ilie.andriuta

We selected the following tests to be part of the list:

Hi @ilie.andriuta. I think you’re missing the explanation of how you came up with this list and the strategy used.

Also what do P1, P2 & P3 mean for testing? Why do we need categories?

Personally I’m fine to go with this list for now and to document it in https://dev.xwiki.org/xwiki/bin/view/Community/ManualTestStrategy. I see there’s already a list there about priorities. Is it different from this new list you’re proposing?

Also there are some outstanding questions regarding to manual testing:

• Adopting a time-box testing strategy, as we discussed. Could you propose that?
• Explain the strategy to use to handle new features: what is the process to add manual tests for them? Who does it? How do we decide which priority they are compared to the existing tests?
• Also describe the strategy from reducing the number of manual tests vs automated test that we just started.

All of this should be proposed and https://dev.xwiki.org/xwiki/bin/view/Community/ManualTestStrategy be udpated.

What I suggest is that you start by updating https://dev.xwiki.org/xwiki/bin/view/Community/ManualTestStrategy so that it’s the best possible and includes answers to all the questions I’ve asked and then you propose it on this forum to make sure everyone agrees. WDYT?

Thanks!

[ilie.andriuta]

Hi! I’ve restructured and updated https://dev.xwiki.org/xwiki/bin/view/Community/ManualTestStrategy to include more detailed explanations.

The list of tests proposed here is the same as the list from the updated page.

The main reason for categorizing tests in a list of testing priorities both for XWiki and for recommended extensions (P1, P2 and P3) is to systematize the process of manual testing and to make sure the important features are tested with increased priority. These priorities were established (also) based on the feedback received from teams/community.

About adopting a time-box strategy, I propose (as we discussed) the following:

• RC and bugfix releases: 1 week/person
• Stable releases: 2 weeks/person
• LTS releases: 3 weeks/person

In order to reduce the number of manual tests, a strategy has been started to identify manual tests run for XWiki Platform for which there are already automated tests or to create tasks for automation of tests that don’t exist yet and can be automated. Briefly, this strategy consists in:

• One manual test per week (usually, depending on the complexity of tests) is analyzed and discussed on #xwiki:matrix.xwiki.com. With the help of Developers team will be searched whether an automated test exists and has or not the same scenario as the manual test;
• If an automated test doesn’t exist or the scenario is different, will be decided if a JIRA ticket should be raised for it (to be automated) or should be dropped and marked as a deprecated manual test.

Regarding the strategy to handle tests for new features, we need to define and detail it a little bit.
Following the brief discussion about it, we said that it should be asked if there are tests created already for the respective new features, this meaning that on every release it will be asked on #xwiki:matrix.xwiki.com about some scenarios, which I don’t know how it will work out.

On the other hand (from past experience) it proved to be useful for many having manual tests for the new features (with steps to reproduce), even if they should/will not be executed manually. There are even cases when these tests are used by other people to see how features work and helps them to reproduce issues for some use-cases.

Thanks for the feedback!