Hi devs/Ilie,
Context: Ilie is a contributor of the xwiki open source project who focuses on doing manual tests for XS (all the tests he executes are on https://test.xwiki.org/). There are too many tests listed there for him to execute all of them. We need a better strategy. One area that is promising is to identify manual tests for which there are already automated tests existing.
Strategy proposal:
- Ilie takes a few of the manual tests every week (not too much so that it doesn’t impact his testing schedule)
- Ilie checks if there are automated tests for them. How to achieve this?
- Initially he’ll ask questions on the #xwiki chat about that but as time progresses he should be able to identify tests himself.
- Once an automated test is found, Ilie tries to read it to understand what it does. Similarly he asks questions on #xwiki when it’s not clear. For functional tests it should not be too hard since we use PageObjects.
- If there’s no automated test, Ilie creates a jira issue for it and a discussion can happen in jira to decide if it’s a won’t fix for whatever reason or where it should be located, what should be done, etc. It’s also important to mark the manual test with metadata to indicate its status and the link to the jira issue.
- When there are some differences in the automated tests, Ilie starts discussing them on #xwiki to decide if these differences are important and should be added to the existing tests or if they should be dropped from the manual test. If some work needs to be done, Ilie creates a jira issue for it. Again the manual test is annotated with metadata and the jira link.
- When there’s an automated test, no jira is created but the manual test is annotated with metadata. Also link the manual test with the automated test github test class link.
- Ilie monitors new tests added every day/week, as they are added to update the test plan (verify if there are manual tests for it or not, add a new test entry and link it to the automated test).
Open questions:
- How can Ilie follow new tests added every day/week?
Other ideas:
- Modify our automated tests to be able to automatically generate a human-readable scenario for them and import/synch them with test.xwiki.org. This could be a solution to the question above “How can Ilie follow new tests added every day/week?”.
Let’s start the discussion. Throw ideas if you have some.
Thanks