Decide if test code, test frameworks and page objects are public API or not

Development
1

Hi devs,

While reviewing some LLM guidance, @MichaelHamann pointed out that CKEditor (page object) has carried @Unstable for 9 years and ForgotUsernameCompletePage has @Unstable methods that are 5 years old. Nothing fails because Checkstyle (and thus UnstableAnnotationCheck) is skipped in those modules. I checked and we never decided what test code is regarding API rules. The only discussion I found is Support retro-compatibility for page objects (Jan 2016): Guillaume proposed to treat POs as API, Marius found it too much of a burden, Thomas suggested to rather take better care of TestUtils and friends. It ended without a vote.

Current state (inconsistent)

  • Test modules in platform (*-test parents): 57 of 69 skip both Revapi and Checkstyle (“we don’t consider them as public APIs”), 9 page-object modules skip Revapi but run Checkstyle (so the @Unstable check).
  • xwiki-platform-test-* (test-ui, test-docker, etc.): Revapi skipped, Checkstyle runs.
  • xwiki-commons-tool-test-*: Revapi skipped (all of xwiki-commons-tools skips it), Checkstyle runs.
  • xwiki-rendering-test: Revapi and Checkstyle both run, i.e. it’s treated as API today.
  • src/test: no @Unstable today (only the checker’s own test fixtures). However 34 modules publish their src/test as a test-jar used by other modules, e.g. MockitoOldcore lives in oldcore’s src/test.
  • JIRA: there’s a “Test Framework” component in XWIKI and a “Test” one in XCOMMONS/XRENDERING, but none for page objects, and we don’t strictly create issues for test framework changes.
  • Usage outside our core repos is real: among the contrib repos I have locally, 36 use the platform test frameworks and 23 import page objects from platform.

Proposal

  1. Test code (src/test, including code published as a test-jar) is never API. No @Unstable, no deprecation/legacy cycle, no JIRA issue required for changing it. This is mostly writing down what we already do. Reusable test code that is meant to be a test framework (shared by several modules) should be extracted into its own module, under src/main, rather than stay in src/test and be shared as a test-jar; it then falls under 2.
  2. Test frameworks (src/main of xwiki-commons-tool-test-*, xwiki-rendering-test, xwiki-platform-test-*, etc.) are not API. Same rules as 1. Rationale: most usage is ours, a break only affects developers at build time when they upgrade their parent version (never users at runtime), and the cost of the full API process (Revapi ignores, @Unstable, deprecation, legacy modules) isn’t worth it. Best effort is still expected: prefer deprecating over removing when it’s cheap, since contrib extensions use them.
  3. Page objects (src/main of *-test-pageobjects) are not API. Same rules and rationale as 2.

For 2 and 3, I propose that we keep requiring @since on new public classes and methods, since it’s cheap and it tells contrib developers which parent version they need.

Consequences if agreed

  • Remove the existing @Unstable annotations on page objects (7 classes today).
  • Align the poms: skip Revapi on all test framework and page-object modules (xwiki-rendering-test included). I’d rather enable Checkstyle on all of them (we lose all style checks on POs today, not just the @Unstable one), which would also catch any new @Unstable.
  • JIRA: no issue required for changes to 1-3. Keep using the “Test Framework” component for changes worth announcing to extension developers (new features of the test framework), but it’s optional.
  • Document this on https://dev.xwiki.org/xwiki/bin/view/Community/DevelopmentPractices#H40UnstableAnnotation

Here’s my +1 for 1, 2 and 3.

WDYT?

Thanks

2

+1 for 1, 2 and 3

Thanks,
Marius

3

OK for keeping the current practice of not seeing test tools as APIs.

Just note that as long as we do that, it makes impossible to run the same integration tests on several versions of XWiki (which was the main subject of the old discussion), which is something that would be very interesting for extensions.

That’s what we do when we can, but sometimes it’s not that easy. A good example is xwiki-platform/xwiki-platform-core/xwiki-platform-oldcore/src/test/java/com/xpn/xwiki/test at master · xwiki/xwiki-platform · GitHub which contains among the most used unit tests tools, but they are also needed by oldcore tests themself. And there are quite a few other similar cases (reusable test tool which are also needed by the module which contains the classes manipulated by the test tool).

4

+1, thanks