Hello! This is my proposal that we should implement a way to do visual regression testing in xwiki-platform, and how to go about it.
Context
We’re integrating BlockNote as a WYSIWYG editor option in XWiki. We discovered that, at least currently, the editor is tightly coupled with the default styling that comes out of the box. An example is this issue (XWiki-24239) in which we discovered that BlockNote hardcodes, as a pixel offset, where the block handle should be put, per block type, particular to their theming.
XWiki has a lot of theming support, so we expect there to be many quirks we’ll have to work around, to adjust for this.
I argue that we should introduce an approach to do visual regression testing, to be able to assert that the editor looks the way we want it to, and also to make it easier to validate, when we do upgrades, that our styling remains the same.
This would be useful for other areas of the product also, AFAIU it is not something we really check for currently, and it can help with subtle UI changes.
Approach
My suggestion is to use Selenium’s WebElement API to take screenshots, and to use the image-comparison library for the comparison step. The screenshots are scoped at the element level (not the whole page) to reduce as much as possible the surface area.
I wrote a generic ScreenshotComparator class which uses the above to achieve this. I propose to move this into xwiki-platform-test-docker to be able to use it with all newly written Selenium tests.
The method i suggest to expose is assertScreenshotMatches(String name, WebElement element), (renamed to assertMatches).
As a developer working on a new test, you’d write the test, run it, it will fail (as there is nothing to compare against) and generate a screenshot. Then you’d take a look, and if it looks correct, you’d have to promote it as the ‘canonical’ screenshot to compare against on future runs. You’d do this by moving the generated screenshots in the folder path corresponding to the test you wrote.
The path for the screenshots is: src/test/resources/screenshots/<Class>/<method>/<browser>/<name>.png
Limitations
- We need separate screenshot sets for both Chrome and Firefox, as rendering differs enough for them to not match.
- I haven’t tested this on an ARM machine but there is a good chance that the Chrome screenshots we have, will not work, since Selenium uses a Chromium driver on arm. And it’s really tricky to regenerate screenshots when you need to update a test, for two different architectures.
- Right now we’re avoiding having to scroll the page, to ensure screenshot stability. We haven’t really experimented much with scrolling but i expect it’ll cause issues. We could also look into increasing the viewport size for the drivers, it is currently quite tight (about 800px tall, and it is different between chrome and firefox.)
Example
Here is an example of how you could use this in a test, from the experiment we did:
@Test
void sideMenuIsAlignedOnLargeHeadings(TestUtils setup, TestReference testReference,
ScreenshotComparator screenshots) throws Exception
{
setup.deletePage(testReference);
setup.createPage(testReference, LARGE_HEADINGS_CONTENT);
assertSideMenuIsAligned(editInplace(), screenshots, LARGE_HEADINGS);
}
private void assertSideMenuIsAligned(BlockNoteRichTextArea textArea, ScreenshotComparator screenshots,
String[] blocks) throws IOException
{
WebElement content = new InplaceEditablePage().getContentContainer();
for (int i = 0; i < blocks.length; i++) {
textArea.hoverBlock(i);
screenshots.assertMatches(blocks[i], content);
}
}
And in the comparator class, the most important parts are:
public void assertMatches(String name, WebElement element) throws IOException
{
// The test name and the browser are part of the file names because the screenshots folder is shared by all
// the tests.
String prefix = "%s-%s-%s-%s".formatted(this.testClassName, this.testMethodName, this.browser, name);
File actualFile = new File(this.outputFolder, prefix + ".png");
BufferedImage actual = takeScreenshot(element);
ImageComparisonUtil.saveImage(actualFile, actual);
String referencePath = "src/test/resources/" + getReferencePath(name);
BufferedImage reference = readReference(name);
assertNotNull(reference, () -> ("There is no reference screenshot for [%s]. Check the screenshot taken by the "
+ "test, at [%s], and copy it to [%s] if it is correct.").formatted(name, actualFile, referencePath));
File differenceFile = new File(this.outputFolder, prefix + "-diff.png");
ImageComparisonResult result = new ImageComparison(reference, actual, differenceFile)
.setPixelToleranceLevel(PIXEL_TOLERANCE_LEVEL).compareImages();
assertEquals(ImageComparisonState.MATCH, result.getImageComparisonState(),
() -> ("The screenshot [%s] doesn't match its reference (%s%% of the pixels are different). Compare the "
+ "screenshot taken by the test, at [%s], with the reference screenshot, at [%s]. The differences are "
+ "highlighted at [%s].").formatted(name, result.getDifferencePercent(), actualFile, referencePath,
differenceFile));
}
private BufferedImage takeScreenshot(WebElement element) throws IOException
{
return ImageIO.read(new ByteArrayInputStream(element.getScreenshotAs(OutputType.BYTES)));
}
Conclusions
Despite some of the limitations, I personally think that this is the best way to go about achieving these kinds of automated UI tests that are aware of styling/positioning. The alternative, which I also tried, to do element-positioning math to check for alignment, is not maintainable IMO. But you can see the attempt here for comparison.
Let me know what you think.
Thank you,
Nicoleta C.
