Hello!
The goal of this topic is to discuss how to test accessibility automatically for Cristal. The solution to this is very dependent on what kind of automatic testing tool we’ll use. This has not been discussed yet, so I decided to think about two likely possibilities:
- Cristal tests rely on the existing functional docker test framework with Selenium.
- Cristal functional tests rely on playwright, like the component tests do.
In XWiki Standard, our accessibility tests only check full page during functional tests. The main point I want to discuss is how to reach the same level of testing for Cristal.
Docker + Selenium functional tests
In this case, we can reuse the existing accessibility testing framework. All the functional tests for Cristal would start by installing the Cristal integration extension, then toggling ON the Cristal UI and then doing the actual functional tests.
With this solution, the only implementation that might be required is a more powerful system to handle the set of failing rules. For Cristal development, there might be a lot of WCAG fails popping up all at once with the first test of a new complex feature. Technically, this means that merging this test would be delayed until all WCAG fixes have been provided. This could slow things down a lot and I think it’d be valuable to provide a way around this.
I think that generalizing the system we had setup for XS as a whole to some modules (assuming they provide the appropriate config ofc) would help in transitioning from no WCAG test to full automated test success. This means that after a first set of tests, the tester/dev sets all the rules violated in Cristal as “non-failing rules”, and we set them as “failing” as soon as there’s no violation detected in the tests anymore.
Note that I got feedback from extension contributors for whom such a system would be useful too.
Playwright tests
Right now, we have some component tests using playwright at xwiki-platform/xwiki-platform-core/xwiki-platform-node/src/main/node/editors/blocknote-react/src/__tests__/BlockNote.spec.tsx at 5ea44a6aaf1a3726149c73560a5ce1507bfc7587 · xwiki/xwiki-platform · GitHub . Playwright functional tests are based on fixtures. Similarly to what we do in selenium, we can extend the base page fixture and add our WCAG checks in its initialization. Axe-core is supported in playwright thanks to the package https://www.npmjs.com/package/@axe-core/playwright . AFAICS, all the axe-core features we use in the selenium part of GitHub - dequelabs/axe-core-maven-html: Tools for using axe for web accessibility testing with JUnit, Selenium, and Playwright are also available in this package. It’s even more complete, it would probably make some of the implementation easier 
The advantage of this tool is that it’d be quite straightforward to use it in component testing.
I tested the package by manually writing a check in a test and importing the package with a few lines and the results look just like I would have expected: a few fails with clear descriptions and enough info to understand, similar to what we get initially when running the selenium accessibility tests. I tried to use the hooks available for the component tester to make it work without any change to the existing test code but infortunately I didn’t figure it out yet. I think it’s mostly because I’m unfamiliar with playwright and the info available for the component tester is sparse since it’s still an experimental feature.
I think we should leverage the component structure of Cristal for accessibility testing. The current accessibility testing on XS does not have a perfect coverage, and running accessibility tests after each “mount” in the component tester would increase the coverage without increasing the computational cost drastically. In addition, it would avoid the issue we have today in XS when more often than not, one violation introduced triggers ten or more fails all at once because the failing UI is used on a few pages in a few tests.
From my experience, those component tests for accessibility are just as valuable than end-to-end accessibility tests. In most cases, a new accessibility failure happens in a UI no matter how we use it exactly, the tests themselves usually accounts for all possible cases. But they are less computationally costly so it’s possible to run them “more frequently” which outweighs possible unaccounted interactions between components.
Conclusion
From an accessibility point of view, using the existing docker framework is definitely the easiest implementation. But if we decide to rely on another framework for Cristal end-to-end testing, we should definitely use the structure provided in Cristal as components to our advantage to reach a higher coverage with lower costs.
AFAICS, accessibility testing works well with playwright. If we decide to pick a new testing framework, we need to make sure there is an available axe-core package and that it’s regularly updated. Given that, the results obtained with axe-core can be handled similarly to what was done in selenium.
I’ll propose a more concrete plan once we agree on a testing framework @mleduc .
Do you agree with these ideas about how to provide accessibility tests for Cristal?
Is there any consideration I forgot to make?
Lucas C.