Hi everyone!
There has been some work done on accessibility for XWiki in the past, however most of this work concerns the DutchWebGuidelines and WCAG 2.0, which are now out of date.
Fulfilling up to date accessibility standards is important to make sure the product is usable by disabled users, which also results in an improved UX for others.
That’s why adding a validation framework for the WCAG 2.1 ruleset should be a significant improvement to the XWiki tests. The goal of this framework is to quickly identify violations of the rules, so those issues can be reported with more ease and only issue solving stands between the project and passing the AA level of WCAG 2.1. .
To do this, I propose we use the axe-core framework.
More details about this proposal, and a description of the implementation proposal, are available on the XWiki Design page here.
In general +1, sounds good. I have two questions regarding this proposal:
I have the fear that the approach of just calling all kinds of documents won’t lead to a complete validation as it might miss many dialogs etc. that are only accessible by clicking on certain elements. Would it be possible to additionally integrate this testing into the existing Selenium-based integration tests we have? Either transparently (by validating after every page load or even after every click when enabled) or also just explicitly that integration tests would explicitly call the validator at certain points in the test? Or alternatively/additionally, could we integrate this in page tests where we have just some HTML fragments usually?
Do you have any plan how we deal with the initial phase where the validator will find a lot of problems? Do we just ignore the test failure? What’s the output of the validator, can we use it to measure progress?
I agree with Michael that we should be careful about that, to not have a build with a huge amount of failing tests that we take months to fix.
Maybe a solution would be to start integrating this with a dedicated maven profile and to only execute this profile on a specific job: we could then decide to add the profile to our standard build when we feel ready.
+1 to integrate it with our Selenium test suites (and Page Tests?), and to only activate it when a property is set to true (e.g., -Dxwiki.test.ui.accessibility=true).
I would do this by configuring the framework to decide the level of failure per WCAG rule (if that’s doable), i.e. initially don’t fail the build but report the errors, then once we’ve fixed all cases that make a rule fail, make it fail the build, and progress by fixing more and more rules.
Good afternoon!
I took your answers in consideration and came up with Vincent to a prototype of integration using axe-core.
How to report errors?
Using a specific parameter to trigger accessibility validation, so it’s not in the build too early. After the initial accessibility issues have been solved, accessibility issues will make the TestSuite fail, but not the tests themselves. This way, even if a change creates a lot of accessibility issues, we avoid making all tests fail.
Where to validate?
It’s important to test it on default available pages, but also on dynamic elements that are already tested in the functional tests. Because functional tests should eventually cover all pages, we can integrate this validation to the tests themselves. This way, we can make sure the interface of XWiki is compliant. However, some content might not be covered and there might be a need to create an “accessibility test” page with various kind of content a user can create (such as a livetable, …). This test page is secondary however, functional tests should already cover most of the standard content too.
An advantage of validating during functional tests is that the environment is already started and we can use this setup for both functional testing and accessibility validation.
How to validate?
As Michael pointed out, there are two ways to do so:
Transparently
Explicitly
For functional testing, it’s mandatory to use a PageObject to communicate between the driver and the test itself. It’s possible to add the validation when instanciating a Base Element.
Pros:
All interactions go through base elements, and this can avoid repeating in one test a validation on an element already validated.
Allows transparency and overriding of behavior to fit the validation more properly the PageObject.
Cons:
Can still have a lot of repetitions from one test to another.
→ Can be reduced by creating a cache of “already validated components”
Isn’t there a problem that, e.g., LiveData might not be fully initialized when the validation is called at the end of the BaseElement constructor as LiveDataElement waits at the end of its constructor? BasePage is now also waiting for the page to be ready but again, the validation will be executed before this as the BaseElement constructor is called before the waiting happens. I fear that this could create quite some flickering regarding the WCAG results as the validation might happen while dynamic elements are still loading and some parts of the document might not be included in the validation when they’re late.
I don’t see how this should work as the same element can be used with very different contents (like a ViewPage can represent any kind of XWiki document or Live Data can have very different contents where one might be accessible and another isn’t).
Given the performance impact, I’m wondering if it is really useful to run accessibility tests on every element constructor or not rather only for BasePage and then provide an easy-to-use helper method that can be triggered to re-run the accessibility tests when, e.g., a popup menu or dialog has been opened. In BasePage, we’re also explicitly waiting for the page load to finish so there we have a much higher chance that the results will be stable.
That’s not how I read our guidelines, while it says that each feature should have a functional test, it explicitly says that if something can be tested in other ways like a page test, it should be done that way. We’re moving some of our integration tests to page tests now due to running time and flickering tests.
Is there a reason not to run accessibility tests outside of integration tests?
Also, another question: most of our functional tests don’t use the full skin I think but a minimal environment that, e.g., doesn’t use FontAwesome icons. Doesn’t this affect accessibility tests when, e.g., the icons are different? Here a screenshot from an integration test:
Notice how the color of the top bar and the icons are very different from a standard installation and there are, e.g., no panels. I mean we should probably test with both icon sets that we currently have but I think it would make sense to run the majority of the tests with our default icon set.
Last weekend I read a very relevant article on this subject: automated testing won’t solve web accessibility that also links to some comparison of validation tools (though all browser-based) and how they fail to find most WCAG violations. While the article (and linked articles) agree that using automated tools is helpful, they also show that they are no replacement for manual testing.
Thanks @MichaelHamann ! Yes, we’re well aware of this. Actually Axe Core even computes it and announces that they’re able to find about 57% of WCAG issues (I don’t consider this small BTW).
Set up some manual test process to complement the automated tests
Right now the focus is on 1) and 2) (and a bit of 3) since Lucas is also using a screen reader to check the accessibility and reporting jira issues) and that’s already a long way from where we stand ATM.
That’s good, the initial post creates a different impression:
This can be easily understood as solving all violations the framework identifies is enough for passing the AA level of WCAG 2.1.
Also, I thought the post or rather the post linked in this post could be interesting as another comparison of different tools that is focused on many (and which kind of) violations they identified which could be interesting also for manual testing (as these tools are all browser-based).
You’re right. In the cache we should store the component class and the URL of the page beoing tested. This way, we can avoid a substantial amount of validations without losing too much detail.
A cache will however prevent variations on context (such as user status, user language or simple/advanced users). Moreover, this will only test the first element of a class on an URL.
This should be a manageable downside because those elements should have very similar (if not completely) UIs.
The guidelines do read that every feature needs at least one guideline, so validating through functional tests should be enough to cover most of the features.
Page tests, on the other hand, consist of a lot of slight variations over the same elements. Validation would be costly and provide minimal benefits. Moreover, some page tests use simplified HTML, and that can be incorrect and trigger false positives in the tests.
It’s where we can add them for a minimal cost and maximal efficiency: there’s no need to start all the containers when already in integration testing, and we can test out the a11y on functional states (which I think are for the most part done during integration tests). We might however need to add some system level tests at some point to further complete the validation.
The full skin is tested in its own functional tests where the appropriate WCAG issues will arise (see. xwiki-platform-flamingo-skin-test-docker). Same for the icons (see. xwiki-platform-icon-test-docker). However those tests are out of the use context (e.g. icons in the breadcrumb are on a different background) and some rules will be impacted by the reduced skin.
See here for a list of rules that might be impacted.
We can decide to skip the validation of these rules during integration tests altogether, and conduct a specific wcag testing suite where we only check those rules.
The first possibility is to start a XWiki standard and crawl through it while validating all pages.
The second possibility is to parse the xars and validate each URL we find here.
This will not be enough to validate interfaces with multiple kinds of users, however it should cover a11y of most of the interface.
These should be useful mostly for “live” testing while correcting issues. The results of these tools should be for the most part similar to what we find using automated testing (if their parameters are precise enough to recreate our testing needs).
I think tools such as this firefox extension are those which will be able to give value to the manual accessibility tests. Using such a tool, the test would be much longer to process than automated checking through an extension, however it would be able to spot out much more of the 43% of a11y violations axe-core won’t find.
yes you’re right. We’ve decided to try using BasePage FTM. We can easily provide access to AxeCore’s validate method (which takes a WebElement) to assert some specific UI element. The question (for the future) will be whether we want to do inside the existing functional tests or add some new functional tests only used to validate something specific regarding WCAG.
We can add support for WCAG validation in PageTest in the future. Right now, the idea is to focus on functional tests and see where that gets us.
We also have distribution tests (ie XS flavor functional tests) which use the proper color and icon themes. They won’t test all the pages though. As Lucas said, there are options for later on, to cover more. But we’d like to start with functional tests first and see where that gets us. This can be tested by checking for example if browser-based WCAG validation catch more errors on xwiki.org’ playground than the ones caught by our functional tests (with WCAG validations).
