Isn’t there a problem that, e.g., LiveData
might not be fully initialized when the validation is called at the end of the BaseElement
constructor as LiveDataElement
waits at the end of its constructor? BasePage
is now also waiting for the page to be ready but again, the validation will be executed before this as the BaseElement
constructor is called before the waiting happens. I fear that this could create quite some flickering regarding the WCAG results as the validation might happen while dynamic elements are still loading and some parts of the document might not be included in the validation when they’re late.
I don’t see how this should work as the same element can be used with very different contents (like a ViewPage
can represent any kind of XWiki document or Live Data can have very different contents where one might be accessible and another isn’t).
Given the performance impact, I’m wondering if it is really useful to run accessibility tests on every element constructor or not rather only for BasePage
and then provide an easy-to-use helper method that can be triggered to re-run the accessibility tests when, e.g., a popup menu or dialog has been opened. In BasePage
, we’re also explicitly waiting for the page load to finish so there we have a much higher chance that the results will be stable.
That’s not how I read our guidelines, while it says that each feature should have a functional test, it explicitly says that if something can be tested in other ways like a page test, it should be done that way. We’re moving some of our integration tests to page tests now due to running time and flickering tests.
Is there a reason not to run accessibility tests outside of integration tests?
Also, another question: most of our functional tests don’t use the full skin I think but a minimal environment that, e.g., doesn’t use FontAwesome icons. Doesn’t this affect accessibility tests when, e.g., the icons are different? Here a screenshot from an integration test:

Notice how the color of the top bar and the icons are very different from a standard installation and there are, e.g., no panels. I mean we should probably test with both icon sets that we currently have but I think it would make sense to run the majority of the tests with our default icon set.