Remove the Dutch Web Guidelines validation

[vmassol]

Hi everyone,

Following Dutch Web Guidelines Validation Status, where we agreed to keep the DWG tests until we’d reviewed what they check beyond WCAG, I’ve now done that rule-by-rule review. Since then, our official commitment is WCAG 2.2 AA (not AAA), and it’s validated by axe-core in the Docker functional tests. So IMO the question is no longer “are the DWG rules good?” but “which DWG rules do we need for WCAG 2.2 AA that axe-core or the HTML5 (Nu) validator don’t already check?”.

Context: I’ve just converted the webstandards tests (HTML5, XWiki, DWG and RSS validation of all the flavor pages) from JUnit 3 to JUnit 5, in [Misc] Convert the flavor web standards functional tests to JUnit 5 by vmassol · Pull Request #7279 · xwiki/xwiki-platform · GitHub, as part of removing the JUnit 3/4 leftovers. That made me look at the DWG part again.

I’ve reviewed HTML5DutchWebGuidelinesValidator:

• It has 125 validateRpdXsY() methods, but only 31 checks actually do something. The other 94 are empty (“cannot be automatically tested”, “checked by XHTMLValidator”, “No DOCTYPE in HTML5”…).
• 6 of the 31 are warnings, which never fail the build (they’re only printed).
• 29 are either not WCAG requirements at all, AAA-only, obsolete (e.g. <q> because of Internet Explorer, target “deprecated” which is false in HTML5), code style (<br><br>, <b>/<i>, inline style), or already checked by axe-core and/or Nu:

Rule	Check	Why it’s not needed
1s3, 13s4	no javascript: href, no href="#" + onclick, every form has a submit button, no select[onchange]	no SC requires it (3.2.2 only forbids an actual change of context on input)
3s1, 3s5	no <b>/<i>	valid HTML5, no SC
3s2	page has an h1	not required by WCAG (axe best-practice rule only)
3s3	no skipped heading levels	not a WCAG failure, and Nu now checks it (our implementation is also buggy)
3s4, 3s9, 3s11	no <br><br>, no <sub>/<sup>, no <q>	code style / obsolete
6s1, 16s1, 16s2, 16s4	doctype, charset meta	not accessibility, checked by Nu (16s1 even rejects HTML5’s <meta charset>)
7s1, 7s4, 7s5	alt on images, image links, image maps	1.1.1/2.4.4, checked by axe (image-alt, link-name, area-alt, input-image-alt) and Nu
8s1	no “click here” link text	2.4.4 (A) allows context, link-only is 2.4.9 (AAA)
8s11	accesskeys must be numbers	no SC
8s14	no target attribute	3.2.5 is AAA, and target isn’t deprecated
8s16, 8s17	mailto address visible / no ?subject=	no SC
9s1	no style attribute, no <style>	no SC (the 1.4.12 part is axe avoid-inline-spacing, Nu flags <style> in body)
11s4, 11s5	scope or headers/id on all tables	only needed for complex tables (H63/H43), axe checks td-headers-attr/th-has-data-cells
12s1	iframe title + role	title checked by axe frame-title, role not required
13s3, 13s5, 13s18	fieldset, onsubmit/onchange, no reset button	no SC

The 2 remaining ones are WCAG A requirements that axe doesn’t fail on today:

• 11s2 (data tables must have th, 1.3.1): axe has td-has-header, which is experimental and thus not run by default. It’s also more accurate than DWG (data cells of tables larger than 3x3, layout tables ignored).
• 13s1 (inputs must have a label, 1.3.1/3.3.2/4.1.2): axe’s label rule is configured as warning-only in WCAGContext.

Proposal

Remove the DWG validation completely: the DWG tests in the webstandards module, CustomDutchWebGuidelinesValidator with its exceptions, the hidden pages skip + class whitelist, and HTML5DutchWebGuidelinesValidator. The HTML5 (Nu) and XWiki validations of all the pages stay. I’d do it right after #7279 is merged, and update the DWG doc page to say it’s been removed and why.

Follow-ups, nice to have but not blocking the removal:

1. Enable axe’s td-has-header rule, as a warning first, and make it fail the build once we’ve checked it doesn’t report false positives.
2. Fix the remaining label violations reported by axe and make the label rule fail the build.
3. Validate more screens with axe: today it only validates pages modelled as BasePage page objects, so screens modelled as BaseElements (e.g. the page creation form, the comments/history/attachments tabs) are never checked even though our functional tests open them.

What we lose: the DWG tests currently report no error on any page, so we don’t lose the detection of any existing problem. We only lose the detection of future regressions of the 2 rules above on pages that our functional tests don’t open (the DWG tests check all the non-hidden flavor pages and ~74 admin URLs, while axe checks the pages our Docker functional tests open). That gap shrinks as we add functional tests, since axe validates every page they visit, and Nu keeps covering alt, lang, doctype and headings on all pages.

Side note: we don’t pass the wcag22aa tag to axe, so target-size (the only WCAG 2.2 AA rule in axe) isn’t actually executed right now. I’ll fix that separately.

Here’s my +1. WDYT?

Thanks

[CharpentierLucas]

Thanks for checking this out!
These tests were definitely out of date and repetitive with our WCAG validation.

@vmassol

Remove the DWG validation completely: the DWG tests in the webstandards module, CustomDutchWebGuidelinesValidator with its exceptions, the hidden pages skip + class whitelist, and HTML5DutchWebGuidelinesValidator.

+1 for the main idea

@vmassol

1. Enable axe’s td-has-header rule, as a warning first, and make it fail the build once we’ve checked it doesn’t report false positives.

+1, so we don’t regress anywhere (assuming all DWG checked pages are checked by WCAG)

@vmassol

2. Fix the remaining label violations reported by axe and make the label rule fail the build.

+1

FYI there’s been quite a few new ones that popped up recently. Here’s the full list of tickets related to these fails:

• XWIKI-24672 - WCAG “Form elements must have labels” warnings on Extension Support / xclass object edit forms (I have a PR for it)
• XWIKI-25625 - The selection checkboxes of the Extension History have no label (unassigned)
• XWIKI-25626 - The invitation message field of the wiki users administration is not linked to its label (unassigned)
• XWIKI-25627 - RichTextAreaElement#dropFile leaves an unlabelled file input in the page (unassigned, test code, not product UI)
• WIKIEDITOR-108 - The text input of the syntax highlighted wiki editor has no label (unassigned, contrib extension, not in XS Docker tests)

@vmassol

3. Validate more screens with axe: today it only validates pages modelled as BasePage page objects, so screens modelled as BaseElements (e.g. the page creation form, the comments/history/attachments tabs) are never checked even though our functional tests open them.

-1, testing every base element will make the tests too heavy. AFAIR we wanted to limit it to BaseModal, that’s the idea in XWIKI-22260

@vmassol

Side note: we don’t pass the wcag22aa tag to axe, so target-size (the only WCAG 2.2 AA rule in axe) isn’t actually executed right now. I’ll fix that separately.

Feel free to ping me if you need help with that. I thought I added it back then but apparently I made a mistake and it’s not running as I’d have expected.

---

Thanks for doing this research and starting the discussion!

Lucas C.

[vmassol]

Thanks @CharpentierLucas!

Re your “assuming all DWG checked pages are checked by WCAG”: they’re not all checked. I compared the DWG targets with what the WCAG job validates, and about 15–20 screens with forms or tables are only covered by DWG today: the create page/space form, the comments/history/attachments tabs, XWikiSyntax, the AllDocs attachments views, some admin sections (user directory, user profile, syntaxes, wiki descriptor…) and the input panels. That’s why I had point 3.

Re 3 and the tests being too heavy: I measured the WCAG job (build 1421). It runs 2778 axe validations taking 1.11h, and most of them check the same UI again: 1190 are ViewPage on 1170 different test pages. Our cache key is URL + page object, and since each test uses its own pages, the cache almost never skips anything. If the key were action + page object, ~91% of the validations would be duplicates.

We can’t simply skip those, since the page content is what we want to check (6 warnings in that build were only on some pages). But we could:

• validate the full page the first time an action + page object combination is seen, then only the content area (#xwikicontent) for the next ones;
• use the action + page object key alone for the editor page objects.

That should cut the WCAG time a lot, and would make it affordable to also validate a few chosen non-modal screens (doc extra tabs, create form…), not all BaseElements. Agreed that modals (XWIKI-22260) are a good start. In any case all of this is a follow-up, not a blocker for the DWG removal.

Re wcag22aa and td-has-header: would you be ok to work on these 2, if you have the time? For context, I only looked at DWG while moving our tests from JUnit 4 to JUnit 5: I converted the webstandards tests, and I thought we could avoid keeping DWG and drop it altogether. I’m not planning to work on the WCAG side myself.

[CharpentierLucas]

@vmassol

Re wcag22aa and td-has-header: would you be ok to work on these 2, if you have the time? For context, I only looked at DWG while moving our tests from JUnit 4 to JUnit 5: I converted the webstandards tests, and I thought we could avoid keeping DWG and drop it altogether. I’m not planning to work on the WCAG side myself.

Created tickets for those tasks and assigned them to myself

• XWIKI-25641
• XWIKI-25642

---

@vmassol

We can’t simply skip those, since the page content is what we want to check (6 warnings in that build were only on some pages). But we could:

• validate the full page the first time an action + page object combination is seen, then only the content area (#xwikicontent) for the next ones;
• use the action + page object key alone for the editor page objects.

Sounds good to me.

@vmassol

In any case all of this is a follow-up, not a blocker for the DWG removal.