Best practice: ensure that page objects can always be used with translated UIs

Hi everyone,

I was about to open a proposal but I’m actually mixed with it so it’s more a brainstorming.
We have an important problem in XWiki that we all know about which is that we have no clear separation of the language of the UI and the language of the doc, leading to weird things like the UI suddenly switching to french when looking a french document in multilingual wiki.

One of the unexpected effect of that issue is that it’s currently very difficult to write UI tests which manipulates document translations: we currently do have many page objects that relies on displayed texts which are by nature translated. So we’d need to write the tests knowing in advance the translation and ensure the translation does not change.

So my first proposal was to add a new best practice, to ensure that PO would always use metadata to find elements and not displayed text. For example, currently the LiveData page objects only offers to find a cell by looking on the column name, instead of having a specific class name on the column to find it. Another example is the Diff page object which currently looks for “Page properties” to retrieve the diff.

I do think that it would make sense to fix those examples and to avoid in the future to rely on displayed text to find elements in new page objects APIs. And I think it would probably faster to fix those APIs now, than to properly fix the general problem we have in the mix of translated document / translated UI.

I’m mixed on adding it in the best practices, mainly because I think it’s a best practice we need because of a problem in XWiki. Now since I don’t have a good perspective on when this will be fixed, it might still be a good idea to write it down somewhere.

wdyt?

I don’t agree with this. One of the main goals of the functional tests is to replicate the user behavior in order to catch problems that an user might have. If we always use hidden meta data to locate UI elements then we won’t catch problems like:

  • missing label (no text is displayed)
  • missing translation (the translation key is shown instead)
  • wrong translation key used (an unexpected label is displayed)
  • wrong parameters passed to translation (e.g. for zero-one-many)

I’m not saying we should always use the displayed text. Sometime is makes more sense to use the hidden meta data. But always using the hidden meta data is definitely not a good rule for me.

Thanks,
Marius

I think I agree with Marius here and my first reaction is that the UI tests should assert what’s displayed on the screen so that the test really validates the UI.

How frequently do we have to fix functional tests because the translation changed? I don’t think I remember that it happens frequently enough to overcome the benefits, does it?

I don’t agree and as Vincent said:

The UI tests should assert there’s no problem with what’s displayed and I agree with both of you on that. That’s the job of the test by making some assertion checks. Not the job of the page objects code, when looking for some elements of the UI to retrieve.

It does not happen because we’re executing tests only with english locale and we’re not supposed to change the wording without deprecating the translations, in which case we do change the related page objects too and generally it’s not the only refactoring (we had to do that for Notifications settings AFAIR for example).
Now as I said my main UC here is to be able to support testing using a different locale than english: e.g. try to write a UI tests for history that manipulates a translation and you’ll see it’s very difficult right now.

I agree with @surli, so +1 for the proposed best practice.

I think what we should try implementing instead to ensure that we don’t break the UI are screenshot-based comparisons. Basically, we should take screenshots of certain UI parts at certain steps and compare these screenshots to known-good screenshots and fail the test when the screenshot changes. It should, however, be easily possible to update the known-good screenshot for a branch in case the change was intentional (or have several good screenshots also to allow some flickering). Basically, that’s what Percy offers I think. Some of these screenshots could also be used for documentation. I think implementing at least parts of this could be a Google Summer of Code project.

1 Like

I get your point and it indeed makes sense. Now, this isn’t exactly (fully) what you propose. There are two (equally important) parts:

  • page objects should “always use metadata to find elements and not displayed text” → what you proposed
  • page objects should always provide an API to assert the displayed text

If we have both in the best practice then I’m +1. My only worry is that the developers will apply the first rule, because you need it to write the test, and forget about the second rule, i.e. forget to assert the displayed text in the test (e.g. forget to assert that the button label is “Create”).

Thanks,
Marius

I’m fine with having both rules in best practices.

To be honest the main problem of that proposed best practice is that the only way to enforce it is by reviewing our code: I don’t see any simple automation we could put in place to guarantee this. So I don’t think we will always systematically comply to it if we adopt it, but at least it gives the direction we want.

This is something we’ve discussed briefly in the past and I’d be very interested that we do some POC to try this out. The first step would be to find an open source library that we can use inside our build (without calling any external service). Then we’ll need to learn it and see how easy it is to use (the screenshots should be taken automatically when training/adjusting the tool).

This would allow us to catch more regressions.

I’d prefer that one of us does it since this is a core dev thing. I’m volunteering if nobody else is interested/has the time. Even if someone else does it, I’m interested to participate.

PS: List to review: GitHub - mojoaxel/awesome-regression-testing: 🕶️ A curated list of resources around the topic: visual regression testing

For ex: