Integration tests and CKEditor

Development
#21

Yes. But there are just some problems with that:

  1. it’s slower and we don’t have more agents
  2. it’s more fragile (more flickers) to use the WYSIWYG editor
  3. the minimum thing that must work in xwiki platform is the wiki editor. The wysiwyg editor is a bonus. Said differently the minimal flavor should not include the WYSIWYG editor, i.e. XWiki can be operated without a WYSIWYG editor. So from a POV of platform it makes sense to test with the wiki editor. From the POV of the XS flavor, it makes less sense but we don’t want to rerun all tests either with the WYSIWYG editor because we don’t have the agent power to do that. Func tests already take too long.

In any case all I’m looking after is some unequivocal rules so that a dev will know for sure if he has to write tests with the WYSIWYG editor not. My preference still goes to not use the WYSIWYG by default and exceptionally use it (it needs to be justified in comment when it’s used), and the dev should think hard first about how that test could be put in the CKEditor module instead.

#22

I agree that we have a problem with our CI and agent power, now I’m really afraid that we loose on usability by taking this perspective. I don’t agree saying that the wysiwyg editor is a bonus, I’m not even sure to agree that it shouldn’t be included in the minimal flavor now. Maybe it’s actually where the debate should took place.

Basically my reasoning is the following:

  1. we need to improve globally perceived usability in XWiki
  2. WYSIWYG editor is a key feature of usability
  3. we push towards more and more automated tests
  4. automated tests are not using WISYWIG editor
  5. there is more and more risk to miss regression linked to side-effect on WISYWIG editor
  6. there is more and more risk to decrease the perceived usability on that side

So here I’m more looking for a long-term perspective for it, do we want to keep considering it as an optional feature of XWiki, or is it on the contrary a key feature that cannot be left aside and that should be considered as a core feature?

#23

The rationale is that it’s a very important feature of XS and thats why it must be well tested in the CKEditor module itself.

There’s very little reasons it would fail outside of it. And the cost of testing with all the other modules is too high for the small risk of failure (which will happen for sure).

So you’re right ofc that testing everything with the WYSIWYG editor will reveal more problems, even if 99% of the cases are useless. My issue is that we’d pay a too high price for the 1%.

A good example could be the “inline macro editing” feature we added a while back. Where should tests for it be? First, in the CKEditor project for sure since we provide a contract for macros (they need to use the proper HTML) and if the macros follow it then the WYSIWYG should work fine. Now we probably also need to add tests in the macros themselves to make sure they follow the contract. But we don’t need to add integration tests for all editable macros because that’d take a lot of time to execute and testing that they follow the contract (testing the generated HTML) is enough to guarantee a good level of confidence (even if there can be problems).

Basically my POV we should have the max number of unit tests and the min number of functional tests because we won’t scale otherwise (we already don’t). It also makes it simpler and faster to debug.

WDYT?

@mflorea, @tmortagne you haven’t express your POV recently and it’s discussion between Simon and myself lately :wink: I’d be curious to hear your POV.

Manual tests vs Automated tests analysis