Generic REST endpoints for writing an external WYSIWYG editor

Development
#1

Hi devs,

Context

We need a way for some external tool to use XWiki as a backend and get content from it, edit it locally and push the changes back to XWiki. An example of such a need is Cristal

What’s missing?

Right now you can ask XWiki to get the content in a given syntax (for example using the get action and outputSyntax query string parameter). For example you can ask for the HTML version of some doc content. However when you need to update the doc, you need to convert it back to the original syntax (same as what the XWiki WYSIWYG editor is doing), and we don’t have a good REST endpoint for this. And we cannot count on the external tool to have a good conversion feature.

Analysis

I see 3 options:

  • Option 1: Introduce a Conversion REST endpoint to convert from a syntax to another. In essence this means moving the current HTMLConverter.java, CKEditor.HTMLConverter, CKEditor.VelocityMacros, and WysiwygScriptService.java code into some more generic place and adding a proper REST endpoint.
  • Option 2: Add a new REST endpoint parameter to the existing PageResource endpoint so that you can tell it to convert the received content into a syntax before saving it.
  • Option 3: Introduce a more specific REST endpoint that extends PageResource and adds the conversion feature, like PageConvertResource.

Option 1:

  • Pro: Generic (XWiki can be used as a conversion service for other use cases)
  • Cons: Less performant than option 2 since Cristal would need 2 HTTP requests, one to perform the conversion and another one to perform the save. This means passing the full doc content back and forth lots of time.

Option 2:

  • Pro: More performant
  • Cons: Possibly a bit more contrived since the use case of converting to another syntax could feel something specific and not generic enough to be in the PageResource endpoint.

Option 3:

  • Pro: Doesn’t have the cons of option 2
  • Cons: A bit more code duplication than option 2 but not much
  • Cons: More narrow/specific than option 2
  • Pro: Performant

Conclusion

My take is that performance is an important aspect so I’d be tempted to go to option 3 (or 2), even if from an architecture POV I prefer option 1.

WDYT?

PS: I’m wondering if GraphQL would allow cleanly chaining operations without the HTTP round trips (ie do it in one pass, and yet still expose 2 REST endpoints: one for conversion and one for saving). I’d need to research this but anyway introducing GraphQL would require another large proposal and is not the goal ATM (yet it’s still interesting to know if GraphQL would provide a solution for this).

Thx

#2

If we go for Option 1, I would like to note that this would expose the conversion feature in a private wiki which might not be desirable. In general, performing conversions without checking that the user has edit right creates quite a big attack surface. If we could get rid of that, it would be quite a security improvement as any security vulnerability in wiki macros is exploitable through the converter.

Regarding option 2, and I think also 3, this might not be enough for a WYSIWYG editor, at least it isn’t for our current WYSIWYG editor. The reason for this is that the current converter is also used to get rendered content for macros that are inserted in the WYSIWYG editor (and possibly also renders other content like links or images), thereby performing an HTML to HTML conversion that simply executes macros. Further, at least the current editor also uses the converter when switching from/to the source view. In case this should also be supported in the new editor, it would also need the conversion.

Also, when the conversion happens in the PageResource, wouldn’t it also need to happen in the ObjectResource for rich text fields?

Further, there are special cases like change request where the content should not be saved directly but instead stored in a field.

All in all, I’m not convinced that option 2 or 3 are enough, and I think option 1 is required for actually implementing a WYSIWYG editor that provides features similar to the current CKEditor integration. However, we should at the very least require that the user has view right on the document before allowing a conversion to at least achieve a security level similar to the current conversion service. Ideally, we should require edit right but as the WYSIWYG editor is also used for comments and in change request, this doesn’t seem feasible.

What about an equivalent to CKEditor.MacroService in the REST API, do you plan a separate proposal for that?

#3

Yes, the current editor needs to:

  • convert to wiki syntax on save, but only when saving from the WYSIWYG mode, not from the Source mode
  • convert to wiki syntax (without save), when switching to Source
  • convert to annotated HTML when switching from Source to WYSIWYG mode
  • re-render the content (annotated HTML to wiki syntax and back to annotated HTML) when inserting a rendering macro
  • handle syntax change, which depends on the current mode:
    • Source mode: convert from the old wiki syntax to the new one
    • WYSIWYG mode: convert from annotated HTML to old wiki syntax, convert from old wiki syntax to new wiki syntax and finally convert from new wiki syntax to annotated HTML

The standalone WYSIWYG editor doesn’t need to convert from wiki syntax to annotated HTML on load because the textarea field that gets replaced is generated with annotated HTML from the server-side. The in-place editor needs this though and an external editor will need this also.

+1, I don’t think we can bind the conversion to the save operation.

Thanks,
Marius

#4

The REST endpoint would check that the user has view rights on the document. The conversion would be linked to a document and not just pure text to be converted - it’s important also for the execution since content execution results depends on the location. Macros (Transformations) would get executed during the conversion (same as when you view the doc though). So the REST endpoint would be name something like EntityConversion (and not just Conversion) and would take an EntityReference as a parameter.

I’m not 100% sure what you have in mind when you say that we should require edit rights when converting but this is currently doable on any document where you have only view rights through https://www.xwiki.org/xwiki/bin/view/Documentation/DevGuide/Architecture/URL%20Architecture/Standard%20URL%20Format/#HParameter:outputSyntaxandoutputSyntaxVersion

Thanks. I was just starting with the first need which is to convert some content from a syntax into another one before saving a page from an external system. I had not yet reviewed the other needs so thank you for starting the work for me :wink:

It’s good to know since that could impact the design decisions.

And thank you Marius for the full list. Will be helpful.

Yes that might also be a valid use case that will need to be handled but it’s further away since it would mean the ability for Cristal to edit xproperties and that’s not generic for all backends, so we would need something specific for the XWiki backend. Most likely, not a MVP use case. But good to have in mind.

ok, thanks. So option 1 seems the way to go even if it has performance consequences.

Yes ofc, that was the idea. I wouldn’t have imagined anything else since it would mean that anyone could execute arbitrary code on XWiki… :slight_smile:

Yes, I still need to review the rest. With this thread, I only wanted to discuss adding a document conversion service.

Thank you to both of you.