@ClementEXWiki and I are working on integrating the BlockNote rich text editor in XWiki, as a replacement of CKEditor. One important difference between BlockNote and CKEditor is the input / output syntax:
CKEditor takes annotated HTML as input and outputs annotated HTML. The transformations on the input HTML are minor, which means for most part the content looks the same in view and edit mode. The HTML rendering is done only server-side. The “annotations” are XML comments needed to reconstruct the wiki syntax on save (e.g. they mark the macro outut and hold the macro parameters and content).
BlockNote supports HTML and Markdown as input, but it’s internal format, and the recommended one, is BlockNote JSON. This means that even if you give HTML as input, it will be converted to the internal JSON format and then rendered back to HTML. It’s worth noting that BlockNote JSON is significantly less powerful than HTML. It’s actually more close to Markdown native capabilities than to HTML.
We had three options for BlockNote input:
Use HTML, by customizing their HTML to BlockNote JSON converter to support our “annotations” (XML comments). This would have allowed us to reuse our annotated HTML renderer / parser server-side, but it meant:
going from: XWiki syntax → HTML → BlockNote JSON, so with an additional step
maintaining a custom converter that could be broken by future upgrades of BlockNote
continue using XML comments as annotations which proved to cause problems when HTML is invalid (it’s easy for rendering macros to produce invalid HTML)
Use Markdown, but:
we would depend on a contributed XWiki extension
there’s no clean way to inject our “annotations”
we would go from XWiki syntax → Markdown → BlockNote JSON, so with an additional step
Use BlockNote JSON, by going directly from XWiki syntax, but we’d have to write a parser / renderer for it
In the end we decided to introduce a new syntax, that is equivalent to BlockNote JSON, so that we’re not tied to BlockNote. We called the new syntax UniAst (Universal Abstract Syntax Tree, right @ClementEXWiki ?) and it was implemented in Cristal. Next we implemented a parser / renderer server-side for this new UniAst syntax, to be able to go directly from XWiki syntax to UniAst / BlockNote JSON.
The reason I wrote this post is:
to let you know about the new syntax
to explain why it was needed
to list some of the problems we encountered while writing the parser / renderer
Regarding the last point, the main problem is that there is a significant mismatch between the XWiki syntax and the UniAst / BlockNote JSON syntax, meaning that there are things that you can do with XWiki syntax but not with UniAst and the other way around, things you can do with UniAst but not with XWiki syntax. The difficulty is avoiding data loss when converting between these two syntaxes. The following limitations of UniAst / BlockNote JSON will have to be addressed sooner or later:
Links
No way to indicate that the link label is generated.
Not possible to distinguish between freestanding and non-freestanding links and images. The effect is that attach:file.pdf becomes [[attach:file.pdf]] on save. We may need an additional link / image block property on UniAst side.
No way to indicate whether the target reference is typed or not (e.g. doc:Sandbox.WebHome vs. Sandbox.WebHome)
No dedicated support for typed resource references (e.g. url:https://www.xwiki.org , mailto:john@doe.com , path:/one/two/three).
No way to specify query string and anchor for internal link targets (for external targets they can be put in the URL).
No support for definition lists. The renderer is currently falling back on bulleted list.
Custom block parameters are not supported. This means that stuff like (% data-foo="bar" %) is lost.
No way to express inline verbatim sequence (i.e. escaping a group of characters rather than each character).
Tables
Tables can have only a single header row, at the top. Header column is not supported.
Table cells can have only inline content.
No support for group block (DIV).
Images
No image ID.
Image caption is plain text only.
Image styles are not supported (because custom block parameters are not supported).
Border (data-xwiki-image-style-border="true") and text wrapping (data-xwiki-image-style-text-wrap="true") are not supported.
Macros
In-place editable block-macros support only inline content.
Only the macro content can be edited in-place (i.e. we can’t edit macro parameters in-place, like the info message title).
Inline macros can be editable in-place, but they can contain only a single inline element.
No heading ID.
Formatting (bold, italic, etc.) is applied only directly to text, so we can’t apply formatting to a read-only block (i.e. macro output)
I’m really worried about this because we devised the XWiki Syntax to be close in capabilities to the HTML syntax so that we could have a one-to-one mapping between both and not loose data/features.
What I understand is that adopting BlockNote will result in data loss , meaning that things that can be expressed in XWiki Syntax won’t be expressable in BlockNote JSON and thus when after saving in the editor, they’ll be lost.
This is what you’ve listed at the end of post.
I don’t understand what’s the way forward to prevent data loss. Does it mean BlockNote needs to be improved to support all the features the XWiki Syntax currently supports and that it doesn’t support? If so, that certainly does look good since we have no guarantee that the BlockNote devs will work on that, nor on a timely manner, and our plan to replace CK4 by BlockNote by EOY 2026 is going to be compromised.
I hope I have misunderstood something and there’s a way forward
Yes, this is the current situation. We hope to reduce this data loss to zero by addressing the mentioned limitations.
Ideally yes, but I’m afraid we’ll have to implement / workaround some of those limitations ourselves. AFAIU we can extend the BlockNote JSON syntax with custom blocks so we should be able to add support for definition lists, groups (DIVs) or inline verbatim ourselves, if needed. But many of the limitations listed above are due to the lack of support for custom properties on standard blocks. If BlockNote doesn’t add support for this then the only alternative I know of is to replace all standard blocks with custom blocks, but this will increase significantly the development cost on our side.
AFAIK @mleduc and @ClementEXWiki have already reported to BlockNote most of the limitations we are facing. We need to see how much of our needs will fit their roadmap and how much will have to be done by us.
While there are some features that we can implement in a custom way, there are some we can’t currently. For instance, it’s impossible to nest custom blocks, which means you can’t put multiple paragraphs inside a list item, a list inside a table cell, etc.. There’s no way around that, it’s a hard limitation of BlockNote, and we have no idea how long it will take until it’s implemented on their side, as it seems pretty complicated (see Custom blocks cannot be nested in the editor · Issue #1540 · TypeCellOS/BlockNote · GitHub ).
We also have a hard limitation on some XWiki features, like indicating if a link has been generated. This would require attaching some kind of metadata to BlockNote’s links, which is not supported (see Attach metadata to blocks and inline contents · Issue #1840 · TypeCellOS/BlockNote · GitHub ). This also prevents us from attaching custom metadata to blocks like XWiki’s syntax allow us to. Heading IDs are the same.
A big problem with the differences between UniAst and XWiki’s syntax is the following dilemma:
Either we update UniAst to match everything XWiki’s syntax needs, and then adapt to BlockNote’s AST.
Or we don’t change UniAst until BlockNote supports the features, in which case it’s blocking from XWiki’s converter’s side.
The problem with approach n°1 is, what do we do when we encounter an unsupported feature with no way around it, such as attaching metadata to blocks? I personally think it’s a better trade-off to prevent this from happening by explicitly showing these limitations when writing a converter from XWiki’s side.
Yes I’m aware that it’s been discussed with them but we don’t really know how long it could take to implement it.
As a workaround, would it not be possible to double click inside a macro and open a dialog box to edit the macro, with a new blocknote editor to edit in wysiwyg the content of the macro? Leading potentially to several dialog boxes open at the same time if a macro is nested inside a macro inside a macro, etc.
