Cristal's Universal AST

Development
1

This forum is not a proposal ; rather an explanation of the current design we use in Cristal for AST handling.

XS uses a format called XDom to represent document and allow macro manipulations. In Cristal, we use a different format called Universal AST, or UniAst for short.

Its fundamental ideas are:

  • To be simple in its definition, fitting in a single file
  • To be designed for hard type safety and soundness - it should be hard to misuse its types and provide invalid states
  • To follow current editor capabilities. Currently, there is no mechanism to avoid data loss when editing a document with unsupported features inside editors like BlockNote. UniAst currently prevents representing most unsupported states.

The reason to not simply map XDom’s types to TypeScript are:

  • Converting Java types to TypeScript is not easy as the typesystems are vastly different
  • The resulting types are not necessarily easy to manipulate from TypeScript due to their unusual layout

Still, UniAst is inspired by XDom, and is designed just like it to be editor-agnostic, syntax-agnostic, and format-agnostic (e.g. can be converted to HTML)

Converting XDom to UniAst should be relatively easy, and UniAst should roughtly have the same capabilities than XDom.

The API is defined entirely in https://www.xwiki.org/xwiki/bin/view/documentation/xs/dev/front-end/rendering/api/. As you can see on that page, the full type list is very concise.

Please ask if you have any questions or remarks regarding the API’s design, as it is very much a work in progress and can absolutely be changed :slight_smile:

2

Thx Clement for this explanation.

Note that it’s XDOM (not XDom). It stands for “XWiki Document Object Model”. Also it should probably be UniAST (for Universal Abstract Syntax Tree, I guess). In XWiki, at least in Java, we decided early on that acronyms/abbreviations must be in upper case (URL and not Url for ex).

Also note that this topic is not about Cristal but about XWiki since we’re talking about the code in the XWiki codebase (and there’s no Cristal there). I understand that this originated from Cristal standalone and there might have been a need there for a client-side rendering API but we need to review this in the context of XWikI/XS.

What I’d like to understand is:

  1. Why we have the need to introduce a client-side rendering API in XS. That’s not only a lot of work to get to the same level as the server-side rendering API (which took us a lot of years to develop), but it’s even more work to maintain 2 rendering APIs (without one being automatically generated by the other for ex). Even with Blocknote, why couldn’t we do as we did for CKEditor and call a server-side REST API to perform the syntax conversions and thus maintain a single API?

  1. Are you sure regarding the “roughly same capabilities”? When I see https://www.xwiki.org/xwiki/bin/view/documentation/xs/dev/front-end/rendering/api/ I see a lot of variations and features supported in XDOM that are not supported in UniAST. Also, that’s only a small part of the rendering API. It’s missing Transformations and Macro for ex, is there an architecture page presenting the full rendering API? Have we really tried to have the same features than XDOM when it was designed? For ex, I don’t see the concept of WordBlock or SpaceBlock or SpecialCharacterBlock (ie it doesn’t seem to operate at the character level). And ofc there several block types that I see in XDOM but not in UniAST.

  2. When it was designed, did you check the known limitations of the XDOM APIs that we noticed over the years so that we don’t repeat the same mistakes and can provide a better API? I don’t remember any discussion/design around this. Globally, a rendering api is a very complex API that requires discussions and reviews. I’m surprised that it wasn’t discussed/proposed on the forum. I think we need that or we’ll pay the price down the line. But first, we need to be sure we need such an API (see point 1).

Sorry for these questions, especially as they come late probably but I believe it’s important that we answer them collectively now, before we push further.

Thanks

1 Like
3

Hi, I agree with the naming, we should go with UniAST to be more consistent with XS.

To answer your questions:

  1. The client-side API is required in Cristal, but not necessarily useful in XS. We may want to integrate Cristal in projects where there isn’t a server to render macros, and thus macros should work entirely client-side.
  2. Good point, there are capabilities that are explicitly missing in UniAST. For instance, macros cannot transform their parent content, as this is a non-goal we decided when designing the new macros system. You can find the full API here: https://cristal.xwiki.org/xwiki/bin/view/Documentation/Extensions/Macros. Macros currently can’t be syntax-specific, but this is something we might want to come up with in the future ; note that it’s not related to UniAST itself, as macros are a separate concept. Also, some features are explicitly missing too (such as character-level features) as, as I stated, features that are not yet available in the target WYSIWYG editors are not represented in UniAST for now.
  3. I only discussed with @mleduc, as I wasn’t familiar with XDOM at all at the time UniAST was designed. Discussing further with @mflorea, some shortcomings appeared and it’s definitely planned to take a closer look at XDOM to fill in some missing capabilities in UniAST.

There should definitely have been some discussion around it on the forum, alas I was unfamiliar with this process when I started designing UniAST. This is on me, and why I write proposals for all major changes now (e.g. removing client-side macros AST).

4

Yes but that’s a lot of work and it’ll take years to reach the same level as what we have on the server-side (supporting all syntaxes, etc). So it’ll probably remain a simplified version with less features and less syntaxes supported.

I don’t think we want that in XS, and it would be a pity to use something less good than what we currently have, in term of features/syntaxes.

Thus I have the feeling that we’d need to move that code to Cristal standalone (i.e. outside of the XWiki code base) and introduce a feature in Cristal (if that doesn’t already exist), so that when a backend has a rendering service, it’s used instead of the built-in Cristal client-side rendering.

And in XWiki, only use our server-side rendering. And thus, develop a parser and renderer for the BlockNote JSON, in Java.

WDYT?

Thanks

5

The two main use cases are:

  • Support for backends with no server-side rendering API (e.g., nextcloud, github, file system)
  • Support for offline mode (since access to the serve is not an option).

XWiki has a server-side rendering API and I don’t think we plan to have an offline mode anytime soon.
Therefore, going through UniAST as an intermediary step that we then convert to Blocknote DOM is not strictly mandatory.

6

As Manuel stated, we need client-side rendering for macros because of the lack of a server in some platforms.

The only question is: do we keep UniAST, as it is, or do we replace it with XDOM? I’m not sure mapping to TypeScript would give a very ergonomic JSON API.

7

yes but only for Cristal standalone (with a reduced rendering feature set), not for XS.

I don’t know enough the client-side code but I’d expect that if we have a REST API to get rendering results (text in, text out), we don’t need much on the client side, just to insert the returned content where it’s needed (HTML, source view/wiki editor, etc). What am I missing?

8

yes but only for Cristal standalone (with a reduced rendering feature set), not for XS.

Indeed.

I don’t know enough the client-side code but I’d expect that if we have a REST API to get rendering results (text in, text out), we don’t need much on the client side, just to insert the returned content where it’s needed (HTML, source view/wiki editor, etc). What am I missing?

Currently that’s how it works for XWiki macros: they are rendered server-side, returning HTML that’s inserted inside the BlockNote editor.