Parsers and serializers refactoring

Cristal
1

Hello all,

Currently the parser/serializer packages of Cristal are not really organized and are based on multiple parsing libraries, leading to code duplication and more expensive development costs for maintenance and improvement.

More precisely, we currently support conversion from/to: markdown, Blocknote DOM through an intermediate abstract format we call UniAST, and HTML.

From/To HTML Markdown BlocknoteDOM
HTML N/A not implemented not implemented
Markdown markdown-it N/A UniAST + remark
BlocknoteDOM not implemented UniAST + remark N/A

UnitAST supports:

  • parsing markdown to UniAST
  • parsing BlocknoteDOM to UniAST
  • rendering BlocknoteDOM from UniAST
  • rendering markdown from UniAST

markdown-it

  • parsing markdown to HTML

The main issue is that we have two mechanisms to parse Markdown.

markdown-it only targeting Markdown to HTML, I suggest to drop it.
Instead, I propose to write an UnitAST to HTML serializer. Consequently, Markdown to HTML conversion would go through the intermediate UniAST representation.
Then, having a single reusable parser for markdown, I will be able to modularily parse links differently based on the backend supported syntax (e.g., XWiki uses flexmark internal links [[...]] while Nextcloud or GitHub doesn’t and needs to rely on other mechanisms for internal links.

In addition, parsers and serializers are currently spread across inconsistent modules. For instance, parsers from/to BlocknoteDOM are located in @xwiki/cristal-editors-blocknote-headless.
I propose to:

  1. remove @xwiki/cristal-uniast
  2. move the api parts of @xwiki/cristal-uniast to @xwiki/cristal-uniast-api
  3. declare individual packages for @xwiki/cristal-uniast-${syntax}-${parser\|serialized} , namely @xwiki/cristal-uniast-markdown-parser, @xwiki/cristal-uniast-markdown-serializer, @xwiki/cristal-uniast-blocknote-dom-serializer, @xwiki/cristal-uniast-blocknote-dom-parser
  4. introduce the new @xwiki/cristal-uniast-html-serializer following the same naming strategy

I’ll likely have to introduce a cristal-uniast-markdown-parser specific API to allow extensions to provide their link parsing strategy and to let backends declare which one they need, but it’s going to be for another proposal.
Having this structure allows us to be more precise in what is imported based on the backend configuration when more syntaxes are supported. For instance, an XWiki instance only supporting the wiki syntax shouldn’t have to import the markdown parser and serializers.

In summary:

  • do you agree to stop using markdown-it and move everything to remark?
  • implement an UniAST to HTML serializer?
  • do you agree with the proposed packages split?
  • drop the unused and unmaintained @xwiki/cristal-rendering, allowing us to also drop the dependency to marked

Thanks!