Handling of multiple empty lines in Markdown

Hello all,

Following CRISTALNC-37 - Empty lines are not saved opened by @paulinebessoles I’d like to start a discussion on the handling of multiple empty lines in Markdown.

To explain the problem quickly, a user edit a page and adds a dozen lines between two paragraphs, then save.
The user expects the large gap between two paragraphs to be persisted. But, it is currently not the case and the gap between the two paragraphs is simply the standard spacing between two paragraphs.

In technical term, this is explained easily. Markdown specific that “Multiple blank lines between paragraphs have no effect” (CommonMark Spec).
But, on the other hand, this is a not aligned with the notion of WYSIWYG. The user did the intentional choice of adding a large gap and this was reflecting in its editing experience.

As it can impact some decisions, the technical scope includes:

  • Two Markdown parsers and serializers:
  • Two editors: CKEditor 4 and Blocknote.
  • The external sources where the markdown can be viewed through other viewers and editors. I’m mainly thinking of Cristal’s backends such as Nextcloud, Github, etc.

Also, not that the XWiki syntax support larger gap and intermediate empty paragraphs (<div class="wikimodel-emptyline"></div> when rendered in HTML).

Option 1 - do nothing

We consider that the possibility to add spaces between paragraphs is giving too much formatting control to users and that this shouldn’t be persisted.

Cons:

  • Not WYSIWYG
  • Contradict what’s done with xwiki/2.1

Pros:

  • Easy as there is nothing to do (close CRISTALNC-37 as won’t fix)

Option 2 - restrict the editor

Restrict the editor and disallow adding several empty lines in a row

Pros:

  • WYSIWYG

Cons:

  • Not sure about the feasibility

Option 3 - become blank lines aware

Pros:

  • I expect this to be an easy parser/serializer change
  • WYSIWYG

Cons:

  • Not necessarily supported by all backends, but I think this is acceptable, but there is a risk that editing the markdown with another editor removes the blank lines

Option 4 - explicitly mark blank lines

Use an explicit separator to mark blank lines (e.g., &nbsp; or <br />)/.

For instance:

A

&nbsp;

B

Pros: Same as 3 - easy implementation + WYSIWYG

Cons:

  • Probably better support for other backends. Except if the separator is not handled, which is a reasonable risk. We can also make the separator configurable based on the backend if needed.

Conclusion

+1 for option 4 as it feels the most aligned with what we already have.

Note that the XWiki Markdown extension supports blank lines. And we cannot not have it since it’s part of our XDOM spec (and CTS). We could have a different impl in Cristal Standalone and XWiki but that’s not a good idea, so +1 to support empty new lines in Cristal standalone too.

Regarding how it’s implemented, idk. From a user POV, it’s some internal details. What’s important is that a round trip works (and no empty lines). So I guess that’s 3 or 4.

Thx

It does not.

While editing:

Once saved:

oh you’re right: syntax-markdown/syntax-markdown-commonmark12/src/test/resources/markdown12/config.properties at cc2b752e5a0f0e80b9a9300554ef732aeea02cba · xwiki-contrib/syntax-markdown · GitHub

I was pretty sure I had implemented it. I also checked jira and didn’t see an open issue for it.

So it means we have a bug/limitation right now with that extension because if you convert some xwiki/2.1 syntax into markdown/1.2, we’re going to loose empty lines and that’s not good. We need to fix that eventually.

Yes, I did not develop too much on the implementation details before an option is validated.
But since option 3/4 seems to be considered, the implementation impact both mentioned parsers and serializers:

You definitely can have more than 1 empty lines with wysiwyg editor. Our users do this often. (I’m not a big fan of it.) I prefer a macro creating a div that is only styling a gap.

What ever you decide don’t use something like &nbsp. This is bad for accessibility and you won’t get bitv (german wcag) approved.

Thanks, that’s really good to know!
Note that even if those blanks lines are serialized as &nbsp; in Markdown, they will be presented as <div class="wikimodel-emptyline"></div> when rendered as HTML in a page inside XWiki.
But this might not be true for other platforms, so I’ll keep that in mind.

Interresting discussion about GitHub MD processing new-lines - Github markdown that respects newlines - Stack Overflow

and maybe this might help too - AsciiDoc Writer’s Guide | Asciidoctor

quick off-topic note about CKEditor4 EoL - CKEditor support status and future - Development - XWiki Forum (latest OSS version with security flaws patched in “commercial version”)

Hi,

I think the first step is to add support for empty lines at AST level. XDOM (server-side) already supports this. We probably need to add support for empty line in UniAst (client-side).

Then we need to detect empty lines in the WYSIWYG editor and treat them as such at the AST level (e.g. when converting from BlockNote to UniAst). For XWiki Standard, where we use the BlockNote syntax directly, the server-side BlockNote parser uses this logic to detect empty lines xwiki-rendering/xwiki-rendering-syntaxes/xwiki-rendering-syntax-blocknote/src/main/java/org/xwiki/rendering/internal/parser/blocknote/blocks/ParagraphBlockParser.java at master · xwiki/xwiki-rendering · GitHub .

Finally, we need to render empty lines from the AST to the target syntax (e.g. Markdown). This depends very much on target syntax, and whether that syntax can represent somehow empty lines. XWiki syntax supports this (and empty lines generated from BlockNote are currently preserved).

It’s not clear for me the difference between option 3 and 4. How do you represent empty lines with option 3? By inventing a dedicated syntax for empty lines (i.e. a Markdown extension)? I’m not fond of this. I’m more in favor of using standard Markdown syntax.

Thanks,
Marius

1 Like

The thing is, there is no Markdown standard to represent empty lines between paragraphs.
Option 3: make our Markdown parsers blank space aware (we just have to make sure to persist these blank lines when serializing).
Option 4: same but we use a dedicate HTML element (this is relatively standard, many Markdown implementation support HTML elements)

For us, they are almost equivalent. The main difference is when we start considering the possibility for the markdown files to be view and edited outside Cristal, which can happen easily when Cristal is installed as a Nextcloud extension for instance.

It’s not exactly true. The spec says that HTML is valid. So using <br> is a supported markdown syntax, see CommonMark Spec

Actually there are 2 other ways:

  1. Two (or more) trailing spaces at the end of a line:
    first line··
    second line
    
    (the ·· are literal spaces — invisible and error-prone, which is why many people dislike it)
  2. A backslash at the end of a line (this is the most robust/visible option):
    first line\
    second line