Change the code macro implementation

[vmassol]

Hi devs,

The syntax highlighting macro that we’re promoting by default in XS is currently the {{code}} macro.

This macro has several problems:

• Performance: It’s written in python and we have to use jython to execute it. This makes the macro very slow on the first usage (since jython is being loaded) and even for subsequent calls, it’s still a lot slower than it could be (and quite slow for large content to style).
• Features: it’s missing several features such as line numbering, copy the content with a button, etc.

OTOH it had a nice feature which made us keep it over the years: it’s executed server side. This was useful in the past for exporting to PDF for example since our PDF exporter was not supporting javascript code.

However since our PDF exporter now supports javascript code, we don’t have this limitation anymore.

I’m thus proposing that we “replace” the server-side implementation by a javascript one. We need to decide what “replace” means. What’s sure is that we need a solution that allows existing content using the code macro to continue being styled.

I think there are 2 options.

Option 1: New macro

• Move the current code macro to xwiki-contrib and add a new macro to XS.
• Main issue: users need to install the code macro. Note that since we support discovering macros, it’s not that hard as our UI will propose that.
• Secondary issue: The code name is quite good for a code macro. For new macros, we would need to find names for them. One idea is to name them after their technology so that we could support several and if we change it in the future, we won’t have again a problem of naming. Example: {{highlightjs}}, {{prismjs}}, etc.

Option 2: Same macro, different implementations

• Make the code macro a generic macro that can support several implementations (through Components)
• Offer an Admin UI screen to change the code macro implementation (dynamically list all supported implementation in the current xwiki instance)
• Move the current code macro to contrib but refactored as a Component for the new generic code macro
• Introduce a new Component for the new code macro that uses some existing js framework such as highlight.js, prism.js or some other.
• Main complexity: map the parameters of the existing code macro to the different implementations. Basically this means keeping existing pygments language ids and each impl would map them to their internal names. Same for other parameters (layout and source) but that doesn’t seem too hard.

WDYT?

To recap, there are 2 decisions to take:

1. Decide to move to a javascript-based solution
2. Choose a way to do that

Thanks

[mleduc]

+1 for using a front-end technology
+1 for option 2 (assuming the new implement does not lead to changes that could be considered regressions)

In addition to the admin UI, we could propose a parameter allowing to choose the rendering implem from the code macro params.

[vmassol]

@mleduc

In addition to the admin UI, we could propose a parameter allowing to choose the rendering implem from the code macro params.

I’m not sure this helps because it would be too cumbersome to have our users specify this parameter (I’m assuming you mean something like {{code type="pygments" ...}}. Also, I don’t think that most wiki instances need more than 1 type of code macro usage.

Last, if you we were to define a default type in the Admin UI, it would also not help since it would mean that users wouldn’t be able to move from one impl to another. For example, if you have a wiki currently using the code macro, the default would need to be pygments and thus you wouldn’t be able to use another impl.

PS: Maybe I misunderstood what you meant.

[tmortagne]

+1 for " Same macro, different implementations" but

@vmassol

Make the code macro a generic macro that can support several implementations (through Components)

Note that this concept already exist, what we mainly need to do is make the default hint (currently leading to a pygments based implementation) configurable. We might also need to pass a few more information than we currently do to HighlightParser#highlight (like the macro parameters and if it’s inline or not depending if it makes a difference with the new highlighter we choose).

@vmassol

Move the current code macro to contrib but refactored as a Component for the new generic code macro

Already a component, but needs to be moved to its own module and change its hint from default to “pygments”.

@vmassol

source

No reason for each highlighter to have its own implementation of source, so there is nothing to do for this one IMO. In the current architecture, the generic part of the code macro handlers the source part and pass the resulting content to the HighlightParser.

@vmassol

keeping existing pygments language ids and each impl would map them to their internal names

I honestly doubt this is really needed, unless the new implementation we choose uses a very special syntax for the names. The goal of most of these frameworks is to let the user enter a natural language name like “javascript”, “css”, etc.

There is a much more problematic retro compatibility problem to fix that you did not list : last time I checked, other frameworks were very very far from Pygments in term of number of supported languages and especially 2 that we use a lot: Velocity (I would not bet of that one being supported, it was supported by Pygments because we asked for it, so we might have to contribute a parser ourself) and Groovy (this one has more chance of being supported).

[vmassol]

@tmortagne

I honestly doubt this is really needed, unless the new implementation we choose uses a very special syntax for the names. The goal of most of this framework is to let the user enter natural language name like “javascript”, “css”, etc.

For example if you currently use {{code language="script"}} and you then switch, to, say, prism.js, you either won’t get the highlighting (since that labguage id doesn’t exist) or you’ll get an error.

@tmortagne

There is a much more problematic retro compatibility problem to fix that you did not list : last time I checked, other frameworks were very very far from Pygments in term of number of supported languages and especially 2 that we use a lot: Velocity (I would not bet of that one being supported, it was supported by Pygments because we asked for it, so we might have to contribute a parser ourself) and Groovy (this one has more chance of being supported).

This should not be a problem. For prism.js for example, they both exist, see Prism

Thanks

[tmortagne]

@vmassol

For prism.js for example, they both exist, see Prism

That’s one good point for prims choice, but it does not seem to be the case for highlightjs, so it’s an important criterion to have in mind.

@vmassol

For example if you currently use {{code language="script"}} and you then switch, to, say, prism.js, you either won’t get the highlighting (since that labguage id doesn’t exist) or you’ll get an error.

No idea what script language name correspond to in Pygments (cannot find it in Languages — Pygments).

Do you have a real use case that actually exist somewhere where the name used in the macro exist in prims but under a different name ? IMO, until we have one of those, I don’t think we need to do much on this front (and when we do, it’s probably more a mapping to put in the prims component, or even prims itself as a contribution, on a case by case basis than something generic).

[vmassol]

@tmortagne

No idea what script language name correspond to in Pygments (cannot find it in Languages — Pygments).

yes sorry, I meant shell which we use on xwiki.org. However, I see that it exists also in the prism.js language list.

Now we can be sure that there are mismatchs. For example in pygments bash, sh, ksh, zsh, shell are all synonyms but not in prism.js (where zsh doesn’t exist for example but the mapper would map it to shell).

Thanks

[mflorea]

@vmassol

Decide to move to a javascript-based solution

+1

@vmassol

Choose a way to do that

+1 for option 2

Thanks,
Marius

[Wardenburg]

@vmassol

Decide to move to a javascript-based solution

+1

@vmassol

Choose a way to do that

+1 for option 2

Looking forward to it, thanks !

[vmassol]

Closing this thread. So the conclusion is:

1. We agreed to move to a javascript-based syntax highlighting in XS
2. We agreed to keep the {{code}} macro and introduce an Admin UI to configure which implementation to use and by default t use the javascript-based one. And once done and working well, to move the pygments-based one to contrib.

Thanks everyone.

Next step will be to choose the javascript-based implementation to use but that’s for another thread.