Very slow response of 'MacroService' after upgrade to 13.9

Hello everybody,

we provide more than 10 XWIKI-Instances for our customers. After the announcement of XWIKI 13.9 we also want to upgrade these too.
The upgrade-process wasn’t really special in case of the Release-Notes. So almost everything was fine. After our internal testing phase we got one critical bug.

If I want to edit a page with the CKEditor, we want to display some macros so we click on the ‘cogwheel’-Icon in the toolbar to list all macros. And here is the Problem:

https://xxxx.xx/bin/get/CKEditor/MacroService?outputSyntax=plain&language=de&data=list&syntaxId=xwiki%2F2.1

This request takes more than 4 MINUTES to respond, so this function is almost useless for a production enviroment.

There will be also no log entry in the catalina.out file.

I hope somebody can help me to solve this.

What we have done so far:

  • SOLR reindexing all pages
  • Restarting all services
  • Clear browser cache
  • Reinstalled CKEditor

Additional Information:

We can confirm it on multiple Instances (at least 3)
13.0 → 13.9
13.5 → 13.9
13.7 → 13.9

  • Browser latest Versions(1.11.2021) -
    Firefox
    Chromium
    Chrome
    Edge

  • Hardware -
    8 GB RAM
    4 Cores
    120 GB Space

  • Software -
    Debian 10
    Tomcat9
    MariaDB (2 Instances) / PostgreSQL (1 Instance)
    Nginx Proxy

-XWIKI 13.9-
CKEditor 1.56

1 Like

This sounds very odd. I recently upgraded to XWiki 13.9 and out of curiosity upgraded CKEditor to 1.56 just now, and I cannot reproduce the behavior (Debian 10/Tomcat9/MariaDB).
When you trigger the long query to the .../bin/get/CKEditor/MacroService do you see any high load on the server?
You should be able to force a thread dump to be written to catalina.out with a command like:

   pkill -3 java

This might create lots of output, but the thread handling the request should have the URL of the request in the name (i.e. searching for MacroService should give you the stack trace for the thread busy collecting the macros).

Thanks for your Answer.

If I trigger the ‘MacroService’ there will be no high load on the Server.

In the thread dump i was able to find something for Macroservice like:
[2021-11-04 09:39:20] [info] "http-nio-8080-exec-93 - http://tomcat/bin/get/CKEditor/MacroService?outputSyntax=plain&language=de&data=list&syntaxId=xwiki%2F2.1" #1728 daemon prio=5 os_prio=0 cpu=27542.27ms elapsed=76342.26s tid=0x00007fa44803a800 nid=0xf93 runnable [0x00007fa412c4d000]

Maybe you find something interesting in the catalina.out File. I just uploaded this in our NextCloud. Feel free to use it :wink:
https://quis.appcenter.biz/nc2019/index.php/s/jTCYLLMKpQkTj9T

I see the code is busy somewhere in the glossary application, at least at the moment where you took the stack trace.

This is … somewhat unexpected. I do not think the glossary should intervene with a Json response, but that does not matter much. The interesting part is why it takes that long to do that, and not on “normal” wiki pages.

May I ask how many entries your glossary has? I tried with just a few manually created ones and could not notice any impact, neither on the output or the performance. (Also I have no idea yet why it should only affect XWiki 13.9 and not older versions - maybe it was just an accident that the glossary showed up at that point in time.)

From what I understand it’s a rendering transformation which mean it’s called every time wiki content is executed (here what is executed is the CKEditor.MacroService page). So I assume that glossary transformation is what is very slow but it’s indeed strange that it only happen for this page. Maybe the wiki content generated by this page is huge or contains stuff which are matched a lot by the Glossary transformation.

Thanks for spending our times at this problem.

And yes, we disabled the transformation of the glossary-entries in the xwiki.properties and the MacroService call respond nealy just-in-time. Thanks!!

Maybe its a kind of bug in the new glossary application. The usage of the transformation in-line were very often used by our customers. Mostly there are more than 150 entries in the glossary and the users ‘loved’ it. Hopefully there is a fix, so we can activate it in the future.

Note that the glossary extension is not supported by the xwiki dev team. You can check that on the extension page.

The solution for the perf issue is to use the glossary macro explicitly and not rely on the transformation.

1 Like

Re perf it’s mentioned at https://extensions.xwiki.org/xwiki/bin/view/Extension/Glossary/GlossaryApplication#HPrerequisites26InstallationInstructions

The contributors to this extension should improve the doc to make this more clear IMO.

Done: https://extensions.xwiki.org/xwiki/bin/view/Extension/Glossary/GlossaryApplication/#HPerformances :slight_smile: