Incessant timeouts; slow performance overall

Help / Discuss
#1

I have been experiencing intermittent CPU spikes and outages using XWiki 10.x series. I believe this behavior started at version 10 but I cannot be 100% certain.

The setup is one AWS t2.medium instance (2 vCPUs, 4GiB RAM). This instance runs nginx (reverse proxy), xwiki, and mysql. I am using the packages provided for Ubuntu. Currently on version 10.7.

Overall this setup is slow. During normal operation, java will be using a minimal amount of CPU and responses return in a few seconds. The regressive behavior is java CPU usage spikes up to 177%+, and seemingly never comes down. nginx starts reporting 504 gateway timed out errors. Running “systemctl restart tomcat8” will get things running again, but the issue can re-appear at any time thereafter. Constantly restarting tomcat8 is not ideal since it takes so long for xwiki to get started again (it seemingly waits for a first http connection and then spends a lot of time caching stuff to memory? Is my guess of what is happening there?)

The functionality of XWiki is terrific but operationally it is a bit of a nightmare. I do not know where I am supposed to be looking when things go wrong. Is there a recommended configuration I can try? A certain vm size or deployment model I can follow? I was using a t2.small (1 vCPU), upgrading to t2.medium helped tremendously at first. However, within a few weeks the site overall felt slower and the hanging CPU issue reasserted itself again. As you can probably tell, we are trying to keep costs down but it is important that the system run consistently.

edit: to be clear, the usage of the wiki is extremely low. We have just over 20 users, many of whom are not very technical. Only about 5 real content producers, who might add content once every couple weeks. Most of the time, this system is idling. It is just as soon as you try to use it, you are almost guaranteed to hit problems fairly quickly, or return to the site to find it not responding.

Any help would be very welcome. Thanks.

#2

When you get such an endless spike would be great to take a thread dump so that we try to figure out where all that power is going.

Note: we do have identified performances issues with the new notification system in 10.x on which we are working on and hope to improve a lot in 10.8.

1 Like
#3

I’m sorry to hear but this is most likely caused by your setup since we don’t observe this on xwiki installs in general (not to that level).

Note that it’s not the first time I’m hearing you complain about performance so I really think you have a setup issue since we don’t hear this from other users in general and we don’t see the issues on our installs. At least not to the level you’re observing.

To be fair, we know that we have reduced performances since XWiki 8.x, see https://test.xwiki.org/xwiki/bin/view/Performances/Jetty%20HSQLDB%20single%20wiki%20845%20to%2099

We’re currently working to get back to 8.x speed.

We had https://www.xwiki.org/xwiki/bin/view/Documentation/AdminGuide/Performances/#HMemory

I’ve now added https://www.xwiki.org/xwiki/bin/view/Documentation/AdminGuide/Performances/#HSizing

Hope it helps

When you have CPU issues, you should take a thread dump. This will tell you what part of the software is being blocked. You can share it with us and we can try to help you analyze it.

Thanks!

1 Like
#4

Thank you! This is helpful. I have run through the performance guide before, but I’ll go through it again as perhaps things have changed. Thank you for adding a sizing guide, that is very helpful.

Is there a step-by-step guide for gathering thread dump information? The last time this came up, the suggested method seemed difficult to execute. Too difficult, to be honest. However, I am reaching a level of desperation where I would be willing to spend much more time on it.

1 Like
#5

This is easy. You need to execute jstack <pid> or jstack -F <pid>, where pid is the java process id for the JVM in which XWiki is running.

3 Likes
#6

This is easy. You need to execute jstack or jstack -F .

…wuuuuut. Last thread it was this super-complicated process. In any case, that is great, I am on it.

#7

Maybe you can setup monitoring as described here with javamelody

Which adds more insight on performance and IIRC thread dumps on a button press.

@tmortagne suggested glowroot somewhere else as an alternative.

2 Likes
#8

Excellent, I have glowroot running now. Thanks for the tip!

I also noticed the recommended performance settings changed since I originally installed back around version 7, so I have updated those. One note, apparently MaxPermSize is deprecated as of Java 1.8.

At least I feel like I am moving forward. Thanks everyone on this thread for the help.

#9

Hi, I wanted to close out this thread. After applying the new recommendations on the Performance wiki page, things are running very well. The dual-core installation hasn’t crashed at all, and the single-core installation has only crashed once since I posted. So that is great! Thanks for the help.

Given how important those Performance settings are, specifically https://www.xwiki.org/xwiki/bin/view/Documentation/AdminGuide/Performances/#HMemory, it would be nice if there were posts here when they are updated, or it would be nice if I could subscribe to that page to get notifications when it is changed.

To be clear, I was running settings from when I originally installed (around version 7) and by version 10, using the old settings somehow contributed to the problems described above; thus I think since they change, advertising those changes is pretty important.

Thanks!

#10

You can watch any page on xwiki.org, go to the bell on the top right corner and enable it for this page and then make sure you have a configured “Notifier” in the section Watchlist of your user profile.

#11

crw, I’d also recommend installing javamelody so you can easily monitor your system and get a good idea of what your normal performance is.

The overhead is light, and the insights are invaluable. This tool will help you identify problems and also help you figure out where those problems might be.

#12

Thank you, I looked for the notification ability but didn’t find it (I am default subscribed to all changes everywhere on the wikis I run, so it isn’t something I think about often).

edit: and, I wasn’t logged in so I couldn’t see the Bell icon; I made the assumption that functionality was turned off on that wiki somehow. As a UX thing, it might be good to have that greyed out with a message to log in to configure to be alerted to changes.

#13

Thank you, I’ll definitely check this out!

#14

You should try glowroot (https://glowroot.org/) which provides even better information that javamelody. FYI we were using javamelody on xwiki.org before and we switched to glowroot.

@tmortagne I think we forgot to update the doc https://www.xwiki.org/xwiki/bin/view/Documentation/AdminGuide/Monitoring It would be nice to document glowroot there.

#15

For your use case maybe but not generically. XWiki can be used for lots of use cases, including a web site use case and it doesn’t make sense to show a greyed out icon for this use case where users are not supposed to register/have a login.

#16

Cool! Glad to hear that :slight_smile:

Could you remind us what “new” recommendations you’re talking about?

Thx

#17

On Ubuntu, my /etc/defaults/tomcat8 changed from:

JAVA_OPTS="-Djava.awt.headless=true -Xmx512m -XX:MaxPermSize=196m -XX:+UseParallelGC -XX:MaxGCPauseMillis=100 -Dfile.encoding=utf-8"

To:

JAVA_OPTS="-server -Xms800m -Xmx1600m -XX:MaxPermSize=222m -Dfile.encoding=utf-8 -Djava.awt.headless=true -XX:+UseParallelGC -XX:MaxGCPauseMillis=100

Those changes seem to have been the difference between crashing / hanging all the time and not. So, relatively important changes.

#18

Ok I misunderstood you then. You mentioned “new recommendations” but these are very old one (since the beginning of XWiki actually, unless we’re talking about different things). So I don’t understand the part about “it would be nice if there were posts here when they are updated, or it would be nice if I could subscribe to that page to get notifications when it is changed.”. Do you mean this generically and not specifically for the memory recommendations?

And you’re right the memory settings are important. They’re so important that they are part of our installation prerequisites: https://www.xwiki.org/xwiki/bin/view/Documentation/AdminGuide/Installation/#HHardwareandSoftwarerequirements

#19

Maybe it will be easier to see the changes if I re-order things (old first, new second):

JAVA_OPTS="-Xmx512m -XX:MaxPermSize=196m -XX:MaxGCPauseMillis=100 -XX:+UseParallelGC -Djava.awt.headless=true -Dfile.encoding=utf-8"

JAVA_OPTS=" -server -Xms800m -Xmx1600m -XX:MaxPermSize=222m -XX:MaxGCPauseMillis=100 -XX:+UseParallelGC -Djava.awt.headless=true -Dfile.encoding=utf-8

Please take a moment to look closely at the two recommendations. Two directives were added to the beginning, and two values were changed. That is a change. You changed your recommendation. Thus becoming a new recommendation.

You may not consider it a very dramatic change, and yet it was the difference between a functioning server.

#20

@crw Could you please provide a link to where you see this and a URL with the history diff so that I can understand what you mean? Thanks!

For the diff you go here for example: https://www.xwiki.org/xwiki/bin/view/Documentation/AdminGuide/Performances/#History and you pick from and to values and then click “compare selected values”. This will give you a URL that you can paste here.

Thx