I have been experiencing intermittent CPU spikes and outages using XWiki 10.x series. I believe this behavior started at version 10 but I cannot be 100% certain.
The setup is one AWS t2.medium instance (2 vCPUs, 4GiB RAM). This instance runs nginx (reverse proxy), xwiki, and mysql. I am using the packages provided for Ubuntu. Currently on version 10.7.
Overall this setup is slow. During normal operation, java will be using a minimal amount of CPU and responses return in a few seconds. The regressive behavior is java CPU usage spikes up to 177%+, and seemingly never comes down. nginx starts reporting 504 gateway timed out errors. Running “systemctl restart tomcat8” will get things running again, but the issue can re-appear at any time thereafter. Constantly restarting tomcat8 is not ideal since it takes so long for xwiki to get started again (it seemingly waits for a first http connection and then spends a lot of time caching stuff to memory? Is my guess of what is happening there?)
The functionality of XWiki is terrific but operationally it is a bit of a nightmare. I do not know where I am supposed to be looking when things go wrong. Is there a recommended configuration I can try? A certain vm size or deployment model I can follow? I was using a t2.small (1 vCPU), upgrading to t2.medium helped tremendously at first. However, within a few weeks the site overall felt slower and the hanging CPU issue reasserted itself again. As you can probably tell, we are trying to keep costs down but it is important that the system run consistently.
edit: to be clear, the usage of the wiki is extremely low. We have just over 20 users, many of whom are not very technical. Only about 5 real content producers, who might add content once every couple weeks. Most of the time, this system is idling. It is just as soon as you try to use it, you are almost guaranteed to hit problems fairly quickly, or return to the site to find it not responding.
I’m sorry to hear but this is most likely caused by your setup since we don’t observe this on xwiki installs in general (not to that level).
Note that it’s not the first time I’m hearing you complain about performance so I really think you have a setup issue since we don’t hear this from other users in general and we don’t see the issues on our installs. At least not to the level you’re observing.
When you have CPU issues, you should take a thread dump. This will tell you what part of the software is being blocked. You can share it with us and we can try to help you analyze it.
Thank you! This is helpful. I have run through the performance guide before, but I’ll go through it again as perhaps things have changed. Thank you for adding a sizing guide, that is very helpful.
Is there a step-by-step guide for gathering thread dump information? The last time this came up, the suggested method seemed difficult to execute. Too difficult, to be honest. However, I am reaching a level of desperation where I would be willing to spend much more time on it.
Excellent, I have glowroot running now. Thanks for the tip!
I also noticed the recommended performance settings changed since I originally installed back around version 7, so I have updated those. One note, apparently MaxPermSize is deprecated as of Java 1.8.
At least I feel like I am moving forward. Thanks everyone on this thread for the help.
Hi, I wanted to close out this thread. After applying the new recommendations on the Performance wiki page, things are running very well. The dual-core installation hasn’t crashed at all, and the single-core installation has only crashed once since I posted. So that is great! Thanks for the help.
To be clear, I was running settings from when I originally installed (around version 7) and by version 10, using the old settings somehow contributed to the problems described above; thus I think since they change, advertising those changes is pretty important.
You can watch any page on xwiki.org, go to the bell on the top right corner and enable it for this page and then make sure you have a configured “Notifier” in the section Watchlist of your user profile.
crw, I’d also recommend installing javamelody so you can easily monitor your system and get a good idea of what your normal performance is.
The overhead is light, and the insights are invaluable. This tool will help you identify problems and also help you figure out where those problems might be.
Thank you, I looked for the notification ability but didn’t find it (I am default subscribed to all changes everywhere on the wikis I run, so it isn’t something I think about often).
edit: and, I wasn’t logged in so I couldn’t see the Bell icon; I made the assumption that functionality was turned off on that wiki somehow. As a UX thing, it might be good to have that greyed out with a message to log in to configure to be alerted to changes.
You should try glowroot (https://glowroot.org/) which provides even better information that javamelody. FYI we were using javamelody on xwiki.org before and we switched to glowroot.
For your use case maybe but not generically. XWiki can be used for lots of use cases, including a web site use case and it doesn’t make sense to show a greyed out icon for this use case where users are not supposed to register/have a login.
Ok I misunderstood you then. You mentioned “new recommendations” but these are very old one (since the beginning of XWiki actually, unless we’re talking about different things). So I don’t understand the part about “it would be nice if there were posts here when they are updated, or it would be nice if I could subscribe to that page to get notifications when it is changed.”. Do you mean this generically and not specifically for the memory recommendations?
Please take a moment to look closely at the two recommendations. Two directives were added to the beginning, and two values were changed. That is a change. You changed your recommendation. Thus becoming a new recommendation.
You may not consider it a very dramatic change, and yet it was the difference between a functioning server.
