java.lang.OutOfMemoryError: Java heap space

Hello,

I have a Multiwiki site hosted in a t2.medium Amazon EC2 Instance (2CPU, 4GB RAM).
The structure of my xwiki is this:

The problem is after some time of use I always recieve an out of memory exception, that freezes the app and that error reflects on the logs of tomcat8 (/var/log/tomcat8/catalina.out):

java.lang.OutOfMemoryError: Java heap space
Exception in thread "ContainerBackgroundProcessor[StandardEngine[Catalina]]" java.lang.OutOfMemoryError: Java heap space
Exception in thread "ContainerBackgroundProcessor[StandardEngine[Catalina]]" java.lang.OutOfMemoryError: Java heap space

My Java Opts are these:
JAVA_OPTS="-Djava.awt.headless=true -XX:+UseConcMarkSweepGC -Xmx3584m"

When this error comes, the only workaround fix that I have is reboot the tomcat8 service (sudo service tomcat8 restart)

Is there any recommendation on the JAVA_OPTS to make this doesn’t happen a lot?

Regards.

I think you should not require more than 1024m of memory for this.

Could you make sure about the memory that the JVM gets? See for ex https://snippets.xwiki.org/xwiki/bin/view/Extension/Available%20Memory

Here is the information after run the script on a page:

Current size of heap in bytes: 1056309248
Maximum size of heap in bytes. 1056309248
Amount of free memory within the heap in bytes: 112249720

I change my JAVA_OPTS to this configuration to manage 1024MB of max heap size:
JAVA_OPTS="-Djava.awt.headless=true -Xmx1024m -XX:+UseParallelGC -XX:MaxGCPauseMillis=100"

Could you make sure that your setup is taken into account using something like:

ps aux | grep java

-Xmx3584m you should be way more what you describe need so I’m really surprised to get a OOM with than much allocation memory.

If ps tells you you are using the memory you think you do then the next step would be to use -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/var/lib/xwiki/data/ to analyze what exactly is using all this memory.

I forgot that before have this issues, I modified a configuration file /usr/share/tomcat8/bin/setenv.sh to fix an issue related to the logs. Is this configuration can be related with my issue?

ubuntu@ip-172-31-32-11:/var/lib/xwiki/data$ cat /usr/share/tomcat8/bin/setenv.sh
export JAVA_OPTS="${JAVA_OPTS} -Djava.awt.headless=true"
CATALINA_OPTS="-Xmx1024m -XX:MaxPermSize=192m"

@tmortagne added parameters to the JAVA_OPTS
Now its set to:

JAVA_OPTS="-Djava.awt.headless=true -XX:+UseConcMarkSweepGC -Xmx3584m -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/var/lib/xwiki/data/"

I just got a downtime once again, but I didn’t see any file on that path /var/lib/xwiki/data/
Also its going really slow on editing or moving through pages.

Is having multiple active users editing and consulting on diferent subwikis can leave xWiki with that kind of behaviours?

Short answer: No.

XWiki.org is having about 10+ subwikis and is running with CATALINA_OPTS="-server -Xms800m -Xmx1600m -Dfile.encoding=utf-8 -Djava.awt.headless=true -XX:+UseParallelGC -XX:MaxGCPauseMillis=100" (see https://www.xwiki.org/xwiki/bin/view/Documentation/AdminGuide/Performances/#HMemory).

Where is the path to configure CATALINA_OPTS in xwiki 10.8?

This is not related to XWiki :slight_smile: It’s related to Tomcat. XWiki is independent of the Servlet container.

If you don’t know how to set the memory on Tomcat you should check Tomcat’s doc. We have some hints on https://www.xwiki.org/xwiki/bin/view/Documentation/AdminGuide/Installation/InstallationWAR/InstallationTomcat/#HOutOfMemoryError but it won’t replace proper Tomcat doc.

Here are my configuration for tomcat8.txt (1.9 KB)

And setenv.sh of tomcat8 setenv.txt (206 Bytes)

Here is also my tomcat and java that it’s running on the server

ubuntu@ip:/etc/xwiki$ java -cp /usr/share/tomcat8/lib/catalina.jar org.apache.catalina.util.ServerInfo
Server version: Apache Tomcat/8.5.30 (Ubuntu)
Server built:   Aug 13 2018 20:23:56 UTC
Server number:  8.5.30.0
OS Name:        Linux
OS Version:     4.15.0-1027-aws
Architecture:   amd64
JVM Version:    1.8.0_181-8u181-b13-1ubuntu0.18.04.1-b13
JVM Vendor:     Oracle Corporation

Still have issues with the performance (Slow response when consulting pages and editing pages, about 30-45 seconds the user needs to wait to see changes)

@tmortagne @vmassol

  1. My xwiki was set it up using the apt-get install tomcat8-xwiki-common, is there a more efficient way to intall?
  2. Upgrading/Migrating from 10.8 to 11.x has improvements to the performance?

This is the recommended way to install on Debian-based system but then you should have looked at https://www.xwiki.org/xwiki/bin/view/Documentation/AdminGuide/Installation/InstallationViaAPT/#HMemory where you can see that the right configuration file for Tomcat is /etc/default/tomcat8 and not /usr/share/tomcat8/bin/setenv.sh.

A bit but there is no reason for 10.8 to be so slow either.

You might want to take a look at https://www.xwiki.org/xwiki/bin/view/Documentation/AdminGuide/Performances/.

That means your downtime was not related to an OutOfMemory.

@tmortagne I just saw today that I got a dump file /var/lib/xwiki/data/java_pid11562.hprof but it takes all of space of my instance volume (8GB).

Once I can download and analyze it. I will post the results of the file.

@tmortagne @vmassol

I just saw someone post a similar issue about xwiki timeouts (https://forum.xwiki.org/t/incessant-timeouts-slow-performance-overall/3595)

So I installed javamelody on the server to monitor the usage of the service, seeing that there is a high consumption of the CPU around 200%.

On the post they say the notification system has some issues about high comsuption of CPU, so I decide to disable the notifications using the documentation on this page https://extensions.xwiki.org/xwiki/bin/view/Extension/Notifications%20Application/

After disable the notifications the cpu usage drastically drops to low levels.