java.lang.OutOfMemoryError: Java heap space

drivera · February 11, 2019, 9:52pm

Hello,

I have a Multiwiki site hosted in a t2.medium Amazon EC2 Instance (2CPU, 4GB RAM).
The structure of my xwiki is this:

xwiki 10.8
Server version: Apache Tomcat/8.5.30 (Ubuntu)
JVM Version: 1.8.0_181-8u181-b13-1ubuntu0.18.04.1-b13
7 Subwikis
10 Global users (This users authenticate throught the wiki using LDAP plugin https://extensions.xwiki.org/xwiki/bin/view/Extension/LDAP/Authenticator/)
16 Local users (distribuited on the subwikis)

The problem is after some time of use I always recieve an out of memory exception, that freezes the app and that error reflects on the logs of tomcat8 (/var/log/tomcat8/catalina.out):

java.lang.OutOfMemoryError: Java heap space
Exception in thread "ContainerBackgroundProcessor[StandardEngine[Catalina]]" java.lang.OutOfMemoryError: Java heap space
Exception in thread "ContainerBackgroundProcessor[StandardEngine[Catalina]]" java.lang.OutOfMemoryError: Java heap space

My Java Opts are these:
JAVA_OPTS="-Djava.awt.headless=true -XX:+UseConcMarkSweepGC -Xmx3584m"

When this error comes, the only workaround fix that I have is reboot the tomcat8 service (sudo service tomcat8 restart)

Is there any recommendation on the JAVA_OPTS to make this doesn’t happen a lot?

Regards.

vmassol · February 11, 2019, 10:00pm

I think you should not require more than 1024m of memory for this.

Could you make sure about the memory that the JVM gets? See for ex https://snippets.xwiki.org/xwiki/bin/view/Extension/Available%20Memory

drivera · February 11, 2019, 10:08pm

Here is the information after run the script on a page:

Current size of heap in bytes: 1056309248
Maximum size of heap in bytes. 1056309248
Amount of free memory within the heap in bytes: 112249720

drivera · February 11, 2019, 11:30pm

I change my JAVA_OPTS to this configuration to manage 1024MB of max heap size:
JAVA_OPTS="-Djava.awt.headless=true -Xmx1024m -XX:+UseParallelGC -XX:MaxGCPauseMillis=100"

tmortagne · February 12, 2019, 9:52am

Could you make sure that your setup is taken into account using something like:

ps aux | grep java

-Xmx3584m you should be way more what you describe need so I’m really surprised to get a OOM with than much allocation memory.

If ps tells you you are using the memory you think you do then the next step would be to use -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/var/lib/xwiki/data/ to analyze what exactly is using all this memory.

drivera · February 12, 2019, 7:47pm

I forgot that before have this issues, I modified a configuration file /usr/share/tomcat8/bin/setenv.sh to fix an issue related to the logs. Is this configuration can be related with my issue?

ubuntu@ip-172-31-32-11:/var/lib/xwiki/data$ cat /usr/share/tomcat8/bin/setenv.sh
export JAVA_OPTS="${JAVA_OPTS} -Djava.awt.headless=true"
CATALINA_OPTS="-Xmx1024m -XX:MaxPermSize=192m"

drivera · February 12, 2019, 8:58pm

@tmortagne added parameters to the JAVA_OPTS
Now its set to:

JAVA_OPTS="-Djava.awt.headless=true -XX:+UseConcMarkSweepGC -Xmx3584m -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/var/lib/xwiki/data/"

I just got a downtime once again, but I didn’t see any file on that path /var/lib/xwiki/data/
Also its going really slow on editing or moving through pages.

Is having multiple active users editing and consulting on diferent subwikis can leave xWiki with that kind of behaviours?

vmassol · February 12, 2019, 9:29pm

Short answer: No.

XWiki.org is having about 10+ subwikis and is running with CATALINA_OPTS="-server -Xms800m -Xmx1600m -Dfile.encoding=utf-8 -Djava.awt.headless=true -XX:+UseParallelGC -XX:MaxGCPauseMillis=100" (see https://www.xwiki.org/xwiki/bin/view/Documentation/AdminGuide/Performances/#HMemory).

drivera · February 12, 2019, 9:37pm

Where is the path to configure CATALINA_OPTS in xwiki 10.8?

vmassol · February 12, 2019, 9:42pm

This is not related to XWiki It’s related to Tomcat. XWiki is independent of the Servlet container.

If you don’t know how to set the memory on Tomcat you should check Tomcat’s doc. We have some hints on https://www.xwiki.org/xwiki/bin/view/Documentation/AdminGuide/Installation/InstallationWAR/InstallationTomcat/#HOutOfMemoryError but it won’t replace proper Tomcat doc.

drivera · February 12, 2019, 10:53pm

Here are my configuration for tomcat8.txt (1.9 KB)

And setenv.sh of tomcat8 setenv.txt (206 Bytes)

Here is also my tomcat and java that it’s running on the server

ubuntu@ip:/etc/xwiki$ java -cp /usr/share/tomcat8/lib/catalina.jar org.apache.catalina.util.ServerInfo
Server version: Apache Tomcat/8.5.30 (Ubuntu)
Server built:   Aug 13 2018 20:23:56 UTC
Server number:  8.5.30.0
OS Name:        Linux
OS Version:     4.15.0-1027-aws
Architecture:   amd64
JVM Version:    1.8.0_181-8u181-b13-1ubuntu0.18.04.1-b13
JVM Vendor:     Oracle Corporation

Still have issues with the performance (Slow response when consulting pages and editing pages, about 30-45 seconds the user needs to wait to see changes)

drivera · February 12, 2019, 11:06pm

@tmortagne @vmassol

My xwiki was set it up using the apt-get install tomcat8-xwiki-common, is there a more efficient way to intall?
Upgrading/Migrating from 10.8 to 11.x has improvements to the performance?

tmortagne · February 13, 2019, 7:38am

This is the recommended way to install on Debian-based system but then you should have looked at https://www.xwiki.org/xwiki/bin/view/Documentation/AdminGuide/Installation/InstallationViaAPT/#HMemory where you can see that the right configuration file for Tomcat is /etc/default/tomcat8 and not /usr/share/tomcat8/bin/setenv.sh.

A bit but there is no reason for 10.8 to be so slow either.

You might want to take a look at https://www.xwiki.org/xwiki/bin/view/Documentation/AdminGuide/Performances/.

That means your downtime was not related to an OutOfMemory.

drivera · February 13, 2019, 6:37pm

@tmortagne I just saw today that I got a dump file /var/lib/xwiki/data/java_pid11562.hprof but it takes all of space of my instance volume (8GB).

Once I can download and analyze it. I will post the results of the file.

drivera · February 13, 2019, 10:19pm

@tmortagne @vmassol

I just saw someone post a similar issue about xwiki timeouts (https://forum.xwiki.org/t/incessant-timeouts-slow-performance-overall/3595)

So I installed javamelody on the server to monitor the usage of the service, seeing that there is a high consumption of the CPU around 200%.

On the post they say the notification system has some issues about high comsuption of CPU, so I decide to disable the notifications using the documentation on this page https://extensions.xwiki.org/xwiki/bin/view/Extension/Notifications%20Application/

After disable the notifications the cpu usage drastically drops to low levels.

golkonda · February 12, 2022, 7:49pm

Hi @vmassol, I’m getting the same error - Caused by: java.lang.OutOfMemoryError: Java heap space
I’m trying to import 1GB and 2GB .xar files. Below is the modified environment variable in /etc/systemd/system/tomcat.service

Environment='CATALINA_OPTS=-Xms2048M -Xmx3072M -XX:MaxPermSize=192m -server -XX:+UseParallelGC -Djava.net.preferIPv4Stack=true -Djava.net.preferIPv4Addresses=true'

Seems to be memory leak issue.
Actually I took maintenance window for this task, migrating xwiki from one to another xwiki instance.
Little urgent please.
Thank you.

golkonda · February 13, 2022, 1:19am

Hi @tmortagne, facing the same issue - OOM. ps aux | grep java showing lot of java processes of only one tomcat instance but not showing processes of another tomcat. Both are running in one machine. Trying to migrate from one XWiki to another XWiki. Below is the ps aux |grep java output. But it shows OOM error for other tomcat instance - /opt/tomcat/tomcat/… And both tomcats are running as systemd/systemctl services. Can we run two tomcat in systemd ? How do I install multiple Tomcat servers in single machines? - Quora Here Anand suggesting don’t install it as a service.

tomcat     737  0.1  0.4 14242340 590156 ?     Sl    2021 518:32 //bin/java -Djava.util.logging.config.file=/opt/tomcat/apache-tomcat-9.0.48/conf/logging.properties -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager -Djdk.tls.ephemeralDHKeySize=2048 -Djava.protocol.handler.pkgs=org.apache.catalina.webresources -Dorg.apache.catalina.security.SecurityListener.UMASK=0027 -Xmx1024m -XX:MaxPermSize=192m -Djava.net.preferIPv4Stack=true -Djava.net.preferIPv4Addresses=true -Dignore.endorsed.dirs= -classpath /opt/tomcat/apache-tomcat-9.0.48/bin/bootstrap.jar:/opt/tomcat/apache-tomcat-9.0.48/bin/tomcat-juli.jar -Dcatalina.base=/opt/tomcat/apache-tomcat-9.0.48 -Dcatalina.home=/opt/tomcat/apache-tomcat-9.0.48 -Djava.io.tmpdir=/opt/tomcat/apache-tomcat-9.0.48/temp org.apache.catalina.startup.Bootstrap start
tomcat    1346  0.0  0.2 7633716 315684 ?      Sl    2021 290:39 //bin/java -Djava.util.logging.config.file=/opt/tomcat/apache-tomcat-9.0.48/conf/logging.properties -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager -Djdk.tls.ephemeralDHKeySize=2048 -Djava.protocol.handler.pkgs=org.apache.catalina.webresources -Dorg.apache.catalina.security.SecurityListener.UMASK=0027 -Xmx1024m -XX:MaxPermSize=192m -Djava.net.preferIPv4Stack=true -Djava.net.preferIPv4Addresses=true -Dignore.endorsed.dirs= -classpath /opt/tomcat/apache-tomcat-9.0.48/bin/bootstrap.jar:/opt/tomcat/apache-tomcat-9.0.48/bin/tomcat-juli.jar -Dcatalina.base=/opt/tomcat/apache-tomcat-9.0.48 -Dcatalina.home=/opt/tomcat/apache-tomcat-9.0.48 -Djava.io.tmpdir=/opt/tomcat/apache-tomcat-9.0.48/temp org.apache.catalina.startup.Bootstrap start
tomcat    2701  0.1  4.4 43124100 5890744 ?    Sl    2021 393:37 //bin/java -Djava.util.logging.config.file=/opt/tomcat/apache-tomcat-9.0.48/conf/logging.properties -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager -Djdk.tls.ephemeralDHKeySize=2048 -Djava.protocol.handler.pkgs=org.apache.catalina.webresources -Dorg.apache.catalina.security.SecurityListener.UMASK=0027 -Dignore.endorsed.dirs= -classpath /opt/tomcat/apache-tomcat-9.0.48/bin/bootstrap.jar:/opt/tomcat/apache-tomcat-9.0.48/bin/tomcat-juli.jar -Dcatalina.base=/opt/tomcat/apache-tomcat-9.0.48 -Dcatalina.home=/opt/tomcat/apache-tomcat-9.0.48 -Djava.io.tmpdir=/opt/tomcat/apache-tomcat-9.0.48/temp org.apache.catalina.startup.Bootstrap start
tomcat    3422  0.0  2.2 39219880 2934608 ?    Sl    2021 294:05 //bin/java -Djava.util.logging.config.file=/opt/tomcat/apache-tomcat-9.0.48/conf/logging.properties -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager -Djdk.tls.ephemeralDHKeySize=2048 -Djava.protocol.handler.pkgs=org.apache.catalina.webresources -Dorg.apache.catalina.security.SecurityListener.UMASK=0027 -Dignore.endorsed.dirs= -classpath /opt/tomcat/apache-tomcat-9.0.48/bin/bootstrap.jar:/opt/tomcat/apache-tomcat-9.0.48/bin/tomcat-juli.jar -Dcatalina.base=/opt/tomcat/apache-tomcat-9.0.48 -Dcatalina.home=/opt/tomcat/apache-tomcat-9.0.48 -Djava.io.tmpdir=/opt/tomcat/apache-tomcat-9.0.48/temp org.apache.catalina.startup.Bootstrap start
tomcat    3907  0.0  1.8 38217324 2409168 ?    Sl    2021 289:29 //bin/java -Djava.util.logging.config.file=/opt/tomcat/apache-tomcat-9.0.48/conf/logging.properties -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager -Djdk.tls.ephemeralDHKeySize=2048 -Djava.protocol.handler.pkgs=org.apache.catalina.webresources -Dorg.apache.catalina.security.SecurityListener.UMASK=0027 -Dignore.endorsed.dirs= -classpath /opt/tomcat/apache-tomcat-9.0.48/bin/bootstrap.jar:/opt/tomcat/apache-tomcat-9.0.48/bin/tomcat-juli.jar -Dcatalina.base=/opt/tomcat/apache-tomcat-9.0.48 -Dcatalina.home=/opt/tomcat/apache-tomcat-9.0.48 -Djava.io.tmpdir=/opt/tomcat/apache-tomcat-9.0.48/temp org.apache.catalina.startup.Bootstrap start
tomcat    4314  0.1  1.0 15161492 1348432 ?    Sl    2021 545:22 //bin/java -Djava.util.logging.config.file=/opt/tomcat/apache-tomcat-9.0.48/conf/logging.properties -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager -Djdk.tls.ephemeralDHKeySize=2048 -Djava.protocol.handler.pkgs=org.apache.catalina.webresources -Dorg.apache.catalina.security.SecurityListener.UMASK=0027 -Xmx1024m -XX:MaxPermSize=192m -Djava.net.preferIPv4Stack=true -Djava.net.preferIPv4Addresses=true -Dignore.endorsed.dirs= -classpath /opt/tomcat/apache-tomcat-9.0.48/bin/bootstrap.jar:/opt/tomcat/apache-tomcat-9.0.48/bin/tomcat-juli.jar -Dcatalina.base=/opt/tomcat/apache-tomcat-9.0.48 -Dcatalina.home=/opt/tomcat/apache-tomcat-9.0.48 -Djava.io.tmpdir=/opt/tomcat/apache-tomcat-9.0.48/temp org.apache.catalina.startup.Bootstrap start
tomcat    4705  0.0  1.8 39151268 2389516 ?    Sl    2021 288:40 //bin/java -Djava.util.logging.config.file=/opt/tomcat/apache-tomcat-9.0.48/conf/logging.properties -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager -Djdk.tls.ephemeralDHKeySize=2048 -Djava.protocol.handler.pkgs=org.apache.catalina.webresources -Dorg.apache.catalina.security.SecurityListener.UMASK=0027 -Dignore.endorsed.dirs= -classpath /opt/tomcat/apache-tomcat-9.0.48/bin/bootstrap.jar:/opt/tomcat/apache-tomcat-9.0.48/bin/tomcat-juli.jar -Dcatalina.base=/opt/tomcat/apache-tomcat-9.0.48 -Dcatalina.home=/opt/tomcat/apache-tomcat-9.0.48 -Djava.io.tmpdir=/opt/tomcat/apache-tomcat-9.0.48/temp org.apache.catalina.startup.Bootstrap start
tomcat    5495  0.0  2.0 39084704 2693768 ?    Sl    2021 294:02 //bin/java -Djava.util.logging.config.file=/opt/tomcat/apache-tomcat-9.0.48/conf/logging.properties -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager -Djdk.tls.ephemeralDHKeySize=2048 -Djava.protocol.handler.pkgs=org.apache.catalina.webresources -Dorg.apache.catalina.security.SecurityListener.UMASK=0027 -Dignore.endorsed.dirs= -classpath /opt/tomcat/apache-tomcat-9.0.48/bin/bootstrap.jar:/opt/tomcat/apache-tomcat-9.0.48/bin/tomcat-juli.jar -Dcatalina.base=/opt/tomcat/apache-tomcat-9.0.48 -Dcatalina.home=/opt/tomcat/apache-tomcat-9.0.48 -Djava.io.tmpdir=/opt/tomcat/apache-tomcat-9.0.48/temp org.apache.catalina.startup.Bootstrap start
tomcat    5995  0.0  1.6 39357104 2129000 ?    Sl    2021 291:33 //bin/java -Djava.util.logging.config.file=/opt/tomcat/apache-tomcat-9.0.48/conf/logging.properties -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager -Djdk.tls.ephemeralDHKeySize=2048 -Djava.protocol.handler.pkgs=org.apache.catalina.webresources -Dorg.apache.catalina.security.SecurityListener.UMASK=0027 -Dignore.endorsed.dirs= -classpath /opt/tomcat/apache-tomcat-9.0.48/bin/bootstrap.jar:/opt/tomcat/apache-tomcat-9.0.48/bin/tomcat-juli.jar -Dcatalina.base=/opt/tomcat/apache-tomcat-9.0.48 -Dcatalina.home=/opt/tomcat/apache-tomcat-9.0.48 -Djava.io.tmpdir=/opt/tomcat/apache-tomcat-9.0.48/temp org.apache.catalina.startup.Bootstrap start
tomcat    6431  0.1  2.0 45671536 2638296 ?    Sl    2021 520:12 //bin/java -Djava.util.logging.config.file=/opt/tomcat/apache-tomcat-9.0.48/conf/logging.properties -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager -Djdk.tls.ephemeralDHKeySize=2048 -Djava.protocol.handler.pkgs=org.apache.catalina.webresources -Dorg.apache.catalina.security.SecurityListener.UMASK=0027 -Dignore.endorsed.dirs= -classpath /opt/tomcat/apache-tomcat-9.0.48/bin/bootstrap.jar:/opt/tomcat/apache-tomcat-9.0.48/bin/tomcat-juli.jar -Dcatalina.base=/opt/tomcat/apache-tomcat-9.0.48 -Dcatalina.home=/opt/tomcat/apache-tomcat-9.0.48 -Djava.io.tmpdir=/opt/tomcat/apache-tomcat-9.0.48/temp org.apache.catalina.startup.Bootstrap start
tomcat    7060  0.0  1.8 39084704 2385004 ?    Sl    2021 295:31 //bin/java -Djava.util.logging.config.file=/opt/tomcat/apache-tomcat-9.0.48/conf/logging.properties -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager -Djdk.tls.ephemeralDHKeySize=2048 -Djava.protocol.handler.pkgs=org.apache.catalina.webresources -Dorg.apache.catalina.security.SecurityListener.UMASK=0027 -Dignore.endorsed.dirs= -classpath /opt/tomcat/apache-tomcat-9.0.48/bin/bootstrap.jar:/opt/tomcat/apache-tomcat-9.0.48/bin/tomcat-juli.jar -Dcatalina.base=/opt/tomcat/apache-tomcat-9.0.48 -Dcatalina.home=/opt/tomcat/apache-tomcat-9.0.48 -Djava.io.tmpdir=/opt/tomcat/apache-tomcat-9.0.48/temp org.apache.catalina.startup.Bootstrap start
tomcat    8913  0.1  3.3 45731916 4429576 ?    Sl    2021 517:57 //bin/java -Djava.util.logging.config.file=/opt/tomcat/apache-tomcat-9.0.48/conf/logging.properties -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager -Djdk.tls.ephemeralDHKeySize=2048 -Djava.protocol.handler.pkgs=org.apache.catalina.webresources -Dorg.apache.catalina.security.SecurityListener.UMASK=0027 -Dignore.endorsed.dirs= -classpath /opt/tomcat/apache-tomcat-9.0.48/bin/bootstrap.jar:/opt/tomcat/apache-tomcat-9.0.48/bin/tomcat-juli.jar -Dcatalina.base=/opt/tomcat/apache-tomcat-9.0.48 -Dcatalina.home=/opt/tomcat/apache-tomcat-9.0.48 -Djava.io.tmpdir=/opt/tomcat/apache-tomcat-9.0.48/temp org.apache.catalina.startup.Bootstrap start
tomcat    9469  0.0  2.2 39084704 2932896 ?    Sl    2021 292:46 //bin/java -Djava.util.logging.config.file=/opt/tomcat/apache-tomcat-9.0.48/conf/logging.properties -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager -Djdk.tls.ephemeralDHKeySize=2048 -Djava.protocol.handler.pkgs=org.apache.catalina.webresources -Dorg.apache.catalina.security.SecurityListener.UMASK=0027 -Dignore.endorsed.dirs= -classpath /opt/tomcat/apache-tomcat-9.0.48/bin/bootstrap.jar:/opt/tomcat/apache-tomcat-9.0.48/bin/tomcat-juli.jar -Dcatalina.base=/opt/tomcat/apache-tomcat-9.0.48 -Dcatalina.home=/opt/tomcat/apache-tomcat-9.0.48 -Djava.io.tmpdir=/opt/tomcat/apache-tomcat-9.0.48/temp org.apache.catalina.startup.Bootstrap start
tomcat   11488  0.1  3.7 45729796 4941000 ?    Sl    2021 525:20 //bin/java -Djava.util.logging.config.file=/opt/tomcat/apache-tomcat-9.0.48/conf/logging.properties -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager -Djdk.tls.ephemeralDHKeySize=2048 -Djava.protocol.handler.pkgs=org.apache.catalina.webresources -Dorg.apache.catalina.security.SecurityListener.UMASK=0027 -Dignore.endorsed.dirs= -classpath /opt/tomcat/apache-tomcat-9.0.48/bin/bootstrap.jar:/opt/tomcat/apache-tomcat-9.0.48/bin/tomcat-juli.jar -Dcatalina.base=/opt/tomcat/apache-tomcat-9.0.48 -Dcatalina.home=/opt/tomcat/apache-tomcat-9.0.48 -Djava.io.tmpdir=/opt/tomcat/apache-tomcat-9.0.48/temp org.apache.catalina.startup.Bootstrap start
tomcat   12023  0.0  2.3 39018140 3118008 ?    Sl    2021 290:17 //bin/java -Djava.util.logging.config.file=/opt/tomcat/apache-tomcat-9.0.48/conf/logging.properties -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager -Djdk.tls.ephemeralDHKeySize=2048 -Djava.protocol.handler.pkgs=org.apache.catalina.webresources -Dorg.apache.catalina.security.SecurityListener.UMASK=0027 -Dignore.endorsed.dirs= -classpath /opt/tomcat/apache-tomcat-9.0.48/bin/bootstrap.jar:/opt/tomcat/apache-tomcat-9.0.48/bin/tomcat-juli.jar -Dcatalina.base=/opt/tomcat/apache-tomcat-9.0.48 -Dcatalina.home=/opt/tomcat/apache-tomcat-9.0.48 -Djava.io.tmpdir=/opt/tomcat/apache-tomcat-9.0.48/temp org.apache.catalina.startup.Bootstrap start
tomcat   12264  0.0  2.3 42460924 3103516 ?    Sl    2021 290:56 //bin/java -Djava.util.logging.config.file=/opt/tomcat/apache-tomcat-9.0.48/conf/logging.properties -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager -Djdk.tls.ephemeralDHKeySize=2048 -Djava.protocol.handler.pkgs=org.apache.catalina.webresources -Dorg.apache.catalina.security.SecurityListener.UMASK=0027 -Dignore.endorsed.dirs= -classpath /opt/tomcat/apache-tomcat-9.0.48/bin/bootstrap.jar:/opt/tomcat/apache-tomcat-9.0.48/bin/tomcat-juli.jar -Dcatalina.base=/opt/tomcat/apache-tomcat-9.0.48 -Dcatalina.home=/opt/tomcat/apache-tomcat-9.0.48 -Djava.io.tmpdir=/opt/tomcat/apache-tomcat-9.0.48/temp org.apache.catalina.startup.Bootstrap start
OktaLDA+ 13812  0.1  0.4 38431048 640788 ?     Sl    2021 233:49 /opt/Okta/OktaLDAPAgent/jre/bin/java -Dlogback.configurationFile=/opt/Okta/OktaLDAPAgent/conf/logback.xml -Dagent_home=/opt/Okta/OktaLDAPAgent -Dfile.encoding=UTF8 -jar /opt/Okta/OktaLDAPAgent/bin/OktaLDAPAgent.jar -mode normal -configFilePath /opt/Okta/OktaLDAPAgent/conf/OktaLDAPAgent.conf
tomcat   15072 24.6  2.5 16361648 3384432 ?    Sl   16:10  15:30 /usr/lib/jvm/jre/bin/java -Djava.util.logging.config.file=/opt/tomcat/tomcat/conf/logging.properties -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager -Djava.awt.headless=true -Djava.security.egd=file:/dev/./urandom -Djdk.tls.ephemeralDHKeySize=2048 -Djava.protocol.handler.pkgs=org.apache.catalina.webresources -Dorg.apache.catalina.security.SecurityListener.UMASK=0027 -Xmx1024m -XX:MaxPermSize=192m -Dignore.endorsed.dirs= -classpath /opt/tomcat/tomcat/bin/bootstrap.jar:/opt/tomcat/tomcat/bin/tomcat-juli.jar -Dcatalina.base=/opt/tomcat/tomcat -Dcatalina.home=/opt/tomcat/tomcat -Djava.io.tmpdir=/opt/tomcat/tomcat/temp org.apache.catalina.startup.Bootstrap start
root     22717  0.1  1.2 39022244 1603888 ?    Ssl  Jan13  80:12 /usr/local/openjdk-11/bin/java -Djava.util.logging.config.file=/usr/local/tomcat/conf/logging.properties -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager -Djdk.tls.ephemeralDHKeySize=2048 -Djava.protocol.handler.pkgs=org.apache.catalina.webresources -Dorg.apache.catalina.security.SecurityListener.UMASK=0027 -Dignore.endorsed.dirs= -classpath /usr/local/tomcat/bin/bootstrap.jar:/usr/local/tomcat/bin/tomcat-juli.jar -Dcatalina.base=/usr/local/tomcat -Dcatalina.home=/usr/local/tomcat -Djava.io.tmpdir=/usr/local/tomcat/temp org.apache.catalina.startup.Bootstrap start
tomcat   23297  0.0  0.0 112816   976 pts/1    S+   17:13   0:00 grep --color=auto java
tomcat   27441  0.0  0.2 7768892 369232 ?      Sl    2021 291:29 //bin/java -Djava.util.logging.config.file=/opt/tomcat/apache-tomcat-9.0.48/conf/logging.properties -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager -Djdk.tls.ephemeralDHKeySize=2048 -Djava.protocol.handler.pkgs=org.apache.catalina.webresources -Dorg.apache.catalina.security.SecurityListener.UMASK=0027 -Xmx1024m -XX:MaxPermSize=192m -Djava.net.preferIPv4Stack=true -Djava.net.preferIPv4Addresses=true -Dignore.endorsed.dirs= -classpath /opt/tomcat/apache-tomcat-9.0.48/bin/bootstrap.jar:/opt/tomcat/apache-tomcat-9.0.48/bin/tomcat-juli.jar -Dcatalina.base=/opt/tomcat/apache-tomcat-9.0.48 -Dcatalina.home=/opt/tomcat/apache-tomcat-9.0.48 -Djava.io.tmpdir=/opt/tomcat/apache-tomcat-9.0.48/temp org.apache.catalina.startup.Bootstrap start
tomcat   27630  0.1  0.8 14298180 1165948 ?    Sl    2021 530:43 //bin/java -Djava.util.logging.config.file=/opt/tomcat/apache-tomcat-9.0.48/conf/logging.properties -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager -Djdk.tls.ephemeralDHKeySize=2048 -Djava.protocol.handler.pkgs=org.apache.catalina.webresources -Dorg.apache.catalina.security.SecurityListener.UMASK=0027 -Xmx1024m -XX:MaxPermSize=192m -Djava.net.preferIPv4Stack=true -Djava.net.preferIPv4Addresses=true -Dignore.endorsed.dirs= -classpath /opt/tomcat/apache-tomcat-9.0.48/bin/bootstrap.jar:/opt/tomcat/apache-tomcat-9.0.48/bin/tomcat-juli.jar -Dcatalina.base=/opt/tomcat/apache-tomcat-9.0.48 -Dcatalina.home=/opt/tomcat/apache-tomcat-9.0.48 -Djava.io.tmpdir=/opt/tomcat/apache-tomcat-9.0.48/temp org.apache.catalina.startup.Bootstrap start
1000     27808  0.3  1.0 37700496 1339980 ?    Ssl  Jan14 139:41 java -Dlogback.configurationFile=/etc/kroki/logback.xml -Dvertx.logger-delegate-factory-class-name=io.vertx.core.logging.SLF4JLogDelegateFactory -jar /usr/local/kroki/kroki-server.jar
tomcat   29149  0.0  0.2 12451224 354104 ?     Sl    2021 292:44 //bin/java -Djava.util.logging.config.file=/opt/tomcat/apache-tomcat-9.0.48/conf/logging.properties -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager -Djdk.tls.ephemeralDHKeySize=2048 -Djava.protocol.handler.pkgs=org.apache.catalina.webresources -Dorg.apache.catalina.security.SecurityListener.UMASK=0027 -Xmx1024m -XX:MaxPermSize=192m -Djava.net.preferIPv4Stack=true -Djava.net.preferIPv4Addresses=true -Dignore.endorsed.dirs= -classpath /opt/tomcat/apache-tomcat-9.0.48/bin/bootstrap.jar:/opt/tomcat/apache-tomcat-9.0.48/bin/tomcat-juli.jar -Dcatalina.base=/opt/tomcat/apache-tomcat-9.0.48 -Dcatalina.home=/opt/tomcat/apache-tomcat-9.0.48 -Djava.io.tmpdir=/opt/tomcat/apache-tomcat-9.0.48/temp org.apache.catalina.startup.Bootstrap start
root     29320  0.2  0.6 38203140 829480 pts/0 Ssl+ Feb08  13:21 /opt/java/openjdk/bin/java -Djava.util.logging.config.file=/usr/local/tomcat/conf/logging.properties -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager -Djdk.tls.ephemeralDHKeySize=2048 -Djava.protocol.handler.pkgs=org.apache.catalina.webresources -Dorg.apache.catalina.security.SecurityListener.UMASK=0027 -Dignore.endorsed.dirs= -classpath /usr/local/tomcat/bin/bootstrap.jar:/usr/local/tomcat/bin/tomcat-juli.jar -Dcatalina.base=/usr/local/tomcat -Dcatalina.home=/usr/local/tomcat -Djava.io.tmpdir=/usr/local/tomcat/temp org.apache.catalina.startup.Bootstrap start
tomcat   30021  0.0  0.2 7702328 376360 ?      Sl    2021 291:22 //bin/java -Djava.util.logging.config.file=/opt/tomcat/apache-tomcat-9.0.48/conf/logging.properties -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager -Djdk.tls.ephemeralDHKeySize=2048 -Djava.protocol.handler.pkgs=org.apache.catalina.webresources -Dorg.apache.catalina.security.SecurityListener.UMASK=0027 -Xmx1024m -XX:MaxPermSize=192m -Djava.net.preferIPv4Stack=true -Djava.net.preferIPv4Addresses=true -Dignore.endorsed.dirs= -classpath /opt/tomcat/apache-tomcat-9.0.48/bin/bootstrap.jar:/opt/tomcat/apache-tomcat-9.0.48/bin/tomcat-juli.jar -Dcatalina.base=/opt/tomcat/apache-tomcat-9.0.48 -Dcatalina.home=/opt/tomcat/apache-tomcat-9.0.48 -Djava.io.tmpdir=/opt/tomcat/apache-tomcat-9.0.48/temp org.apache.catalina.startup.Bootstrap start
tomcat   31089  0.1  0.8 14887092 1164116 ?    Sl    2021 528:11 //bin/java -Djava.util.logging.config.file=/opt/tomcat/apache-tomcat-9.0.48/conf/logging.properties -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager -Djdk.tls.ephemeralDHKeySize=2048 -Djava.protocol.handler.pkgs=org.apache.catalina.webresources -Dorg.apache.catalina.security.SecurityListener.UMASK=0027 -Xmx1024m -XX:MaxPermSize=192m -Djava.net.preferIPv4Stack=true -Djava.net.preferIPv4Addresses=true -Dignore.endorsed.dirs= -classpath /opt/tomcat/apache-tomcat-9.0.48/bin/bootstrap.jar:/opt/tomcat/apache-tomcat-9.0.48/bin/tomcat-juli.jar -Dcatalina.base=/opt/tomcat/apache-tomcat-9.0.48 -Dcatalina.home=/opt/tomcat/apache-tomcat-9.0.48 -Djava.io.tmpdir=/opt/tomcat/apache-tomcat-9.0.48/temp org.apache.catalina.startup.Bootstrap start
tomcat   32074  0.0  0.2 7768892 374796 ?      Sl    2021 291:37 //bin/java -Djava.util.logging.config.file=/opt/tomcat/apache-tomcat-9.0.48/conf/logging.properties -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager -Djdk.tls.ephemeralDHKeySize=2048 -Djava.protocol.handler.pkgs=org.apache.catalina.webresources -Dorg.apache.catalina.security.SecurityListener.UMASK=0027 -Xmx1024m -XX:MaxPermSize=192m -Djava.net.preferIPv4Stack=true -Djava.net.preferIPv4Addresses=true -Dignore.endorsed.dirs= -classpath /opt/tomcat/apache-tomcat-9.0.48/bin/bootstrap.jar:/opt/tomcat/apache-tomcat-9.0.48/bin/tomcat-juli.jar -Dcatalina.base=/opt/tomcat/apache-tomcat-9.0.48 -Dcatalina.home=/opt/tomcat/apache-tomcat-9.0.48 -Djava.io.tmpdir=/opt/tomcat/apache-tomcat-9.0.48/temp org.apache.catalina.startup.Bootstrap start
tomcat   32745 13.4  4.1 48769872 5445980 ?    Sl   14:49  19:23 /usr/lib/jvm/jre/bin/java -Djava.util.logging.config.file=/opt/tomcat/tomcat9/conf/logging.properties -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager -Djava.awt.headless=true -Djava.security.egd=file:/dev/./urandom -Djdk.tls.ephemeralDHKeySize=2048 -Djava.protocol.handler.pkgs=org.apache.catalina.webresources -Dorg.apache.catalina.security.SecurityListener.UMASK=0027 -Dorg.apache.tomcat.util.buf.UDecoder.ALLOW_ENCODED_SLASH=true -Dignore.endorsed.dirs= -classpath /opt/tomcat/tomcat9/bin/bootstrap.jar:/opt/tomcat/tomcat9/bin/tomcat-juli.jar -Dcatalina.base=/opt/tomcat/tomcat9 -Dcatalina.home=/opt/tomcat/tomcat9 -Djava.io.tmpdir=/opt/tomcat/tomcat9/temp org.apache.catalina.startup.Bootstrap start

vmassol · February 14, 2022, 9:00am

Hi @golkonda .

Small note for next time: please do not reuse an existing thread, especially since this one was already closed.

That’s too hasty to know. It’s a lot more likely that you’re missing JVM memory instead.

Several questions:

what version of XWiki is this?
please verify the memory available to XWiki using https://snippets.xwiki.org/xwiki/bin/view/Extension/Available%20Memory

Thanks

PS: The various tomcat processes look weird. You should look into that since each one will use memory.

golkonda · February 14, 2022, 11:23am

Thank you, @vmassol.
Sorry, next I’ll open a fresh thread, I don’t know that I shouldn’t reuse a closed thread.

I’m running two XWiki versions on two different tomcat instances but both tomcat instances are running on one machine - CentOS7.
a. XWiki v13.10.2, tomcat v9.0.54 - Seeing this error - Caused by: java.lang.OutOfMemoryError: Java heap space, when trying import wiki pages as .xar file (2GB) from the other version.
b. XWiki v12.10.8, tomcat -v9.0.48 - seeing Mulitple Java threads running as replied to @tmortagne when checked with ps aux |grep java. Exporting .xar file from here.
Not able to run that groovy script but I installed visualvm, check the attached screenshot for heap dump details.

vmassol · February 14, 2022, 2:24pm

ok thanks.

Some more ideas:

What’s the max JVM memory you’ve tried to setup in your xwiki instance to import the 1GB and 2GB files?
Have you tried modifying your XAR so that it’s smaller to see if it imports fine?

IMO the issue is that you need N times the size of your XAR in free memory in order to be able to import it (N could be > 1). I’m a bit surprised though since we refactored our import feature a long time ago to not require much memory. This is why I’m asking the 2 questions above.

On your screenshot, I don’t see how much free memory you have in your xwiki instance you’re importing into.

We’d need to get our hands on a large XAR (or generate one) to try to reproduce but in the meantime, it would help if you could try either increasing the memory more or testing with a smaller XAR to see what’s the limit with 3GB of JVM memory.

Also maybe @tmortagne knows more about the memory required for imports (note that’s he’s not available until early next week though).