Shutting down Tomcat with errors (The web application [xwiki] created a ThreadLocal ...)

Help / Discuss
,
1

When I shut down tomcat with sudo service tomcat9 stop or shutdown my server with sudo reboot the following errors occur. There are threads which were started by XWiki but couldn’t be stopped when Tomcat was stopped.

Here is my log file /var/log/tomcat9/catalina.log

29-May-2020 21:50:59.499 INFO [Thread-14] org.apache.coyote.AbstractProtocol.pause Pausing ProtocolHandler ["http-nio-8080"]
29-May-2020 21:50:59.505 INFO [Thread-14] org.apache.coyote.AbstractProtocol.pause Pausing ProtocolHandler ["ajp-nio-0.0.0.0-8009"]
29-May-2020 21:50:59.507 INFO [Thread-14] org.apache.catalina.core.StandardService.stopInternal Stopping service [Catalina]
29-May-2020 21:50:59.831 WARNING [Thread-14] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesJdbc The web application [xwiki] registered the JDBC driver [com.mysql.jdbc.Driver] but failed to unregister it when the web application was stopped. To prevent a memory leak, the JDBC Driver has been forcibly unregistered.
29-May-2020 21:50:59.832 WARNING [Thread-14] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The web application [xwiki] appears to have started a thread named [searcherExecutor-10-thread-1] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:
 sun.misc.Unsafe.park(Native Method)
 java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
 java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
 java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
 java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
 java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
 java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
 java.lang.Thread.run(Thread.java:748)
29-May-2020 21:50:59.833 WARNING [Thread-14] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The web application [xwiki] appears to have started a thread named [mysql-cj-abandoned-connection-cleanup] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:
 java.lang.Object.wait(Native Method)
 java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:144)
 com.mysql.jdbc.AbandonedConnectionCleanupThread.run(AbandonedConnectionCleanupThread.java:80)
 java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
 java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
 java.lang.Thread.run(Thread.java:748)
29-May-2020 21:50:59.834 SEVERE [Thread-14] org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks The web application [xwiki] created a ThreadLocal with key of type [java.lang.ThreadLocal] (value [java.lang.ThreadLocal@69da3f8]) and a value of type [org.codehaus.plexus.classworlds.realm.ClassRealm] (value [ClassRealm[plexus.core, parent: null]]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
29-May-2020 21:50:59.841 SEVERE [Thread-14] org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks The web application [xwiki] created a ThreadLocal with key of type [java.lang.ThreadLocal] (value [java.lang.ThreadLocal@7da24a18]) and a value of type [org.restlet.ext.jaxrs.internal.core.MatchedInfo] (value [org.restlet.ext.jaxrs.internal.core.MatchedInfo@733d858d]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
29-May-2020 21:50:59.841 SEVERE [Thread-14] org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks The web application [xwiki] created a ThreadLocal with key of type [java.lang.ThreadLocal] (value [java.lang.ThreadLocal@105c7268]) and a value of type [org.restlet.ext.jaxrs.internal.core.MatchedInfo] (value [org.restlet.ext.jaxrs.internal.core.MatchedInfo@517b0e12]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
29-May-2020 21:50:59.842 SEVERE [Thread-14] org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks The web application [xwiki] created a ThreadLocal with key of type [java.lang.ThreadLocal] (value [java.lang.ThreadLocal@7da24a18]) and a value of type [org.restlet.ext.jaxrs.internal.core.MatchedInfo] (value [org.restlet.ext.jaxrs.internal.core.MatchedInfo@3e666caf]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
29-May-2020 21:50:59.842 SEVERE [Thread-14] org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks The web application [xwiki] created a ThreadLocal with key of type [java.lang.ThreadLocal] (value [java.lang.ThreadLocal@21689703]) and a value of type [java.util.Stack] (value [[org.xwiki.context.ExecutionContext@52d46e74]]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
29-May-2020 21:50:59.842 SEVERE [Thread-14] org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks The web application [xwiki] created a ThreadLocal with key of type [java.lang.ThreadLocal] (value [java.lang.ThreadLocal@5bd6f1a2]) and a value of type [java.util.Stack] (value [[org.xwiki.container.servlet.ServletRequest@56d8ed63]]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
29-May-2020 21:50:59.843 SEVERE [Thread-14] org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks The web application [xwiki] created a ThreadLocal with key of type [java.lang.ThreadLocal] (value [java.lang.ThreadLocal@336db834]) and a value of type [java.util.Stack] (value [[org.xwiki.container.servlet.ServletResponse@423bd7e3]]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
29-May-2020 21:50:59.843 SEVERE [Thread-14] org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks The web application [xwiki] created a ThreadLocal with key of type [java.lang.ThreadLocal] (value [java.lang.ThreadLocal@21689703]) and a value of type [java.util.Stack] (value [[org.xwiki.context.ExecutionContext@2e8c75e7]]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
29-May-2020 21:50:59.843 SEVERE [Thread-14] org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks The web application [xwiki] created a ThreadLocal with key of type [java.lang.ThreadLocal] (value [java.lang.ThreadLocal@336db834]) and a value of type [java.util.Stack] (value [[org.xwiki.container.servlet.ServletResponse@3fae36a0]]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
29-May-2020 21:50:59.844 SEVERE [Thread-14] org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks The web application [xwiki] created a ThreadLocal with key of type [java.lang.ThreadLocal] (value [java.lang.ThreadLocal@5bd6f1a2]) and a value of type [java.util.Stack] (value [[org.xwiki.container.servlet.ServletRequest@615c9579]]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
29-May-2020 21:50:59.844 SEVERE [Thread-14] org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks The web application [xwiki] created a ThreadLocal with key of type [java.lang.ThreadLocal] (value [java.lang.ThreadLocal@21689703]) and a value of type [java.util.Stack] (value [[org.xwiki.context.ExecutionContext@7dc4c27e]]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
29-May-2020 21:50:59.897 INFO [Thread-14] org.apache.coyote.AbstractProtocol.stop Stopping ProtocolHandler ["http-nio-8080"]
29-May-2020 21:50:59.904 INFO [Thread-14] org.apache.coyote.AbstractProtocol.stop Stopping ProtocolHandler ["ajp-nio-0.0.0.0-8009"]
29-May-2020 21:50:59.929 INFO [Thread-14] org.apache.coyote.AbstractProtocol.destroy Destroying ProtocolHandler ["http-nio-8080"]
29-May-2020 21:50:59.930 INFO [Thread-14] org.apache.coyote.AbstractProtocol.destroy Destroying ProtocolHandler ["ajp-nio-0.0.0.0-8009"]

My problem is that tomcat9 is not started after rebooting. I guess those errors are the cause so that I can’t ignore them.

Versions
xwiki-tomcat9-mysql 12.4
Tomcat 9.0.16-3ubuntu0.18.04.1
Ubuntu 18.04.4 LTS

2

Hi,

This is a known limitation, see Loading... for details and status.

I don’t understand what you mean. Stopping and restarting tomcat should work just fine. What you shouldn’t do is stop the xwiki webapp and restart it using the tomcat manager. But if you stop tomcat itself it’ll work fine.

Thanks

1 Like
3

Thanks Vincent for your fast reply!

I think I’m wrong that those errors message keep my Tomcat from starting automatically after reboot.

But my Tomcat doesn’t start after booting and I’d like to fix it.

My Tomcat service runs fine. I can start and stop it manually without a problem.

I tried to enable the service with sudo systemctl enable tomcat9 but it says

Failed to lookup unit file state: Invalid argument

I guess I found the problem. It’s the link of the systemd drop-in

$ ls -al /etc/systemd/system/tomcat9.service.d
total 8
drwxr-xr-x 2 root root 4096 May 29 23:19 .
drwxr-xr-x 19 root root 4096 May 24 14:18 ..
lrwxrwxrwx 1 root root 37 May 26 14:50 xwiki-tomcat9-systemd.conf -> /etc/xwiki/xwiki-tomcat9-systemd.conf

The link is not allowed as it points to /etc. I found a hint about this rule here: install: follow unit file symlinks in /usr, but not /etc when looking for [Install] data and more by poettering · Pull Request #1515 · systemd/systemd · GitHub

After removing the symlink and copying the former linked file to the same location everything works fine:

$ ls -al /etc/systemd/system/tomcat9.service.d
total 12
drwxr-xr-x  2 root root 4096 May 30 21:40 .
drwxr-xr-x 19 root root 4096 May 24 14:18 ..
-rw-r--r--  1 root root  116 May 30 21:40 xwiki-tomcat9-systemd.conf
4

OK thanks for the debug @bjoerne I will fix that in the Debian package. It’s strange that service xwiki start work fine and not the boot.

Just deployed patched 12.4+1 and 11.10.5+1 Debian packages.

5

Thank you, @vmassol. Stopping and starting the tomcat worked straight away.