This problem also occurs on a “vanilla” XWiki install on Windows as demonstrated in my video. If XWiki is deployed, Tomcat shuts down very slowly. If I remove the xwiki directory from Tomcat, Tomcat shutdown is very fast. Below is the Tomcat the log data from when Tomcat is stopped when XWiki 13.0 is deployed on a Windows server running Tomcat 9.0.43 (Windows Server 2016 x64).
First there are a bunch of memory leak warnings, but scroll to the end for another interesting message: Illegal access: this web application instance has been stopped already. Could not load [org.jboss.threads.EnhancedQueueExecutor$2].
Log information follows:
17-Feb-2021 10:45:54.292 INFO [Thread-36] org.apache.catalina.core.StandardService.stopInternal Stopping service [Catalina]
17-Feb-2021 10:45:59.088 WARNING [Thread-36] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The web application [xwiki] appears to have started a thread named [RxSchedulerPurge-1] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:
java.base@11.0.10/jdk.internal.misc.Unsafe.park(Native Method)
java.base@11.0.10/java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source)
java.base@11.0.10/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(Unknown Source)
java.base@11.0.10/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown Source)
java.base@11.0.10/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown Source)
java.base@11.0.10/java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)
java.base@11.0.10/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
java.base@11.0.10/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
java.base@11.0.10/java.lang.Thread.run(Unknown Source)
17-Feb-2021 10:45:59.089 WARNING [Thread-36] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The web application [xwiki] appears to have started a thread named [RxCachedWorkerPoolEvictor-1] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:
java.base@11.0.10/jdk.internal.misc.Unsafe.park(Native Method)
java.base@11.0.10/java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source)
java.base@11.0.10/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(Unknown Source)
java.base@11.0.10/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown Source)
java.base@11.0.10/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown Source)
java.base@11.0.10/java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)
java.base@11.0.10/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
java.base@11.0.10/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
java.base@11.0.10/java.lang.Thread.run(Unknown Source)
17-Feb-2021 10:45:59.090 WARNING [Thread-36] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The web application [xwiki] appears to have started a thread named [expiration-thread--p5-t1] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:
java.base@11.0.10/jdk.internal.misc.Unsafe.park(Native Method)
java.base@11.0.10/java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source)
java.base@11.0.10/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(Unknown Source)
java.base@11.0.10/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown Source)
java.base@11.0.10/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown Source)
java.base@11.0.10/java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)
java.base@11.0.10/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
java.base@11.0.10/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
java.base@11.0.10/java.lang.Thread.run(Unknown Source)
17-Feb-2021 10:45:59.091 WARNING [Thread-36] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The web application [xwiki] appears to have started a thread named [searcherExecutor-12-thread-1] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:
java.base@11.0.10/jdk.internal.misc.Unsafe.park(Native Method)
java.base@11.0.10/java.util.concurrent.locks.LockSupport.park(Unknown Source)
java.base@11.0.10/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(Unknown Source)
java.base@11.0.10/java.util.concurrent.LinkedBlockingQueue.take(Unknown Source)
java.base@11.0.10/java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)
java.base@11.0.10/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
java.base@11.0.10/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
java.base@11.0.10/java.lang.Thread.run(Unknown Source)
17-Feb-2021 10:45:59.092 WARNING [Thread-36] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The web application [xwiki] appears to have started a thread named [searcherExecutor-10-thread-1] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:
java.base@11.0.10/jdk.internal.misc.Unsafe.park(Native Method)
java.base@11.0.10/java.util.concurrent.locks.LockSupport.park(Unknown Source)
java.base@11.0.10/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(Unknown Source)
java.base@11.0.10/java.util.concurrent.LinkedBlockingQueue.take(Unknown Source)
java.base@11.0.10/java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)
java.base@11.0.10/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
java.base@11.0.10/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
java.base@11.0.10/java.lang.Thread.run(Unknown Source)
17-Feb-2021 10:45:59.094 WARNING [Thread-36] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The web application [xwiki] appears to have started a thread named [searcherExecutor-11-thread-1] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:
java.base@11.0.10/jdk.internal.misc.Unsafe.park(Native Method)
java.base@11.0.10/java.util.concurrent.locks.LockSupport.park(Unknown Source)
java.base@11.0.10/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(Unknown Source)
java.base@11.0.10/java.util.concurrent.LinkedBlockingQueue.take(Unknown Source)
java.base@11.0.10/java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)
java.base@11.0.10/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
java.base@11.0.10/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
java.base@11.0.10/java.lang.Thread.run(Unknown Source)
17-Feb-2021 10:45:59.094 WARNING [Thread-36] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The web application [xwiki] appears to have started a thread named [searcherExecutor-25-thread-1] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:
java.base@11.0.10/jdk.internal.misc.Unsafe.park(Native Method)
java.base@11.0.10/java.util.concurrent.locks.LockSupport.park(Unknown Source)
java.base@11.0.10/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(Unknown Source)
java.base@11.0.10/java.util.concurrent.LinkedBlockingQueue.take(Unknown Source)
java.base@11.0.10/java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)
java.base@11.0.10/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
java.base@11.0.10/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
java.base@11.0.10/java.lang.Thread.run(Unknown Source)
17-Feb-2021 10:45:59.094 WARNING [Thread-36] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The web application [xwiki] appears to have started a thread named [non-blocking-thread--p2-t1] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:
java.base@11.0.10/jdk.internal.misc.Unsafe.park(Native Method)
java.base@11.0.10/java.util.concurrent.locks.LockSupport.park(Unknown Source)
java.base@11.0.10/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(Unknown Source)
java.base@11.0.10/java.util.concurrent.LinkedBlockingQueue.take(Unknown Source)
java.base@11.0.10/java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)
java.base@11.0.10/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
java.base@11.0.10/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
java.base@11.0.10/java.lang.Thread.run(Unknown Source)
17-Feb-2021 10:45:59.095 WARNING [Thread-36] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The web application [xwiki] appears to have started a thread named [non-blocking-thread--p2-t2] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:
java.base@11.0.10/jdk.internal.misc.Unsafe.park(Native Method)
java.base@11.0.10/java.util.concurrent.locks.LockSupport.park(Unknown Source)
java.base@11.0.10/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(Unknown Source)
java.base@11.0.10/java.util.concurrent.LinkedBlockingQueue.take(Unknown Source)
java.base@11.0.10/java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)
java.base@11.0.10/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
java.base@11.0.10/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
java.base@11.0.10/java.lang.Thread.run(Unknown Source)
17-Feb-2021 10:45:59.099 WARNING [Thread-36] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The web application [xwiki] appears to have started a thread named [blocking-thread--p3-t1] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:
java.base@11.0.10/jdk.internal.misc.Unsafe.park(Native Method)
java.base@11.0.10/java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source)
org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1421)
java.base@11.0.10/java.lang.Thread.run(Unknown Source)
17-Feb-2021 10:45:59.100 WARNING [Thread-36] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The web application [xwiki] appears to have started a thread named [blocking-thread--p3-t2] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:
java.base@11.0.10/jdk.internal.misc.Unsafe.park(Native Method)
java.base@11.0.10/java.util.concurrent.locks.LockSupport.park(Unknown Source)
org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1409)
java.base@11.0.10/java.lang.Thread.run(Unknown Source)
17-Feb-2021 10:45:59.101 SEVERE [Thread-36] org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks The web application [xwiki] created a ThreadLocal with key of type [java.lang.ThreadLocal] (value [java.lang.ThreadLocal@4098f395]) and a value of type [org.codehaus.plexus.classworlds.realm.ClassRealm] (value [ClassRealm[plexus.core, parent: null]]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
17-Feb-2021 10:45:59.103 SEVERE [Thread-36] org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks The web application [xwiki] created a ThreadLocal with key of type [java.lang.ThreadLocal] (value [java.lang.ThreadLocal@2287e8e7]) and a value of type [com.google.javascript.jscomp.Tracer.ThreadTrace] (value []) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
17-Feb-2021 10:45:59.105 SEVERE [Thread-36] org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks The web application [xwiki] created a ThreadLocal with key of type [java.lang.ThreadLocal] (value [java.lang.ThreadLocal@78055235]) and a value of type [org.restlet.ext.jaxrs.internal.core.MatchedInfo] (value [org.restlet.ext.jaxrs.internal.core.MatchedInfo@1d8bd379]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
17-Feb-2021 10:45:59.105 SEVERE [Thread-36] org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks The web application [xwiki] created a ThreadLocal with key of type [java.lang.ThreadLocal] (value [java.lang.ThreadLocal@2287e8e7]) and a value of type [com.google.javascript.jscomp.Tracer.ThreadTrace] (value []) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
17-Feb-2021 10:45:59.105 SEVERE [Thread-36] org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks The web application [xwiki] created a ThreadLocal with key of type [java.lang.ThreadLocal] (value [java.lang.ThreadLocal@2287e8e7]) and a value of type [com.google.javascript.jscomp.Tracer.ThreadTrace] (value []) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
17-Feb-2021 10:45:59.106 SEVERE [Thread-36] org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks The web application [xwiki] created a ThreadLocal with key of type [java.lang.ThreadLocal] (value [java.lang.ThreadLocal@78055235]) and a value of type [org.restlet.ext.jaxrs.internal.core.MatchedInfo] (value [org.restlet.ext.jaxrs.internal.core.MatchedInfo@6851551b]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
17-Feb-2021 10:45:59.106 SEVERE [Thread-36] org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks The web application [xwiki] created a ThreadLocal with key of type [java.lang.ThreadLocal] (value [java.lang.ThreadLocal@2287e8e7]) and a value of type [com.google.javascript.jscomp.Tracer.ThreadTrace] (value []) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
17-Feb-2021 10:45:59.106 SEVERE [Thread-36] org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks The web application [xwiki] created a ThreadLocal with key of type [java.lang.ThreadLocal] (value [java.lang.ThreadLocal@2287e8e7]) and a value of type [com.google.javascript.jscomp.Tracer.ThreadTrace] (value []) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
17-Feb-2021 10:45:59.108 SEVERE [Thread-36] org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks The web application [xwiki] created a ThreadLocal with key of type [java.lang.ThreadLocal] (value [java.lang.ThreadLocal@3a9736de]) and a value of type [java.util.Stack] (value [[org.xwiki.context.ExecutionContext@79c43aee]]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
17-Feb-2021 10:45:59.108 SEVERE [Thread-36] org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks The web application [xwiki] created a ThreadLocal with key of type [java.lang.ThreadLocal] (value [java.lang.ThreadLocal@1708b903]) and a value of type [java.util.Stack] (value [[org.xwiki.container.servlet.ServletResponse@7513df74]]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
17-Feb-2021 10:45:59.108 SEVERE [Thread-36] org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks The web application [xwiki] created a ThreadLocal with key of type [java.lang.ThreadLocal] (value [java.lang.ThreadLocal@3a9736de]) and a value of type [java.util.Stack] (value [[org.xwiki.context.ExecutionContext@76b136ce]]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
17-Feb-2021 10:45:59.108 SEVERE [Thread-36] org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks The web application [xwiki] created a ThreadLocal with key of type [java.lang.ThreadLocal] (value [java.lang.ThreadLocal@2efa71bc]) and a value of type [java.util.Stack] (value [[org.xwiki.container.servlet.ServletRequest@7ce805c1]]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
17-Feb-2021 10:45:59.109 SEVERE [Thread-36] org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks The web application [xwiki] created a ThreadLocal with key of type [java.lang.ThreadLocal] (value [java.lang.ThreadLocal@2efa71bc]) and a value of type [java.util.Stack] (value [[org.xwiki.container.servlet.ServletRequest@17eb9d57]]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
17-Feb-2021 10:45:59.109 SEVERE [Thread-36] org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks The web application [xwiki] created a ThreadLocal with key of type [java.lang.ThreadLocal] (value [java.lang.ThreadLocal@1708b903]) and a value of type [java.util.Stack] (value [[org.xwiki.container.servlet.ServletResponse@6893ce22]]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
17-Feb-2021 10:45:59.109 SEVERE [Thread-36] org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks The web application [xwiki] created a ThreadLocal with key of type [java.lang.ThreadLocal] (value [java.lang.ThreadLocal@1599351d]) and a value of type [org.apache.solr.request.SolrRequestInfo] (value [org.apache.solr.request.SolrRequestInfo@2963ec62]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
Then after the memory leak messages, the log has this:
17-Feb-2021 10:45:59.222 INFO [Thread-36] org.apache.coyote.AbstractProtocol.stop Stopping ProtocolHandler ["http-nio-8080"]
17-Feb-2021 10:45:59.223 INFO [blocking-thread--p3-t2] org.apache.catalina.loader.WebappClassLoaderBase.checkStateForResourceLoading Illegal access: this web application instance has been stopped already. Could not load [org.jboss.threads.EnhancedQueueExecutor$2]. The following stack trace is thrown for debugging purposes as well as to attempt to terminate the thread which caused the illegal access.
java.lang.IllegalStateException: Illegal access: this web application instance has been stopped already. Could not load [org.jboss.threads.EnhancedQueueExecutor$2]. The following stack trace is thrown for debugging purposes as well as to attempt to terminate the thread which caused the illegal access.
at org.apache.catalina.loader.WebappClassLoaderBase.checkStateForResourceLoading(WebappClassLoaderBase.java:1384)
at org.apache.catalina.loader.WebappClassLoaderBase.checkStateForClassLoading(WebappClassLoaderBase.java:1372)
at org.apache.catalina.loader.WebappClassLoaderBase.loadClass(WebappClassLoaderBase.java:1225)
at org.apache.catalina.loader.WebappClassLoaderBase.loadClass(WebappClassLoaderBase.java:1187)
at org.jboss.threads.EnhancedQueueExecutor.completeTermination(EnhancedQueueExecutor.java:1770)
at org.jboss.threads.EnhancedQueueExecutor.tryDeallocateThread(EnhancedQueueExecutor.java:1583)
at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1396)
at java.base/java.lang.Thread.run(Unknown Source)
17-Feb-2021 10:45:59.235 INFO [Thread-36] org.apache.coyote.AbstractProtocol.stop Stopping ProtocolHandler ["https-openssl-apr-443"]
17-Feb-2021 10:45:59.247 INFO [Thread-36] org.apache.coyote.AbstractProtocol.destroy Destroying ProtocolHandler ["http-nio-8080"]
17-Feb-2021 10:45:59.250 INFO [Thread-36] org.apache.coyote.AbstractProtocol.destroy Destroying ProtocolHandler ["https-openssl-apr-443"]
The Windows service doesn’t show as “stopped” (i.e., the service remains in the “stopping” state) until about 65 seconds after Tomcat writes the last line to the log. It does not appear to be related to the speed of the machine – this info is taken from a relatively slow VM (for testing), but I see the same behavior (wait a little more than 60 seconds to stop) on other fast production VMs (with much more memory and faster disks).
Any ideas?
