XWiki/Tomcat/slow shutdown/Windows service

Help / Discuss
1

We are using XWiki 12.10 on Windows and have been (for a couple of versions now) experiencing a problem where the Tomcat service running XWiki takes over 60 seconds to shutdown. Before the problem started, it used to take only a few seconds.

The stdout, stderr, and catalina log data from a sample start and stop are below.

stdout:

2020-12-11 16:14:15 Apache Commons Daemon procrun stdout initialized.
2020-12-11 16:14:27,942 [main] INFO iPropertiesConfigurationSource - Loading [xwiki.properties] from [file:/C:/Program%20Files/Apache%20Tomcat/webapps/xwiki/WEB-INF/xwiki.properties]
2020-12-11 16:14:28,238 [main] INFO o.x.e.i.ServletEnvironment - Using permanent directory [D:\ProgramData\xwiki]
2020-12-11 16:14:34,020 [main] WARN .x.e.m.i.MavenExtensionScanner - Collision between core extension [org.slf4j:jcl-over-slf4j/1.7.29 (jar:file:/C:/Program%20Files/Apache%20Tomcat/lib/jcl-over-slf4j-1.7.29.jar!/META-INF/maven/org.slf4j/jcl-over-slf4j/pom.xml)] and [org.slf4j:jcl-over-slf4j/1.7.30 (file:/C:/Program%20Files/Apache%20Tomcat/webapps/xwiki/WEB-INF/lib/jcl-over-slf4j-1.7.30.xed)]
2020-12-11 16:14:34,020 [main] WARN .x.e.m.i.MavenExtensionScanner - [org.slf4j:jcl-over-slf4j/1.7.30 (file:/C:/Program%20Files/Apache%20Tomcat/webapps/xwiki/WEB-INF/lib/jcl-over-slf4j-1.7.30.xed)] is selected
2020-12-11 16:14:34,020 [main] WARN .x.e.m.i.MavenExtensionScanner - Collision between core extension [org.slf4j:slf4j-api/1.7.29 (jar:file:/C:/Program%20Files/Apache%20Tomcat/lib/slf4j-api-1.7.29.jar!/META-INF/maven/org.slf4j/slf4j-api/pom.xml)] and [org.slf4j:slf4j-api/1.7.30 (file:/C:/Program%20Files/Apache%20Tomcat/webapps/xwiki/WEB-INF/lib/slf4j-api-1.7.30.xed)]
2020-12-11 16:14:34,020 [main] WARN .x.e.m.i.MavenExtensionScanner - [org.slf4j:slf4j-api/1.7.30 (file:/C:/Program%20Files/Apache%20Tomcat/webapps/xwiki/WEB-INF/lib/slf4j-api-1.7.30.xed)] is selected
2020-12-11 16:14:41,442 [main] INFO o.x.s.s.i.EmbeddedSolr - Starting embedded Solr server…
2020-12-11 16:14:41,442 [main] INFO o.x.s.s.i.EmbeddedSolr - Using Solr home directory: [D:\ProgramData\xwiki\store\solr]
2020-12-11 16:14:43,567 [main] WARN o.e.j.u.s.S.config - Trusting all certificates configured for Client@49bf8d66[provider=null,keyStore=null,trustStore=null]
2020-12-11 16:14:43,567 [main] WARN o.e.j.u.s.S.config - No Client EndPointIdentificationAlgorithm configured for Client@49bf8d66[provider=null,keyStore=null,trustStore=null]
2020-12-11 16:14:43,974 [main] WARN o.e.j.u.s.S.config - Trusting all certificates configured for Client@3b1e2d5d[provider=null,keyStore=null,trustStore=null]
2020-12-11 16:14:43,974 [main] WARN o.e.j.u.s.S.config - No Client EndPointIdentificationAlgorithm configured for Client@3b1e2d5d[provider=null,keyStore=null,trustStore=null]
2020-12-11 16:14:44,215 [main] WARN o.a.s.c.CoreContainer - Not all security plugins configured! authentication=disabled authorization=disabled. Solr is only as secure as you make it. Consider configuring authentication/authorization before exposing Solr to users internal or external. See Securing Solr :: Apache Solr Reference Guide for more info
2020-12-11 16:14:50,215 [main] INFO o.x.s.s.i.EmbeddedSolr - Started embedded Solr server.
2020-12-11 16:14:54,184 [main] INFO o.x.s.f.i.FilesystemStoreTools - Using filesystem store directory [D:\ProgramData\xwiki\store\file]
2020-12-11 16:14:56,439 [main] INFO o.x.o.i.s.DefaultOfficeServer - Open Office instance started.
2020-12-11 16:15:20,423 [XWiki initialization] INFO .HibernateDataMigrationManager - Storage schema updates and data migrations are enabled
2020-12-11 16:15:20,845 [XWiki initialization] INFO .HibernateDataMigrationManager - No data migration to apply for wiki [xwiki] currently in version [120901000]
2020-12-11 16:15:20,845 [XWiki initialization] INFO .HibernateDataMigrationManager - Checking Hibernate mapping and updating schema if needed for wiki [xwiki]
2020-12-11 16:15:35,783 [extendion.index job group daemon thread - org.xwiki.extension.index.internal.job.ExtensionIndexJob@a0cf545([extension, index])] ERROR o.x.e.i.i.j.ExtensionIndexJob - Exception thrown during job execution
java.lang.NullPointerException: null
at org.xwiki.extension.internal.converter.ExtensionAuthorConverter.toString(ExtensionAuthorConverter.java:152)
at org.xwiki.extension.internal.converter.ExtensionAuthorConverter.convertToString(ExtensionAuthorConverter.java:186)
at org.xwiki.extension.internal.converter.ExtensionAuthorConverter.convertToString(ExtensionAuthorConverter.java:43)
at org.xwiki.properties.converter.AbstractConverter.convert(AbstractConverter.java:46)
at org.xwiki.search.solr.internal.DefaultSolrUtils.toString(DefaultSolrUtils.java:416)
at org.xwiki.search.solr.internal.DefaultSolrUtils.lambda$setString$2(DefaultSolrUtils.java:396)
at java.base/java.util.ArrayList.forEach(Unknown Source)
at java.base/java.util.Collections$UnmodifiableCollection.forEach(Unknown Source)
at org.xwiki.search.solr.internal.DefaultSolrUtils.setString(DefaultSolrUtils.java:396)
at org.xwiki.extension.index.internal.ExtensionIndexStore.add(ExtensionIndexStore.java:450)
at org.xwiki.extension.index.internal.job.ExtensionIndexJob.addLocalExtensions(ExtensionIndexJob.java:580)
at org.xwiki.extension.index.internal.job.ExtensionIndexJob.runInternal(ExtensionIndexJob.java:202)
at org.xwiki.job.AbstractJob.runInContext(AbstractJob.java:243)
at org.xwiki.job.AbstractJob.run(AbstractJob.java:220)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.base/java.lang.Thread.run(Unknown Source)
2020-12-11 16:15:36,486 [solr/indexer job group daemon thread - org.xwiki.search.solr.internal.job.IndexerJob@3d9337dc([solr, indexer])] INFO o.x.s.s.i.j.IndexerJob - Starting job of type [solr.indexer] with identifier [[solr, indexer]]
2020-12-11 16:15:38,814 [solr/indexer job group daemon thread - org.xwiki.search.solr.internal.job.IndexerJob@3d9337dc([solr, indexer])] INFO o.x.s.s.i.j.IndexerJob - 612 documents added, 389 deleted and 0 updated during the synchronization of the Solr index.
2020-12-11 16:15:38,830 [solr/indexer job group daemon thread - org.xwiki.search.solr.internal.job.IndexerJob@3d9337dc([solr, indexer])] INFO o.x.s.s.i.j.IndexerJob - Finished job of type [solr.indexer] with identifier [[solr, indexer]]
2020-12-11 16:17:28,968 [Thread-88] INFO o.x.o.i.s.DefaultOfficeServer - Open Office instance stopped.

stderr:

2020-12-11 16:14:15 Apache Commons Daemon procrun stderr initialized.
WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by com.thoughtworks.xstream.converters.reflection.FieldDictionary (file:/C:/Program%20Files/Apache%20Tomcat/webapps/xwiki/WEB-INF/lib/xstream-1.4.14.jar) to field java.util.Collections$UnmodifiableMap.serialVersionUID
WARNING: Please consider reporting this to the maintainers of com.thoughtworks.xstream.converters.reflection.FieldDictionary
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release
ticBinder]
[https-openssl-apr-443-exec-7 - https://servername/xwiki/bin/view/Main/] INFO waffle.apache.NegotiateAuthenticator - successfully logged in user: domain\username
[Thread-88] INFO waffle.apache.NegotiateAuthenticator - [waffle.apache.NegotiateAuthenticator] stopped

catalina log data:

11-Dec-2020 16:14:16.254 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Server version name: Apache Tomcat/9.0.40
11-Dec-2020 16:14:16.254 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Server built: Nov 12 2020 15:35:02 UTC
11-Dec-2020 16:14:16.254 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Server version number: 9.0.40.0
11-Dec-2020 16:14:16.254 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log OS Name: Windows Server 2012 R2
11-Dec-2020 16:14:16.254 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log OS Version: 6.3
11-Dec-2020 16:14:16.254 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Architecture: amd64
11-Dec-2020 16:14:16.254 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Java Home: C:\Program Files\AdoptOpenJDK\JRE11
11-Dec-2020 16:14:16.254 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log JVM Version: 11.0.9.1+1
11-Dec-2020 16:14:16.254 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log JVM Vendor: AdoptOpenJDK
11-Dec-2020 16:14:16.254 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log CATALINA_BASE: C:\Program Files\Apache Tomcat
11-Dec-2020 16:14:16.254 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log CATALINA_HOME: C:\Program Files\Apache Tomcat
11-Dec-2020 16:14:16.285 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Dcatalina.home=C:\Program Files\Apache Tomcat
11-Dec-2020 16:14:16.285 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Dcatalina.base=C:\Program Files\Apache Tomcat
11-Dec-2020 16:14:16.285 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Djava.io.tmpdir=C:\Program Files\Apache Tomcat\temp
11-Dec-2020 16:14:16.285 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager
11-Dec-2020 16:14:16.285 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Djava.util.logging.config.file=C:\Program Files\Apache Tomcat\conf\logging.properties
11-Dec-2020 16:14:16.285 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Djavax.net.ssl.trustStoreType=WINDOWS-ROOT
11-Dec-2020 16:14:16.285 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Dorg.apache.tomcat.util.buf.UDecoder.ALLOW_ENCODED_SLASH=true
11-Dec-2020 16:14:16.285 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Dorg.apache.catalina.connector.CoyoteAdapter.ALLOW_BACKSLASH=true
11-Dec-2020 16:14:16.285 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: --add-opens=java.base/java.lang=ALL-UNNAMED
11-Dec-2020 16:14:16.285 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: --add-opens=java.base/java.io=ALL-UNNAMED
11-Dec-2020 16:14:16.285 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: --add-opens=java.rmi/sun.rmi.transport=ALL-UNNAMED
11-Dec-2020 16:14:16.285 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: exit
11-Dec-2020 16:14:16.285 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: abort
11-Dec-2020 16:14:16.285 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Xms1792m
11-Dec-2020 16:14:16.285 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Xmx1792m
11-Dec-2020 16:14:16.301 INFO [main] org.apache.catalina.core.AprLifecycleListener.lifecycleEvent Loaded Apache Tomcat Native library [1.2.25] using APR version [1.7.0].
11-Dec-2020 16:14:16.301 INFO [main] org.apache.catalina.core.AprLifecycleListener.lifecycleEvent APR capabilities: IPv6 [true], sendfile [true], accept filters [false], random [true].
11-Dec-2020 16:14:16.301 INFO [main] org.apache.catalina.core.AprLifecycleListener.lifecycleEvent APR/OpenSSL configuration: useAprConnector [false], useOpenSSL [true]
11-Dec-2020 16:14:16.301 INFO [main] org.apache.catalina.core.AprLifecycleListener.initializeSSL OpenSSL successfully initialized [OpenSSL 1.1.1g 21 Apr 2020]
11-Dec-2020 16:14:16.894 INFO [main] org.apache.coyote.AbstractProtocol.init Initializing ProtocolHandler [“http-nio-80”]
11-Dec-2020 16:14:17.051 INFO [main] org.apache.coyote.http11.AbstractHttp11Protocol.configureUpgradeProtocol The [“https-openssl-apr-443”] connector has been configured to support negotiation to [h2] via ALPN
11-Dec-2020 16:14:17.051 INFO [main] org.apache.coyote.AbstractProtocol.init Initializing ProtocolHandler [“https-openssl-apr-443”]
11-Dec-2020 16:14:17.207 INFO [main] org.apache.catalina.startup.Catalina.load Server initialization in [1313] milliseconds
11-Dec-2020 16:14:17.301 INFO [main] org.apache.catalina.core.StandardService.startInternal Starting service [Catalina]
11-Dec-2020 16:14:17.301 INFO [main] org.apache.catalina.core.StandardEngine.startInternal Starting Servlet engine: [Apache Tomcat/9.0.40]
11-Dec-2020 16:14:17.332 INFO [main] org.apache.catalina.startup.HostConfig.deployDirectory Deploying web application directory [C:\Program Files\Apache Tomcat\webapps\ROOT]
11-Dec-2020 16:14:18.379 INFO [main] org.apache.jasper.servlet.TldScanner.scanJars At least one JAR was scanned for TLDs yet contained no TLDs. Enable debug logging for this logger for a complete list of JARs that were scanned but no TLDs were found in them. Skipping unneeded JARs during scanning can improve startup time and JSP compilation time.
11-Dec-2020 16:14:18.488 INFO [main] org.apache.catalina.startup.HostConfig.deployDirectory Deployment of web application directory [C:\Program Files\Apache Tomcat\webapps\ROOT] has finished in [1,156] ms
11-Dec-2020 16:14:18.488 INFO [main] org.apache.catalina.startup.HostConfig.deployDirectory Deploying web application directory [C:\Program Files\Apache Tomcat\webapps\xwiki]
11-Dec-2020 16:14:57.142 INFO [main] org.apache.catalina.startup.HostConfig.deployDirectory Deployment of web application directory [C:\Program Files\Apache Tomcat\webapps\xwiki] has finished in [38,654] ms
11-Dec-2020 16:14:57.142 INFO [main] org.apache.coyote.AbstractProtocol.start Starting ProtocolHandler [“http-nio-80”]
11-Dec-2020 16:14:57.158 INFO [main] org.apache.coyote.AbstractProtocol.start Starting ProtocolHandler [“https-openssl-apr-443”]
11-Dec-2020 16:14:57.158 INFO [main] org.apache.catalina.startup.Catalina.start Server startup in [39955] milliseconds
11-Dec-2020 16:15:57.518 INFO [https-openssl-apr-443-exec-7 - https://servername/xwiki/rest/notifications/count?media=json&userId=xwiki%3AXWiki.username&useUserPreferences=true&currentWiki=xwiki&async=true&_=1607728553528] org.apache.catalina.core.ApplicationContext.log RestletServlet: [Restlet] ServerServlet: component class is null
11-Dec-2020 16:17:28.377 INFO [Thread-88] org.apache.coyote.AbstractProtocol.pause Pausing ProtocolHandler [“http-nio-80”]
11-Dec-2020 16:17:28.429 INFO [Thread-88] org.apache.coyote.AbstractProtocol.pause Pausing ProtocolHandler [“https-openssl-apr-443”]
11-Dec-2020 16:17:28.468 INFO [Thread-88] org.apache.catalina.core.StandardService.stopInternal Stopping service [Catalina]
11-Dec-2020 16:17:29.202 WARNING [Thread-88] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The web application [xwiki] appears to have started a thread named [RxSchedulerPurge-1] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:
java.base@11.0.9.1/jdk.internal.misc.Unsafe.park(Native Method)
java.base@11.0.9.1/java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source)
java.base@11.0.9.1/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(Unknown Source)
java.base@11.0.9.1/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown Source)
java.base@11.0.9.1/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown Source)
java.base@11.0.9.1/java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)
java.base@11.0.9.1/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
java.base@11.0.9.1/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
java.base@11.0.9.1/java.lang.Thread.run(Unknown Source)
11-Dec-2020 16:17:29.202 WARNING [Thread-88] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The web application [xwiki] appears to have started a thread named [RxCachedWorkerPoolEvictor-1] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:
java.base@11.0.9.1/jdk.internal.misc.Unsafe.park(Native Method)
java.base@11.0.9.1/java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source)
java.base@11.0.9.1/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(Unknown Source)
java.base@11.0.9.1/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown Source)
java.base@11.0.9.1/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown Source)
java.base@11.0.9.1/java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)
java.base@11.0.9.1/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
java.base@11.0.9.1/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
java.base@11.0.9.1/java.lang.Thread.run(Unknown Source)
11-Dec-2020 16:17:29.202 WARNING [Thread-88] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The web application [xwiki] appears to have started a thread named [searcherExecutor-12-thread-1] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:
java.base@11.0.9.1/jdk.internal.misc.Unsafe.park(Native Method)
java.base@11.0.9.1/java.util.concurrent.locks.LockSupport.park(Unknown Source)
java.base@11.0.9.1/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(Unknown Source)
java.base@11.0.9.1/java.util.concurrent.LinkedBlockingQueue.take(Unknown Source)
java.base@11.0.9.1/java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)
java.base@11.0.9.1/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
java.base@11.0.9.1/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
java.base@11.0.9.1/java.lang.Thread.run(Unknown Source)
11-Dec-2020 16:17:29.202 WARNING [Thread-88] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The web application [xwiki] appears to have started a thread named [searcherExecutor-11-thread-1] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:
java.base@11.0.9.1/jdk.internal.misc.Unsafe.park(Native Method)
java.base@11.0.9.1/java.util.concurrent.locks.LockSupport.park(Unknown Source)
java.base@11.0.9.1/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(Unknown Source)
java.base@11.0.9.1/java.util.concurrent.LinkedBlockingQueue.take(Unknown Source)
java.base@11.0.9.1/java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)
java.base@11.0.9.1/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
java.base@11.0.9.1/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
java.base@11.0.9.1/java.lang.Thread.run(Unknown Source)
11-Dec-2020 16:17:29.202 WARNING [Thread-88] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The web application [xwiki] appears to have started a thread named [searcherExecutor-10-thread-1] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:
java.base@11.0.9.1/jdk.internal.misc.Unsafe.park(Native Method)
java.base@11.0.9.1/java.util.concurrent.locks.LockSupport.park(Unknown Source)
java.base@11.0.9.1/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(Unknown Source)
java.base@11.0.9.1/java.util.concurrent.LinkedBlockingQueue.take(Unknown Source)
java.base@11.0.9.1/java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)
java.base@11.0.9.1/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
java.base@11.0.9.1/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
java.base@11.0.9.1/java.lang.Thread.run(Unknown Source)
11-Dec-2020 16:17:29.202 WARNING [Thread-88] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The web application [xwiki] appears to have started a thread named [searcherExecutor-25-thread-1] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:
java.base@11.0.9.1/jdk.internal.misc.Unsafe.park(Native Method)
java.base@11.0.9.1/java.util.concurrent.locks.LockSupport.park(Unknown Source)
java.base@11.0.9.1/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(Unknown Source)
java.base@11.0.9.1/java.util.concurrent.LinkedBlockingQueue.take(Unknown Source)
java.base@11.0.9.1/java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)
java.base@11.0.9.1/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
java.base@11.0.9.1/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
java.base@11.0.9.1/java.lang.Thread.run(Unknown Source)
11-Dec-2020 16:17:29.202 WARNING [Thread-88] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The web application [xwiki] appears to have started a thread named [OfficeProcessThread-0] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:
java.base@11.0.9.1/jdk.internal.misc.Unsafe.park(Native Method)
java.base@11.0.9.1/java.util.concurrent.locks.LockSupport.park(Unknown Source)
java.base@11.0.9.1/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(Unknown Source)
java.base@11.0.9.1/java.util.concurrent.LinkedBlockingQueue.take(Unknown Source)
java.base@11.0.9.1/java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)
java.base@11.0.9.1/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
java.base@11.0.9.1/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
java.base@11.0.9.1/java.lang.Thread.run(Unknown Source)
11-Dec-2020 16:17:29.202 SEVERE [Thread-88] org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks The web application [xwiki] created a ThreadLocal with key of type [java.lang.ThreadLocal] (value [java.lang.ThreadLocal@7c3c21b]) and a value of type [org.codehaus.plexus.classworlds.realm.ClassRealm] (value [ClassRealm[plexus.core, parent: null]]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
11-Dec-2020 16:17:29.218 SEVERE [Thread-88] org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks The web application [xwiki] created a ThreadLocal with key of type [java.lang.ThreadLocal] (value [java.lang.ThreadLocal@2fe74811]) and a value of type [com.google.javascript.jscomp.Tracer.ThreadTrace] (value ) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
11-Dec-2020 16:17:29.218 SEVERE [Thread-88] org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks The web application [xwiki] created a ThreadLocal with key of type [java.lang.ThreadLocal] (value [java.lang.ThreadLocal@5eecf510]) and a value of type [org.restlet.ext.jaxrs.internal.core.MatchedInfo] (value [org.restlet.ext.jaxrs.internal.core.MatchedInfo@71615d23]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
11-Dec-2020 16:17:29.218 SEVERE [Thread-88] org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks The web application [xwiki] created a ThreadLocal with key of type [java.lang.ThreadLocal] (value [java.lang.ThreadLocal@5eecf510]) and a value of type [org.restlet.ext.jaxrs.internal.core.MatchedInfo] (value [org.restlet.ext.jaxrs.internal.core.MatchedInfo@21effbf4]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
11-Dec-2020 16:17:29.218 SEVERE [Thread-88] org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks The web application [xwiki] created a ThreadLocal with key of type [java.lang.ThreadLocal] (value [java.lang.ThreadLocal@776b1c21]) and a value of type [java.util.Stack] (value [[org.xwiki.context.ExecutionContext@4e1633a9]]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
11-Dec-2020 16:17:29.218 SEVERE [Thread-88] org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks The web application [xwiki] created a ThreadLocal with key of type [java.lang.ThreadLocal] (value [java.lang.ThreadLocal@776b1c21]) and a value of type [java.util.Stack] (value [[org.xwiki.context.ExecutionContext@35e5e70d]]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
11-Dec-2020 16:17:29.218 SEVERE [Thread-88] org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks The web application [xwiki] created a ThreadLocal with key of type [java.lang.ThreadLocal] (value [java.lang.ThreadLocal@6b8b9a1e]) and a value of type [java.util.Stack] (value [[org.xwiki.container.servlet.ServletRequest@44ae195a]]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
11-Dec-2020 16:17:29.218 SEVERE [Thread-88] org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks The web application [xwiki] created a ThreadLocal with key of type [java.lang.ThreadLocal] (value [java.lang.ThreadLocal@3b5ae11d]) and a value of type [java.util.Stack] (value [[org.xwiki.container.servlet.ServletResponse@211db4f4]]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
11-Dec-2020 16:17:29.249 INFO [Thread-88] org.apache.coyote.AbstractProtocol.stop Stopping ProtocolHandler [“http-nio-80”]
11-Dec-2020 16:17:29.280 INFO [Thread-88] org.apache.coyote.AbstractProtocol.stop Stopping ProtocolHandler [“https-openssl-apr-443”]
11-Dec-2020 16:17:29.358 INFO [Thread-88] org.apache.coyote.AbstractProtocol.destroy Destroying ProtocolHandler [“http-nio-80”]
11-Dec-2020 16:17:29.358 INFO [Thread-88] org.apache.coyote.AbstractProtocol.destroy Destroying ProtocolHandler [“https-openssl-apr-443”]

The interesting thing is that the service stays in a “stop pending” for about 60 more seconds after the log entries cease, then the service finally registers as stopped.

I appreciate any ideas on what might be causing this.

2

This looks like a bug. Probably something to bulletproof when xwiki is stopped. cc @tmortagne. You could open a jira about this.

Note that it’s unlikely it’s the cause of your slow stop but it’s something to fix.

I see a lot of those. This seems to suggest that you’re not stopping tomcat to start/stop XWiki but you’re using the Tomcat Manager. This is currently not well supported by XWiki and we recommend starting/stopping Tomcat to start/stop xwiki. We have some jiras open about this already.

Thanks

3

Looks like Loading....

4

Indeed looks like it, thanks Thomas.

@bstewart would be good if you could upgrade to 12.10.2 BTW since there are plenty of important issues we had in 12.10 and 12.10.1 that were fixed there and it’s now the LTS.

5

This seems to suggest that you’re not stopping tomcat to start/stop XWiki but you’re using the Tomcat Manager. This is currently not well supported by XWiki and we recommend starting/stopping Tomcat to start/stop xwiki.

If by “Tomcat Manager” you mean the webapps\manager web application in Tomcat, then no, I am not using that. I am simply stopping the Windows service (uses Apache procrun).

Bill

6

yes I meant that. What’s important is that the tomcat JVM is stopped and restarted (if the JVM is left running then you’ll get these warnings which could be real problems).

7

So i believe that your windows service is not stopping the JVM (and probably using the tomcat manager).

8

What’s important is that the tomcat JVM is stopped and restarted.

I don’t have webapps/manager installed at all and simply stop using the Windows service. This should mean that procrun is stopping the JVM properly.

9

I’ve never seen these warnings when the JVM is stopped so that should be double-checked.

10

I’ve never seen these warnings when the JVM is stopped so that should be double-checked.

The Tomcat service install parameters I’m using look like the following:

--Classpath "<tomcathome>\bin\bootstrap.jar;<tomcathome>\bin\tomcat-juli.jar"
--Jvm "<javahome>\bin\server\jvm.dll"
--JvmMs 1792
--JvmMx 1792
--JvmOptions "-Dcatalina.home=<tomcathome>;-Dcatalina.base=<tomcathome>;-Djava.io.tmpdir=<tomcathome>temp;-Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager;-Djava.util.logging.config.file=<tomcathome>\conf\logging.properties"
--JvmOptions9 "--add-opens=java.base/java.lang=ALL-UNNAMED#--add-opens=java.base/java.io=ALL-UNNAMED#--add-opens=java.rmi/sun.rmi.transport=ALL-UNNAMED"
--StartClass org.apache.catalina.startup.Bootstrap
--StartMode jvm
--StartParams start
--StartPath "<tomcathome>"
--StopClass org.apache.catalina.startup.Bootstrap
--StopMode jvm
--StopParams stop
--StopPath "<tomcathome>"

This is what was working fine until probably 2 XWiki versions ago and seems to be standard on the Windows platform for running Tomcat as a Windows service.

11

I installed a new, separate copy of Tomcat 9.0.41 into a separate directory and deployed our existing 12.10 XWiki instance to it, but this didn’t solve the issue of Tomcat stopping extremely slowly. We will try upgrading to 12.10.2 and see if upgrading fixes it.

12

I upgraded my XWiki instances to 12.10.2 but this did not fix it. It still takes about 61 seconds to stop the service.

13

Could it be coming from the service? What step is taking time in the logs? Have you enabled tomcat debug logging to have detailed logs to see what takes time (in case you don’t see it in the default logs - see https://www.xwiki.org/xwiki/bin/view/Documentation/AdminGuide/Installation/InstallationWAR/InstallationTomcat/#HSEVERE:ErrorlistenerStart)?

You could also try to start/stop xwiki without the service and see how long it takes.

Thanks

14

I added the following lines to the WEB-INF/classes/logging.properties file and restarted the service:

org.apache.catalina.core.ContainerBase.[Catalina].level = INFO org.apache.catalina.core.ContainerBase.[Catalina].handlers = java.util.logging.ConsoleHandler

I believe I already had something like this in the Tomcat conf/logging.properties file, but in any case, this is what is in the catalina log file when stopping the Tomcat service:

12-Jan-2021 08:15:48.029 INFO [Thread-60] org.apache.coyote.AbstractProtocol.pause Pausing ProtocolHandler ["http-nio-80"]
12-Jan-2021 08:15:48.076 INFO [Thread-60] org.apache.coyote.AbstractProtocol.pause Pausing ProtocolHandler ["https-openssl-apr-443"]
12-Jan-2021 08:15:48.123 INFO [Thread-60] org.apache.catalina.core.StandardService.stopInternal Stopping service [Catalina]
12-Jan-2021 08:15:48.748 WARNING [Thread-60] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The web application [xwiki] appears to have started a thread named [RxSchedulerPurge-1] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:
java.base@11.0.9.1/jdk.internal.misc.Unsafe.park(Native Method)
java.base@11.0.9.1/java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source)
java.base@11.0.9.1/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(Unknown Source)
java.base@11.0.9.1/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown Source)
java.base@11.0.9.1/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown Source)
java.base@11.0.9.1/java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)
java.base@11.0.9.1/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
java.base@11.0.9.1/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
java.base@11.0.9.1/java.lang.Thread.run(Unknown Source)
12-Jan-2021 08:15:48.748 WARNING [Thread-60] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The web application [xwiki] appears to have started a thread named [RxCachedWorkerPoolEvictor-1] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:
java.base@11.0.9.1/jdk.internal.misc.Unsafe.park(Native Method)
java.base@11.0.9.1/java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source)
java.base@11.0.9.1/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(Unknown Source)
java.base@11.0.9.1/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown Source)
java.base@11.0.9.1/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown Source)
java.base@11.0.9.1/java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)
java.base@11.0.9.1/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
java.base@11.0.9.1/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
java.base@11.0.9.1/java.lang.Thread.run(Unknown Source)
12-Jan-2021 08:15:48.748 WARNING [Thread-60] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The web application [xwiki] appears to have started a thread named [searcherExecutor-10-thread-1] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:
java.base@11.0.9.1/jdk.internal.misc.Unsafe.park(Native Method)
java.base@11.0.9.1/java.util.concurrent.locks.LockSupport.park(Unknown Source)
java.base@11.0.9.1/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(Unknown Source)
java.base@11.0.9.1/java.util.concurrent.LinkedBlockingQueue.take(Unknown Source)
java.base@11.0.9.1/java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)
java.base@11.0.9.1/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
java.base@11.0.9.1/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
java.base@11.0.9.1/java.lang.Thread.run(Unknown Source)
12-Jan-2021 08:15:48.748 WARNING [Thread-60] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The web application [xwiki] appears to have started a thread named [searcherExecutor-12-thread-1] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:
java.base@11.0.9.1/jdk.internal.misc.Unsafe.park(Native Method)
java.base@11.0.9.1/java.util.concurrent.locks.LockSupport.park(Unknown Source)
java.base@11.0.9.1/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(Unknown Source)
java.base@11.0.9.1/java.util.concurrent.LinkedBlockingQueue.take(Unknown Source)
java.base@11.0.9.1/java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)
java.base@11.0.9.1/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
java.base@11.0.9.1/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
java.base@11.0.9.1/java.lang.Thread.run(Unknown Source)
12-Jan-2021 08:15:48.748 WARNING [Thread-60] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The web application [xwiki] appears to have started a thread named [searcherExecutor-11-thread-1] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:
java.base@11.0.9.1/jdk.internal.misc.Unsafe.park(Native Method)
java.base@11.0.9.1/java.util.concurrent.locks.LockSupport.park(Unknown Source)
java.base@11.0.9.1/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(Unknown Source)
java.base@11.0.9.1/java.util.concurrent.LinkedBlockingQueue.take(Unknown Source)
java.base@11.0.9.1/java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)
java.base@11.0.9.1/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
java.base@11.0.9.1/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
java.base@11.0.9.1/java.lang.Thread.run(Unknown Source)
12-Jan-2021 08:15:48.748 WARNING [Thread-60] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The web application [xwiki] appears to have started a thread named [searcherExecutor-25-thread-1] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:
java.base@11.0.9.1/jdk.internal.misc.Unsafe.park(Native Method)
java.base@11.0.9.1/java.util.concurrent.locks.LockSupport.park(Unknown Source)
java.base@11.0.9.1/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(Unknown Source)
java.base@11.0.9.1/java.util.concurrent.LinkedBlockingQueue.take(Unknown Source)
java.base@11.0.9.1/java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)
java.base@11.0.9.1/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
java.base@11.0.9.1/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
java.base@11.0.9.1/java.lang.Thread.run(Unknown Source)
12-Jan-2021 08:15:48.748 WARNING [Thread-60] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The web application [xwiki] appears to have started a thread named [OfficeProcessThread-0] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:
java.base@11.0.9.1/jdk.internal.misc.Unsafe.park(Native Method)
java.base@11.0.9.1/java.util.concurrent.locks.LockSupport.park(Unknown Source)
java.base@11.0.9.1/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(Unknown Source)
java.base@11.0.9.1/java.util.concurrent.LinkedBlockingQueue.take(Unknown Source)
java.base@11.0.9.1/java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)
java.base@11.0.9.1/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
java.base@11.0.9.1/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
java.base@11.0.9.1/java.lang.Thread.run(Unknown Source)
12-Jan-2021 08:15:48.748 SEVERE [Thread-60] org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks The web application [xwiki] created a ThreadLocal with key of type [java.lang.ThreadLocal] (value [java.lang.ThreadLocal@17039b29]) and a value of type [org.codehaus.plexus.classworlds.realm.ClassRealm] (value [ClassRealm[plexus.core, parent: null]]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
12-Jan-2021 08:15:48.748 SEVERE [Thread-60] org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks The web application [xwiki] created a ThreadLocal with key of type [java.lang.ThreadLocal] (value [java.lang.ThreadLocal@348f93ed]) and a value of type [com.google.javascript.jscomp.Tracer.ThreadTrace] (value []) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
12-Jan-2021 08:15:48.748 SEVERE [Thread-60] org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks The web application [xwiki] created a ThreadLocal with key of type [java.lang.ThreadLocal] (value [java.lang.ThreadLocal@a9a80c]) and a value of type [java.util.Stack] (value [[org.xwiki.context.ExecutionContext@23e6f284]]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
12-Jan-2021 08:15:48.748 SEVERE [Thread-60] org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks The web application [xwiki] created a ThreadLocal with key of type [java.lang.ThreadLocal] (value [java.lang.ThreadLocal@a9a80c]) and a value of type [java.util.Stack] (value [[org.xwiki.context.ExecutionContext@4c4dc21a]]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
12-Jan-2021 08:15:48.748 SEVERE [Thread-60] org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks The web application [xwiki] created a ThreadLocal with key of type [java.lang.ThreadLocal] (value [java.lang.ThreadLocal@2c818169]) and a value of type [java.util.Stack] (value [[org.xwiki.container.servlet.ServletResponse@6125a53d]]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
12-Jan-2021 08:15:48.748 SEVERE [Thread-60] org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks The web application [xwiki] created a ThreadLocal with key of type [java.lang.ThreadLocal] (value [java.lang.ThreadLocal@7543094f]) and a value of type [java.util.Stack] (value [[org.xwiki.container.servlet.ServletRequest@57bc48ef]]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
12-Jan-2021 08:15:48.841 INFO [Thread-60] org.apache.coyote.AbstractProtocol.stop Stopping ProtocolHandler ["http-nio-80"]
12-Jan-2021 08:15:48.904 INFO [Thread-60] org.apache.coyote.AbstractProtocol.stop Stopping ProtocolHandler ["https-openssl-apr-443"]
12-Jan-2021 08:15:48.966 INFO [Thread-60] org.apache.coyote.AbstractProtocol.destroy Destroying ProtocolHandler ["http-nio-80"]
12-Jan-2021 08:15:48.966 INFO [Thread-60] org.apache.coyote.AbstractProtocol.destroy Destroying ProtocolHandler ["https-openssl-apr-443"]

Thanks

Bill

15

so what’s taking time?

16

The timestamps at stop in this example and log are all around 08:15:48, so I am not sure how to answer your question.

17

The service doesn’t show as “stopped” until about 60-61 seconds after the last log entry appears in the catalina log.

18

I stopped Tomcat and revered the conf/logging.properties to the Tomcat default file. The recommended xwiki/WEB-INF/classes/logging.properties is also in place. I started and stopped the service again, and the catalina log file contains the following when stopping:

12-Jan-2021 08:51:18.548 INFO [Thread-52] org.apache.coyote.AbstractProtocol.pause Pausing ProtocolHandler ["http-nio-80"]
12-Jan-2021 08:51:18.610 INFO [Thread-52] org.apache.coyote.AbstractProtocol.pause Pausing ProtocolHandler ["https-openssl-apr-443"]
12-Jan-2021 08:51:18.657 INFO [Thread-52] org.apache.catalina.core.StandardService.stopInternal Stopping service [Catalina]
12-Jan-2021 08:51:19.688 WARNING [Thread-52] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The web application [xwiki] appears to have started a thread named [RxSchedulerPurge-1] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:
java.base@11.0.9.1/jdk.internal.misc.Unsafe.park(Native Method)
java.base@11.0.9.1/java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source)
java.base@11.0.9.1/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(Unknown Source)
java.base@11.0.9.1/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown Source)
java.base@11.0.9.1/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown Source)
java.base@11.0.9.1/java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)
java.base@11.0.9.1/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
java.base@11.0.9.1/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
java.base@11.0.9.1/java.lang.Thread.run(Unknown Source)
12-Jan-2021 08:51:19.688 WARNING [Thread-52] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The web application [xwiki] appears to have started a thread named [RxCachedWorkerPoolEvictor-1] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:
java.base@11.0.9.1/jdk.internal.misc.Unsafe.park(Native Method)
java.base@11.0.9.1/java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source)
java.base@11.0.9.1/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(Unknown Source)
java.base@11.0.9.1/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown Source)
java.base@11.0.9.1/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown Source)
java.base@11.0.9.1/java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)
java.base@11.0.9.1/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
java.base@11.0.9.1/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
java.base@11.0.9.1/java.lang.Thread.run(Unknown Source)
12-Jan-2021 08:51:19.688 WARNING [Thread-52] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The web application [xwiki] appears to have started a thread named [searcherExecutor-10-thread-1] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:
java.base@11.0.9.1/jdk.internal.misc.Unsafe.park(Native Method)
java.base@11.0.9.1/java.util.concurrent.locks.LockSupport.park(Unknown Source)
java.base@11.0.9.1/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(Unknown Source)
java.base@11.0.9.1/java.util.concurrent.LinkedBlockingQueue.take(Unknown Source)
java.base@11.0.9.1/java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)
java.base@11.0.9.1/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
java.base@11.0.9.1/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
java.base@11.0.9.1/java.lang.Thread.run(Unknown Source)
12-Jan-2021 08:51:19.688 WARNING [Thread-52] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The web application [xwiki] appears to have started a thread named [searcherExecutor-12-thread-1] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:
java.base@11.0.9.1/jdk.internal.misc.Unsafe.park(Native Method)
java.base@11.0.9.1/java.util.concurrent.locks.LockSupport.park(Unknown Source)
java.base@11.0.9.1/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(Unknown Source)
java.base@11.0.9.1/java.util.concurrent.LinkedBlockingQueue.take(Unknown Source)
java.base@11.0.9.1/java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)
java.base@11.0.9.1/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
java.base@11.0.9.1/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
java.base@11.0.9.1/java.lang.Thread.run(Unknown Source)
12-Jan-2021 08:51:19.688 WARNING [Thread-52] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The web application [xwiki] appears to have started a thread named [searcherExecutor-11-thread-1] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:
java.base@11.0.9.1/jdk.internal.misc.Unsafe.park(Native Method)
java.base@11.0.9.1/java.util.concurrent.locks.LockSupport.park(Unknown Source)
java.base@11.0.9.1/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(Unknown Source)
java.base@11.0.9.1/java.util.concurrent.LinkedBlockingQueue.take(Unknown Source)
java.base@11.0.9.1/java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)
java.base@11.0.9.1/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
java.base@11.0.9.1/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
java.base@11.0.9.1/java.lang.Thread.run(Unknown Source)
12-Jan-2021 08:51:19.688 WARNING [Thread-52] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The web application [xwiki] appears to have started a thread named [searcherExecutor-25-thread-1] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:
java.base@11.0.9.1/jdk.internal.misc.Unsafe.park(Native Method)
java.base@11.0.9.1/java.util.concurrent.locks.LockSupport.park(Unknown Source)
java.base@11.0.9.1/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(Unknown Source)
java.base@11.0.9.1/java.util.concurrent.LinkedBlockingQueue.take(Unknown Source)
java.base@11.0.9.1/java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)
java.base@11.0.9.1/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
java.base@11.0.9.1/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
java.base@11.0.9.1/java.lang.Thread.run(Unknown Source)
12-Jan-2021 08:51:19.688 WARNING [Thread-52] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The web application [xwiki] appears to have started a thread named [OfficeProcessThread-0] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:
java.base@11.0.9.1/jdk.internal.misc.Unsafe.park(Native Method)
java.base@11.0.9.1/java.util.concurrent.locks.LockSupport.park(Unknown Source)
java.base@11.0.9.1/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(Unknown Source)
java.base@11.0.9.1/java.util.concurrent.LinkedBlockingQueue.take(Unknown Source)
java.base@11.0.9.1/java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)
java.base@11.0.9.1/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
java.base@11.0.9.1/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
java.base@11.0.9.1/java.lang.Thread.run(Unknown Source)
12-Jan-2021 08:51:19.688 WARNING [Thread-52] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The web application [xwiki] appears to have started a thread named [non-blocking-thread--p2-t1] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:
java.base@11.0.9.1/jdk.internal.misc.Unsafe.park(Native Method)
java.base@11.0.9.1/java.util.concurrent.locks.LockSupport.park(Unknown Source)
java.base@11.0.9.1/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(Unknown Source)
java.base@11.0.9.1/java.util.concurrent.LinkedBlockingQueue.take(Unknown Source)
java.base@11.0.9.1/java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)
java.base@11.0.9.1/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
java.base@11.0.9.1/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
java.base@11.0.9.1/java.lang.Thread.run(Unknown Source)
12-Jan-2021 08:51:19.688 WARNING [Thread-52] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The web application [xwiki] appears to have started a thread named [non-blocking-thread--p2-t2] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:
java.base@11.0.9.1/jdk.internal.misc.Unsafe.park(Native Method)
java.base@11.0.9.1/java.util.concurrent.locks.LockSupport.park(Unknown Source)
java.base@11.0.9.1/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(Unknown Source)
java.base@11.0.9.1/java.util.concurrent.LinkedBlockingQueue.take(Unknown Source)
java.base@11.0.9.1/java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)
java.base@11.0.9.1/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
java.base@11.0.9.1/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
java.base@11.0.9.1/java.lang.Thread.run(Unknown Source)
12-Jan-2021 08:51:19.688 WARNING [Thread-52] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The web application [xwiki] appears to have started a thread named [blocking-thread--p3-t2] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:
java.base@11.0.9.1/jdk.internal.misc.Unsafe.park(Native Method)
java.base@11.0.9.1/java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source)
org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1421)
java.base@11.0.9.1/java.lang.Thread.run(Unknown Source)
12-Jan-2021 08:51:19.704 SEVERE [Thread-52] org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks The web application [xwiki] created a ThreadLocal with key of type [java.lang.ThreadLocal] (value [java.lang.ThreadLocal@49d20cf6]) and a value of type [org.codehaus.plexus.classworlds.realm.ClassRealm] (value [ClassRealm[plexus.core, parent: null]]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
12-Jan-2021 08:51:19.704 SEVERE [Thread-52] org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks The web application [xwiki] created a ThreadLocal with key of type [java.lang.ThreadLocal] (value [java.lang.ThreadLocal@6e14c07b]) and a value of type [com.google.javascript.jscomp.Tracer.ThreadTrace] (value []) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
12-Jan-2021 08:51:19.704 SEVERE [Thread-52] org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks The web application [xwiki] created a ThreadLocal with key of type [java.lang.ThreadLocal] (value [java.lang.ThreadLocal@6e14c07b]) and a value of type [com.google.javascript.jscomp.Tracer.ThreadTrace] (value []) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
12-Jan-2021 08:51:19.704 SEVERE [Thread-52] org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks The web application [xwiki] created a ThreadLocal with key of type [java.lang.ThreadLocal] (value [java.lang.ThreadLocal@5a3e82dd]) and a value of type [java.util.Stack] (value [[org.xwiki.context.ExecutionContext@7b0bd55d]]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
12-Jan-2021 08:51:19.704 SEVERE [Thread-52] org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks The web application [xwiki] created a ThreadLocal with key of type [java.lang.ThreadLocal] (value [java.lang.ThreadLocal@5a3e82dd]) and a value of type [java.util.Stack] (value [[org.xwiki.context.ExecutionContext@6932f194]]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
12-Jan-2021 08:51:19.704 SEVERE [Thread-52] org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks The web application [xwiki] created a ThreadLocal with key of type [java.lang.ThreadLocal] (value [java.lang.ThreadLocal@4bfb245f]) and a value of type [java.util.Stack] (value [[org.xwiki.container.servlet.ServletResponse@657d6f97]]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
12-Jan-2021 08:51:19.704 SEVERE [Thread-52] org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks The web application [xwiki] created a ThreadLocal with key of type [java.lang.ThreadLocal] (value [java.lang.ThreadLocal@4079ef58]) and a value of type [java.util.Stack] (value [[org.xwiki.container.servlet.ServletRequest@3eb90892]]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
12-Jan-2021 08:51:19.782 INFO [148] org.apache.catalina.loader.WebappClassLoaderBase.checkStateForResourceLoading Illegal access: this web application instance has been stopped already. Could not load [org.jboss.threads.EnhancedQueueExecutor$2]. The following stack trace is thrown for debugging purposes as well as to attempt to terminate the thread which caused the illegal access.
java.lang.IllegalStateException: Illegal access: this web application instance has been stopped already. Could not load [org.jboss.threads.EnhancedQueueExecutor$2]. The following stack trace is thrown for debugging purposes as well as to attempt to terminate the thread which caused the illegal access.
at org.apache.catalina.loader.WebappClassLoaderBase.checkStateForResourceLoading(WebappClassLoaderBase.java:1385)
at org.apache.catalina.loader.WebappClassLoaderBase.checkStateForClassLoading(WebappClassLoaderBase.java:1373)
at org.apache.catalina.loader.WebappClassLoaderBase.loadClass(WebappClassLoaderBase.java:1226)
at org.apache.catalina.loader.WebappClassLoaderBase.loadClass(WebappClassLoaderBase.java:1188)
at org.jboss.threads.EnhancedQueueExecutor.completeTermination(EnhancedQueueExecutor.java:1770)
at org.jboss.threads.EnhancedQueueExecutor.tryDeallocateThread(EnhancedQueueExecutor.java:1583)
at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1396)
at java.base/java.lang.Thread.run(Unknown Source)
12-Jan-2021 08:51:19.798 INFO [Thread-52] org.apache.coyote.AbstractProtocol.stop Stopping ProtocolHandler ["http-nio-80"]
12-Jan-2021 08:51:19.860 INFO [Thread-52] org.apache.coyote.AbstractProtocol.stop Stopping ProtocolHandler ["https-openssl-apr-443"]
12-Jan-2021 08:51:19.907 INFO [Thread-52] org.apache.coyote.AbstractProtocol.destroy Destroying ProtocolHandler ["http-nio-80"]
12-Jan-2021 08:51:19.907 INFO [Thread-52] org.apache.coyote.AbstractProtocol.destroy Destroying ProtocolHandler ["https-openssl-apr-443"]

New this time is the “illegal access” stack trace that appears after the SEVERE lines and just before the final INFO lines.

19

FYI I’ve documented this at https://www.xwiki.org/xwiki/bin/view/Documentation/AdminGuide/Installation/InstallationWAR/InstallationTomcat/#HThreadLocalErrors

Note that I was wrong above. These SEVERE ThreadLocal errors happen independently of how you stop Tomcat. What’s important is that the JVM is stopped when restarting XWiki. If you do that then you can disregard these warnings.

In any case these should’t impact the shutdown speed of XWiki I think.

20

This problem also occurs on a “vanilla” XWiki install on Windows as demonstrated in my video. If XWiki is deployed, Tomcat shuts down very slowly. If I remove the xwiki directory from Tomcat, Tomcat shutdown is very fast. Below is the Tomcat the log data from when Tomcat is stopped when XWiki 13.0 is deployed on a Windows server running Tomcat 9.0.43 (Windows Server 2016 x64).

First there are a bunch of memory leak warnings, but scroll to the end for another interesting message: Illegal access: this web application instance has been stopped already. Could not load [org.jboss.threads.EnhancedQueueExecutor$2].

Log information follows:

17-Feb-2021 10:45:54.292 INFO [Thread-36] org.apache.catalina.core.StandardService.stopInternal Stopping service [Catalina]
17-Feb-2021 10:45:59.088 WARNING [Thread-36] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The web application [xwiki] appears to have started a thread named [RxSchedulerPurge-1] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:
java.base@11.0.10/jdk.internal.misc.Unsafe.park(Native Method)
java.base@11.0.10/java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source)
java.base@11.0.10/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(Unknown Source)
java.base@11.0.10/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown Source)
java.base@11.0.10/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown Source)
java.base@11.0.10/java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)
java.base@11.0.10/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
java.base@11.0.10/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
java.base@11.0.10/java.lang.Thread.run(Unknown Source)
17-Feb-2021 10:45:59.089 WARNING [Thread-36] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The web application [xwiki] appears to have started a thread named [RxCachedWorkerPoolEvictor-1] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:
java.base@11.0.10/jdk.internal.misc.Unsafe.park(Native Method)
java.base@11.0.10/java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source)
java.base@11.0.10/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(Unknown Source)
java.base@11.0.10/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown Source)
java.base@11.0.10/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown Source)
java.base@11.0.10/java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)
java.base@11.0.10/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
java.base@11.0.10/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
java.base@11.0.10/java.lang.Thread.run(Unknown Source)
17-Feb-2021 10:45:59.090 WARNING [Thread-36] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The web application [xwiki] appears to have started a thread named [expiration-thread--p5-t1] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:
java.base@11.0.10/jdk.internal.misc.Unsafe.park(Native Method)
java.base@11.0.10/java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source)
java.base@11.0.10/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(Unknown Source)
java.base@11.0.10/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown Source)
java.base@11.0.10/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown Source)
java.base@11.0.10/java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)
java.base@11.0.10/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
java.base@11.0.10/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
java.base@11.0.10/java.lang.Thread.run(Unknown Source)
17-Feb-2021 10:45:59.091 WARNING [Thread-36] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The web application [xwiki] appears to have started a thread named [searcherExecutor-12-thread-1] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:
java.base@11.0.10/jdk.internal.misc.Unsafe.park(Native Method)
java.base@11.0.10/java.util.concurrent.locks.LockSupport.park(Unknown Source)
java.base@11.0.10/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(Unknown Source)
java.base@11.0.10/java.util.concurrent.LinkedBlockingQueue.take(Unknown Source)
java.base@11.0.10/java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)
java.base@11.0.10/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
java.base@11.0.10/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
java.base@11.0.10/java.lang.Thread.run(Unknown Source)
17-Feb-2021 10:45:59.092 WARNING [Thread-36] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The web application [xwiki] appears to have started a thread named [searcherExecutor-10-thread-1] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:
java.base@11.0.10/jdk.internal.misc.Unsafe.park(Native Method)
java.base@11.0.10/java.util.concurrent.locks.LockSupport.park(Unknown Source)
java.base@11.0.10/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(Unknown Source)
java.base@11.0.10/java.util.concurrent.LinkedBlockingQueue.take(Unknown Source)
java.base@11.0.10/java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)
java.base@11.0.10/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
java.base@11.0.10/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
java.base@11.0.10/java.lang.Thread.run(Unknown Source)
17-Feb-2021 10:45:59.094 WARNING [Thread-36] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The web application [xwiki] appears to have started a thread named [searcherExecutor-11-thread-1] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:
java.base@11.0.10/jdk.internal.misc.Unsafe.park(Native Method)
java.base@11.0.10/java.util.concurrent.locks.LockSupport.park(Unknown Source)
java.base@11.0.10/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(Unknown Source)
java.base@11.0.10/java.util.concurrent.LinkedBlockingQueue.take(Unknown Source)
java.base@11.0.10/java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)
java.base@11.0.10/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
java.base@11.0.10/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
java.base@11.0.10/java.lang.Thread.run(Unknown Source)
17-Feb-2021 10:45:59.094 WARNING [Thread-36] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The web application [xwiki] appears to have started a thread named [searcherExecutor-25-thread-1] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:
java.base@11.0.10/jdk.internal.misc.Unsafe.park(Native Method)
java.base@11.0.10/java.util.concurrent.locks.LockSupport.park(Unknown Source)
java.base@11.0.10/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(Unknown Source)
java.base@11.0.10/java.util.concurrent.LinkedBlockingQueue.take(Unknown Source)
java.base@11.0.10/java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)
java.base@11.0.10/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
java.base@11.0.10/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
java.base@11.0.10/java.lang.Thread.run(Unknown Source)
17-Feb-2021 10:45:59.094 WARNING [Thread-36] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The web application [xwiki] appears to have started a thread named [non-blocking-thread--p2-t1] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:
java.base@11.0.10/jdk.internal.misc.Unsafe.park(Native Method)
java.base@11.0.10/java.util.concurrent.locks.LockSupport.park(Unknown Source)
java.base@11.0.10/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(Unknown Source)
java.base@11.0.10/java.util.concurrent.LinkedBlockingQueue.take(Unknown Source)
java.base@11.0.10/java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)
java.base@11.0.10/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
java.base@11.0.10/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
java.base@11.0.10/java.lang.Thread.run(Unknown Source)
17-Feb-2021 10:45:59.095 WARNING [Thread-36] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The web application [xwiki] appears to have started a thread named [non-blocking-thread--p2-t2] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:
java.base@11.0.10/jdk.internal.misc.Unsafe.park(Native Method)
java.base@11.0.10/java.util.concurrent.locks.LockSupport.park(Unknown Source)
java.base@11.0.10/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(Unknown Source)
java.base@11.0.10/java.util.concurrent.LinkedBlockingQueue.take(Unknown Source)
java.base@11.0.10/java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)
java.base@11.0.10/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
java.base@11.0.10/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
java.base@11.0.10/java.lang.Thread.run(Unknown Source)
17-Feb-2021 10:45:59.099 WARNING [Thread-36] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The web application [xwiki] appears to have started a thread named [blocking-thread--p3-t1] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:
java.base@11.0.10/jdk.internal.misc.Unsafe.park(Native Method)
java.base@11.0.10/java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source)
org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1421)
java.base@11.0.10/java.lang.Thread.run(Unknown Source)
17-Feb-2021 10:45:59.100 WARNING [Thread-36] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The web application [xwiki] appears to have started a thread named [blocking-thread--p3-t2] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:
java.base@11.0.10/jdk.internal.misc.Unsafe.park(Native Method)
java.base@11.0.10/java.util.concurrent.locks.LockSupport.park(Unknown Source)
org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1409)
java.base@11.0.10/java.lang.Thread.run(Unknown Source)
17-Feb-2021 10:45:59.101 SEVERE [Thread-36] org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks The web application [xwiki] created a ThreadLocal with key of type [java.lang.ThreadLocal] (value [java.lang.ThreadLocal@4098f395]) and a value of type [org.codehaus.plexus.classworlds.realm.ClassRealm] (value [ClassRealm[plexus.core, parent: null]]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
17-Feb-2021 10:45:59.103 SEVERE [Thread-36] org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks The web application [xwiki] created a ThreadLocal with key of type [java.lang.ThreadLocal] (value [java.lang.ThreadLocal@2287e8e7]) and a value of type [com.google.javascript.jscomp.Tracer.ThreadTrace] (value []) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
17-Feb-2021 10:45:59.105 SEVERE [Thread-36] org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks The web application [xwiki] created a ThreadLocal with key of type [java.lang.ThreadLocal] (value [java.lang.ThreadLocal@78055235]) and a value of type [org.restlet.ext.jaxrs.internal.core.MatchedInfo] (value [org.restlet.ext.jaxrs.internal.core.MatchedInfo@1d8bd379]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
17-Feb-2021 10:45:59.105 SEVERE [Thread-36] org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks The web application [xwiki] created a ThreadLocal with key of type [java.lang.ThreadLocal] (value [java.lang.ThreadLocal@2287e8e7]) and a value of type [com.google.javascript.jscomp.Tracer.ThreadTrace] (value []) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
17-Feb-2021 10:45:59.105 SEVERE [Thread-36] org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks The web application [xwiki] created a ThreadLocal with key of type [java.lang.ThreadLocal] (value [java.lang.ThreadLocal@2287e8e7]) and a value of type [com.google.javascript.jscomp.Tracer.ThreadTrace] (value []) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
17-Feb-2021 10:45:59.106 SEVERE [Thread-36] org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks The web application [xwiki] created a ThreadLocal with key of type [java.lang.ThreadLocal] (value [java.lang.ThreadLocal@78055235]) and a value of type [org.restlet.ext.jaxrs.internal.core.MatchedInfo] (value [org.restlet.ext.jaxrs.internal.core.MatchedInfo@6851551b]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
17-Feb-2021 10:45:59.106 SEVERE [Thread-36] org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks The web application [xwiki] created a ThreadLocal with key of type [java.lang.ThreadLocal] (value [java.lang.ThreadLocal@2287e8e7]) and a value of type [com.google.javascript.jscomp.Tracer.ThreadTrace] (value []) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
17-Feb-2021 10:45:59.106 SEVERE [Thread-36] org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks The web application [xwiki] created a ThreadLocal with key of type [java.lang.ThreadLocal] (value [java.lang.ThreadLocal@2287e8e7]) and a value of type [com.google.javascript.jscomp.Tracer.ThreadTrace] (value []) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
17-Feb-2021 10:45:59.108 SEVERE [Thread-36] org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks The web application [xwiki] created a ThreadLocal with key of type [java.lang.ThreadLocal] (value [java.lang.ThreadLocal@3a9736de]) and a value of type [java.util.Stack] (value [[org.xwiki.context.ExecutionContext@79c43aee]]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
17-Feb-2021 10:45:59.108 SEVERE [Thread-36] org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks The web application [xwiki] created a ThreadLocal with key of type [java.lang.ThreadLocal] (value [java.lang.ThreadLocal@1708b903]) and a value of type [java.util.Stack] (value [[org.xwiki.container.servlet.ServletResponse@7513df74]]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
17-Feb-2021 10:45:59.108 SEVERE [Thread-36] org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks The web application [xwiki] created a ThreadLocal with key of type [java.lang.ThreadLocal] (value [java.lang.ThreadLocal@3a9736de]) and a value of type [java.util.Stack] (value [[org.xwiki.context.ExecutionContext@76b136ce]]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
17-Feb-2021 10:45:59.108 SEVERE [Thread-36] org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks The web application [xwiki] created a ThreadLocal with key of type [java.lang.ThreadLocal] (value [java.lang.ThreadLocal@2efa71bc]) and a value of type [java.util.Stack] (value [[org.xwiki.container.servlet.ServletRequest@7ce805c1]]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
17-Feb-2021 10:45:59.109 SEVERE [Thread-36] org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks The web application [xwiki] created a ThreadLocal with key of type [java.lang.ThreadLocal] (value [java.lang.ThreadLocal@2efa71bc]) and a value of type [java.util.Stack] (value [[org.xwiki.container.servlet.ServletRequest@17eb9d57]]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
17-Feb-2021 10:45:59.109 SEVERE [Thread-36] org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks The web application [xwiki] created a ThreadLocal with key of type [java.lang.ThreadLocal] (value [java.lang.ThreadLocal@1708b903]) and a value of type [java.util.Stack] (value [[org.xwiki.container.servlet.ServletResponse@6893ce22]]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
17-Feb-2021 10:45:59.109 SEVERE [Thread-36] org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks The web application [xwiki] created a ThreadLocal with key of type [java.lang.ThreadLocal] (value [java.lang.ThreadLocal@1599351d]) and a value of type [org.apache.solr.request.SolrRequestInfo] (value [org.apache.solr.request.SolrRequestInfo@2963ec62]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.

Then after the memory leak messages, the log has this:

17-Feb-2021 10:45:59.222 INFO [Thread-36] org.apache.coyote.AbstractProtocol.stop Stopping ProtocolHandler ["http-nio-8080"]
17-Feb-2021 10:45:59.223 INFO [blocking-thread--p3-t2] org.apache.catalina.loader.WebappClassLoaderBase.checkStateForResourceLoading Illegal access: this web application instance has been stopped already. Could not load [org.jboss.threads.EnhancedQueueExecutor$2]. The following stack trace is thrown for debugging purposes as well as to attempt to terminate the thread which caused the illegal access.
java.lang.IllegalStateException: Illegal access: this web application instance has been stopped already. Could not load [org.jboss.threads.EnhancedQueueExecutor$2]. The following stack trace is thrown for debugging purposes as well as to attempt to terminate the thread which caused the illegal access.
at org.apache.catalina.loader.WebappClassLoaderBase.checkStateForResourceLoading(WebappClassLoaderBase.java:1384)
at org.apache.catalina.loader.WebappClassLoaderBase.checkStateForClassLoading(WebappClassLoaderBase.java:1372)
at org.apache.catalina.loader.WebappClassLoaderBase.loadClass(WebappClassLoaderBase.java:1225)
at org.apache.catalina.loader.WebappClassLoaderBase.loadClass(WebappClassLoaderBase.java:1187)
at org.jboss.threads.EnhancedQueueExecutor.completeTermination(EnhancedQueueExecutor.java:1770)
at org.jboss.threads.EnhancedQueueExecutor.tryDeallocateThread(EnhancedQueueExecutor.java:1583)
at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1396)
at java.base/java.lang.Thread.run(Unknown Source)
17-Feb-2021 10:45:59.235 INFO [Thread-36] org.apache.coyote.AbstractProtocol.stop Stopping ProtocolHandler ["https-openssl-apr-443"]
17-Feb-2021 10:45:59.247 INFO [Thread-36] org.apache.coyote.AbstractProtocol.destroy Destroying ProtocolHandler ["http-nio-8080"]
17-Feb-2021 10:45:59.250 INFO [Thread-36] org.apache.coyote.AbstractProtocol.destroy Destroying ProtocolHandler ["https-openssl-apr-443"]

The Windows service doesn’t show as “stopped” (i.e., the service remains in the “stopping” state) until about 65 seconds after Tomcat writes the last line to the log. It does not appear to be related to the speed of the machine – this info is taken from a relatively slow VM (for testing), but I see the same behavior (wait a little more than 60 seconds to stop) on other fast production VMs (with much more memory and faster disks).

Any ideas?