Tomcat 9 doesn't stop even after 180 seconds

I’ve just upgraded from 14.2.1 to 14.6; everything went smooth but I discovered (didn’t notice it before), that Tomcat doesn’t stop even after three minutes.

XWiki runs inside Apache Tomcat/9.0.62, JVM 11.0.16+8-post-Ubuntu-0ubuntu120.04; installed manually from the WAR distribution; on a Ubuntu 20.04 virtualized box.

On shutdown, I see several of:

17-Sep-2022 17:24:43.516 SEVERE [main] org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks The web application [xwiki] created a ThreadLocal with key of type [java.lang.ThreadLocal] (value [java.lang.ThreadLocal@17e8e8d6]) and a value of type [java.util.Stack] (value [[org.xwiki.container.servlet.ServletResponse@54fcbf16]]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
17-Sep-2022 17:24:43.516 SEVERE [main] org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks The web application [xwiki] created a ThreadLocal with key of type [java.lang.ThreadLocal] (value [java.lang.ThreadLocal@ac7b35f]) and a value of type [java.util.Stack] (value [[org.xwiki.container.servlet.ServletRequest@2514fb25]]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
17-Sep-2022 17:24:43.516 SEVERE [main] org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks The web application [xwiki] created a ThreadLocal with key of type [java.lang.ThreadLocal] (value [java.lang.ThreadLocal@17e8e8d6]) and a value of type [java.util.Stack] (value [[org.xwiki.container.servlet.ServletResponse@75ea0909]]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
17-Sep-2022 17:24:43.516 SEVERE [main] org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks The web application [xwiki] created a ThreadLocal with key of type [java.lang.ThreadLocal] (value [java.lang.ThreadLocal@ac7b35f]) and a value of type [java.util.Stack] (value [[org.xwiki.container.servlet.ServletRequest@445f0cad]]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
17-Sep-2022 17:24:43.516 SEVERE [main] org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks The web application [xwiki] created a ThreadLocal with key of type [java.lang.ThreadLocal] (value [java.lang.ThreadLocal@1678e9d5]) and a value of type [java.util.Stack] (value [[org.xwiki.context.ExecutionContext@73350232]]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.

and these

17-Sep-2022 17:24:43.498 WARNING [main] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The web application [xwiki] appears to have started a thread named [Connection evictor] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:
 java.base@11.0.16/java.lang.Thread.sleep(Native Method)
 org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
 java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
17-Sep-2022 17:24:43.498 WARNING [main] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The web application [xwiki] appears to have started a thread named [Connection evictor] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:
 java.base@11.0.16/java.lang.Thread.sleep(Native Method)
 org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
 java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
17-Sep-2022 17:24:43.499 WARNING [main] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The web application [xwiki] appears to have started a thread named [h2sc-8-thread-1] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:
 java.base@11.0.16/sun.nio.ch.EPoll.wait(Native Method)
 java.base@11.0.16/sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:120)
 java.base@11.0.16/sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:124)
 java.base@11.0.16/sun.nio.ch.SelectorImpl.select(SelectorImpl.java:141)
 org.eclipse.jetty.io.ManagedSelector.nioSelect(ManagedSelector.java:149)
 org.eclipse.jetty.io.ManagedSelector.select(ManagedSelector.java:156)
 org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:572)
 org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:509)
 org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produceTask(EatWhatYouKill.java:360)
 org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:184)
 org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:171)
 org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produce(EatWhatYouKill.java:135)
 org.eclipse.jetty.io.ManagedSelector$$Lambda$544/0x00000008409ad440.run(Unknown Source)
 org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:218)
 org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$$Lambda$545/0x00000008409acc40.run(Unknown Source)
 java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
 java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
 java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
17-Sep-2022 17:24:43.499 WARNING [main] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The web application [xwiki] appears to have started a thread named [SolrRrdBackendFactory-11-thread-1] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:
 java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
 java.base@11.0.16/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
 java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
 java.base@11.0.16/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1182)
 java.base@11.0.16/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:899)
 java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)
 java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
 java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
 java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
17-Sep-2022 17:24:43.500 WARNING [main] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The web application [xwiki] appears to have started a thread named [MetricsHistoryHandler-12-thread-1] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:
 java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
 java.base@11.0.16/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
 java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
 java.base@11.0.16/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1182)
 java.base@11.0.16/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:899)
 java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)
 java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
 java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
 java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
17-Sep-2022 17:24:43.500 WARNING [main] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The web application [xwiki] appears to have started a thread named [Thread-3] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:
 java.base@11.0.16/java.lang.Object.wait(Native Method)
 java.base@11.0.16/java.lang.Object.wait(Object.java:328)
 org.apache.solr.core.CloserThread.run(CoreContainer.java:2281)
17-Sep-2022 17:24:43.501 WARNING [main] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The web application [xwiki] appears to have started a thread named [searcherExecutor-15-thread-1] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:
 java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
 java.base@11.0.16/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
 java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081)
 java.base@11.0.16/java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:433)
 java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)
 java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
 java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
 java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
17-Sep-2022 17:24:43.501 WARNING [main] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The web application [xwiki] appears to have started a thread named [searcherExecutor-21-thread-1] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:
 java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
 java.base@11.0.16/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
 java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081)
 java.base@11.0.16/java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:433)
 java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)
 java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
 java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
 java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
17-Sep-2022 17:24:43.502 WARNING [main] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The web application [xwiki] appears to have started a thread named [timeout-thread--p4-t1] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:
 java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
 java.base@11.0.16/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
 java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
 java.base@11.0.16/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1182)
 java.base@11.0.16/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:899)
 java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)
 java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
 java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
 java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
17-Sep-2022 17:24:43.503 WARNING [main] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The web application [xwiki] appears to have started a thread named [User event dispatcher thread] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:
 java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
 java.base@11.0.16/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
 java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081)
 java.base@11.0.16/java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:433)
 org.xwiki.notifications.notifiers.internal.UserEventDispatcher.run(UserEventDispatcher.java:214)
 java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
17-Sep-2022 17:24:43.503 WARNING [main] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The web application [xwiki] appears to have started a thread named [searcherExecutor-27-thread-1] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:
 java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
 java.base@11.0.16/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
 java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081)
 java.base@11.0.16/java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:433)
 java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)
 java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
 java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
 java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
17-Sep-2022 17:24:43.505 WARNING [main] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The web application [xwiki] appears to have started a thread named [XWiki's extension job history saving thread] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:
 java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
 java.base@11.0.16/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
 java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081)
 java.base@11.0.16/java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:433)
 org.xwiki.extension.job.history.internal.DefaultExtensionJobHistory$SaveRunnable.run(DefaultExtensionJobHistory.java:73)
 java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
17-Sep-2022 17:24:43.505 WARNING [main] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The web application [xwiki] appears to have started a thread named [SolrRrdBackendFactory-11-thread-2] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:
 java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
 java.base@11.0.16/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
 java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081)
 java.base@11.0.16/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1177)
 java.base@11.0.16/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:899)
 java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)
 java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
 java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
 java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
17-Sep-2022 17:24:43.506 WARNING [main] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The web application [xwiki] appears to have started a thread named [non-blocking-thread--p2-t1] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:
 java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
 java.base@11.0.16/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
 java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081)
 java.base@11.0.16/java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:433)
 java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)
 java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
 java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
 java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
17-Sep-2022 17:24:43.507 WARNING [main] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The web application [xwiki] appears to have started a thread named [non-blocking-thread--p2-t2] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:
 java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
 java.base@11.0.16/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
 java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081)
 java.base@11.0.16/java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:433)
 java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)
 java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
 java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
 java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
17-Sep-2022 17:24:43.507 WARNING [main] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The web application [xwiki] appears to have started a thread named [blocking-thread--p3-t1] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:
 java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
 java.base@11.0.16/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
 org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1421)
 java.base@11.0.16/java.lang.Thread.run(Thread.java:829)

plus this:

17-Sep-2022 17:24:43.508 WARNING [main] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesObjectStreamClassCaches Failed to clear soft references from ObjectStreamClass$Caches for web application [xwiki]
        java.lang.ClassCastException: class java.io.ObjectStreamClass$Caches$1 cannot be cast to class java.util.Map (java.io.ObjectStreamClass$Caches$1 and java.util.Map are in module java.base of loader 'bootstrap')
                at org.apache.catalina.loader.WebappClassLoaderBase.clearCache(WebappClassLoaderBase.java:2336)
                at org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesObjectStreamClassCaches(WebappClassLoaderBase.java:2311)
                at org.apache.catalina.loader.WebappClassLoaderBase.clearReferences(WebappClassLoaderBase.java:1674)
                at org.apache.catalina.loader.WebappClassLoaderBase.stop(WebappClassLoaderBase.java:1602)
                at org.apache.catalina.loader.WebappLoader.stopInternal(WebappLoader.java:463)
                at org.apache.catalina.util.LifecycleBase.stop(LifecycleBase.java:257)
                at org.apache.catalina.core.StandardContext.stopInternal(StandardContext.java:5515)
                at org.apache.catalina.util.LifecycleBase.stop(LifecycleBase.java:257)
                at org.apache.catalina.core.ContainerBase$StopChild.call(ContainerBase.java:1412)
                at org.apache.catalina.core.ContainerBase$StopChild.call(ContainerBase.java:1401)
                at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
                at org.apache.tomcat.util.threads.InlineExecutorService.execute(InlineExecutorService.java:75)
                at java.base/java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:140)
                at org.apache.catalina.core.ContainerBase.stopInternal(ContainerBase.java:986)
                at org.apache.catalina.util.LifecycleBase.stop(LifecycleBase.java:257)
                at org.apache.catalina.core.ContainerBase$StopChild.call(ContainerBase.java:1412)
                at org.apache.catalina.core.ContainerBase$StopChild.call(ContainerBase.java:1401)
                at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
                at org.apache.tomcat.util.threads.InlineExecutorService.execute(InlineExecutorService.java:75)
                at java.base/java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:140)
                at org.apache.catalina.core.ContainerBase.stopInternal(ContainerBase.java:986)
                at org.apache.catalina.util.LifecycleBase.stop(LifecycleBase.java:257)
                at org.apache.catalina.core.StandardService.stopInternal(StandardService.java:497)
                at org.apache.catalina.util.LifecycleBase.stop(LifecycleBase.java:257)
                at org.apache.catalina.core.StandardServer.stopInternal(StandardServer.java:979)
                at org.apache.catalina.util.LifecycleBase.stop(LifecycleBase.java:257)
                at org.apache.catalina.startup.Catalina.stop(Catalina.java:849)
                at org.apache.catalina.startup.Catalina.start(Catalina.java:811)
                at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
                at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
                at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
                at java.base/java.lang.reflect.Method.invoke(Method.java:566)
                at org.apache.catalina.startup.Bootstrap.start(Bootstrap.java:345)
                at org.apache.catalina.startup.Bootstrap.main(Bootstrap.java:476)

OK, they’re warnings, but is there anything I should do or check?

As a side note, I think it shouldn’t make any difference, however despite what’s suggested in the docs, I put the MySQL driver in Tomcat common /lib directory (this TC is only used for XWiki anyway).

I’ve run journalctl -r -x -u tomcat9 and this is the output:

root@xwikihost:/etc/xwiki# journalctl -r -x -u tomcat9
-- Logs begin at Fri 2022-08-26 05:09:09 CEST, end at Sat 2022-09-17 17:27:43 CEST. --
Sep 17 17:26:24 xwikihost systemd[1]: Stopped Tomcat 9 servlet container.
-- Subject: A stop job for unit tomcat9.service has finished
-- Defined-By: systemd
-- Support: http://www.ubuntu.com/support
--
-- A stop job for unit tomcat9.service has finished.
--
-- The job identifier is 548769 and the job result is done.
Sep 17 17:26:24 xwikihost systemd[1]: tomcat9.service: Failed with result 'timeout'.
-- Subject: Unit failed
-- Defined-By: systemd
-- Support: http://www.ubuntu.com/support
--
-- The unit tomcat9.service has entered the 'failed' state with result 'timeout'.
Sep 17 17:26:24 xwikihost systemd[1]: tomcat9.service: Main process exited, code=exited, status=143/n/a
-- Subject: Unit process exited
-- Defined-By: systemd
-- Support: http://www.ubuntu.com/support
--
-- An ExecStart= process belonging to unit tomcat9.service has exited.
--
-- The process' exit code is 'exited' and its exit status is 143.
Sep 17 17:26:23 xwikihost systemd[1]: tomcat9.service: Control process exited, code=killed, status=15/TERM
-- Subject: Unit process exited
-- Defined-By: systemd
-- Support: http://www.ubuntu.com/support
--
-- An ExecStop= process belonging to unit tomcat9.service has exited.
--
-- The process' exit code is 'killed' and its exit status is 15.
Sep 17 17:26:23 xwikihost systemd[1]: tomcat9.service: Stopping timed out. Terminating.
Sep 17 17:24:42 xwikihost shutdown.sh[3046723]: NOTE: Picked up JDK_JAVA_OPTIONS:  --add-opens=java.base/java.lang=ALL-UNNAMED --add-opens=java.base/java.io=ALL-UNNAMED --add-opens=java.base/java.util=AL>
Sep 17 17:24:42 xwikihost systemd[1]: Stopping Tomcat 9 servlet container...
-- Subject: A stop job for unit tomcat9.service has begun execution
-- Defined-By: systemd
-- Support: http://www.ubuntu.com/support
--
-- A stop job for unit tomcat9.service has begun execution.
--
-- The job identifier is 548769.
Sep 17 17:11:20 xwikihost systemd[1]: Started Tomcat 9 servlet container.
-- Subject: A start job for unit tomcat9.service has finished successfully
-- Defined-By: systemd
-- Support: http://www.ubuntu.com/support
--
-- A start job for unit tomcat9.service has finished successfully.
--
-- The job identifier is 548702.
Sep 17 17:11:20 xwikihost startup.sh[3045721]: Tomcat started.
Sep 17 17:11:20 xwikihost startup.sh[3045721]: Removing/clearing stale PID file.
Sep 17 17:11:20 xwikihost startup.sh[3045721]: Existing PID file found during start.
Sep 17 17:11:20 xwikihost systemd[1]: Starting Tomcat 9 servlet container...
-- Subject: A start job for unit tomcat9.service has begun execution

I’ve done another Tomcat shutdown, and this one is new:

17-Sep-2022 18:00:00.000 INFO [DefaultQuartzScheduler_QuartzSchedulerThread] org.apache.catalina.loader.WebappClassLoaderBase.checkStateForResourceLoading Illegal access: this web application instance has been stopped already. Could not load [org.quartz.spi.TriggerFiredBundle]. The following stack trace is thrown for debugging purposes as well as to attempt to terminate the thread which caused the illegal access.
        java.lang.IllegalStateException: Illegal access: this web application instance has been stopped already. Could not load [org.quartz.spi.TriggerFiredBundle]. The following stack trace is thrown for debugging purposes as well as to attempt to terminate the thread which caused the illegal access.
                at org.apache.catalina.loader.WebappClassLoaderBase.checkStateForResourceLoading(WebappClassLoaderBase.java:1432)
                at org.apache.catalina.loader.WebappClassLoaderBase.checkStateForClassLoading(WebappClassLoaderBase.java:1420)
                at org.apache.catalina.loader.WebappClassLoaderBase.loadClass(WebappClassLoaderBase.java:1259)
                at org.apache.catalina.loader.WebappClassLoaderBase.loadClass(WebappClassLoaderBase.java:1220)
                at org.quartz.simpl.RAMJobStore.triggersFired(RAMJobStore.java:1572)
                at org.quartz.core.QuartzSchedulerThread.run(QuartzSchedulerThread.java:353)
Exception in thread "DefaultQuartzScheduler_QuartzSchedulerThread" java.lang.NoClassDefFoundError: org/quartz/spi/TriggerFiredBundle
        at org.quartz.simpl.RAMJobStore.triggersFired(RAMJobStore.java:1572)
        at org.quartz.core.QuartzSchedulerThread.run(QuartzSchedulerThread.java:353)
Caused by: java.lang.ClassNotFoundException: Illegal access: this web application instance has been stopped already. Could not load [org.quartz.spi.TriggerFiredBundle]. The following stack trace is thrown for debugging purposes as well as to attempt to terminate the thread which caused the illegal access.
        at org.apache.catalina.loader.WebappClassLoaderBase.checkStateForClassLoading(WebappClassLoaderBase.java:1422)
        at org.apache.catalina.loader.WebappClassLoaderBase.loadClass(WebappClassLoaderBase.java:1259)
        at org.apache.catalina.loader.WebappClassLoaderBase.loadClass(WebappClassLoaderBase.java:1220)
        ... 2 more
Caused by: java.lang.IllegalStateException: Illegal access: this web application instance has been stopped already. Could not load [org.quartz.spi.TriggerFiredBundle]. The following stack trace is thrown for debugging purposes as well as to attempt to terminate the thread which caused the illegal access.
        at org.apache.catalina.loader.WebappClassLoaderBase.checkStateForResourceLoading(WebappClassLoaderBase.java:1432)
        at org.apache.catalina.loader.WebappClassLoaderBase.checkStateForClassLoading(WebappClassLoaderBase.java:1420)
        ... 4 more

Regarding the ThreadLocals warning, I found this paragraph from this thread.

What would be interesting is to take a thread dump after 1 min for example. It will tell us exactly which thread is blocking Tomcat from stopping.

Sure. After a couple of attempts at elevating the timeout plus reading the threads here, I think there’s a high chance that there’s a thread that simply doesn’t end, hence no matter the wait time, it will simply not stop.

I have very little time to work on my XWiki instance, but I’ll check this ASAP.

My suggestion was more to share with us the thread dump to help with the analysis :slight_smile:

Yep, that was my intention :slight_smile: I just need to find time to do it.

Here it is: jstack-60s.txt (55.3 KB)

As a side note, this is how the service is setup (no setenv.sh or other config has been set in Tomcat folders):

[Unit]
Description=Tomcat 9 servlet container
After=network.target
[Service]
Type=forking
User=tomcat
Group=tomcat
Environment="JAVA_HOME=/usr/lib/jvm/java-11-openjdk-amd64"
Environment="JAVA_OPTS=-Djava.security.egd=file:///dev/urandom -Djava.awt.headless=true"
Environment="CATALINA_BASE=/opt/tomcat/9/latest"
Environment="CATALINA_HOME=/opt/tomcat/9/latest"
Environment="CATALINA_PID=/opt/tomcat/9/latest/temp/tomcat.pid"
Environment="CATALINA_OPTS=-Xms512M -Xmx2048M -server -XX:+UseParallelGC -Dfile.encoding=utf-8 -Dcom.sun.management.jmxremote.port=8090 -Dcom.sun.management.jmxremote.rmi.port=8090 -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.authenticate=false"
ExecStart=/opt/tomcat/9/latest/bin/startup.sh
ExecStop=/opt/tomcat/9/latest/bin/shutdown.sh 300
[Install]
WantedBy=multi-user.target

But journalctl reports:

Sep 24 15:03:45 xwikihost systemd[1]: Stopped Tomcat 9 servlet container.
Sep 24 15:03:45 xwikihost systemd[1]: tomcat9.service: Failed with result 'timeout'.
Sep 24 15:03:45 xwikihost systemd[1]: tomcat9.service: Main process exited, code=exited, status=143/n/a
Sep 24 15:03:45 xwikihost systemd[1]: tomcat9.service: Control process exited, code=killed, status=15/TERM
Sep 24 15:03:45 xwikihost systemd[1]: tomcat9.service: Stopping timed out. Terminating.
Sep 24 15:02:04 xwikihost shutdown.sh[3460718]: NOTE: Picked up JDK_JAVA_OPTIONS:  --add-opens=java.base/java.lang=ALL-UNNAMED --add-opens=java.base/java.io=ALL-UNNAMED --add-opens=java.base/java.util=AL>
Sep 24 15:02:04 xwikihost systemd[1]: Stopping Tomcat 9 servlet container...

It seems it’s waiting around 90 seconds then the timeout is triggered.

I took another dump: jstack-noinit-60s.txt (15.3 KB)

In this, TC has been stopped after a complete startup but without XWiki initialization (I mean with no access to the pages, which triggers the “XWiki is initializing (12%)…” message).

NVM, it’s the expected behaviour of SystemD:
https://www.freedesktop.org/software/systemd/man/systemd.service.html#TimeoutStopSec=
https://www.freedesktop.org/software/systemd/man/systemd-system.conf.html#DefaultTimeoutStartSec=

That’s something you took just before the timeout ?

It really does not look like an instance where stop was triggered.

Both have been taken 60 seconds after issuing systemctl stop tomcat9, which should call the shutdown.sh script. I say should because that’s what it is supposed to do (see the SystemD service config).
The timeout there was still the SystemD default of 90 seconds.

This is an exceprt I quickly got off catalina.out of the same day of my posts:

24-Sep-2022 16:20:47.377 INFO [main] org.apache.coyote.AbstractProtocol.start Starting ProtocolHandler ["http-nio-8080"]
24-Sep-2022 16:20:47.427 INFO [main] org.apache.catalina.startup.Catalina.start Server startup in [24357] milliseconds
24-Sep-2022 16:31:09.888 INFO [main] org.apache.catalina.core.StandardServer.await A valid shutdown command was received via the shutdown port. Stopping the Server instance.
24-Sep-2022 16:31:09.888 INFO [main] org.apache.coyote.AbstractProtocol.pause Pausing ProtocolHandler ["http-nio-8080"]
24-Sep-2022 16:31:09.891 INFO [main] org.apache.catalina.core.StandardService.stopInternal Stopping service [Catalina]
24-Sep-2022 16:31:09.905 WARNING [main] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesObjectStreamClassCaches Failed to clear soft references from ObjectStreamClass$Caches for web application [ROOT]
24-Sep-2022 16:31:09.955 WARNING [main] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesObjectStreamClassCaches Failed to clear soft references from ObjectStreamClass$Caches for web application [examples]
24-Sep-2022 16:31:09.982 WARNING [main] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesObjectStreamClassCaches Failed to clear soft references from ObjectStreamClass$Caches for web application [host-manager]
24-Sep-2022 16:31:09.998 WARNING [main] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesObjectStreamClassCaches Failed to clear soft references from ObjectStreamClass$Caches for web application [manager]
24-Sep-2022 16:31:10.005 WARNING [main] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesObjectStreamClassCaches Failed to clear soft references from ObjectStreamClass$Caches for web application [docs]
24-Sep-2022 16:31:10.304 WARNING [main] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesJdbc The web application [xwiki] registered the JDBC driver [org.apache.calcite.jdbc.Driver] but failed to unregister it when the web application was stopped. To prevent a memory leak, the JDBC Driver has been forcibly unregistered.

As soon as I find time to, I could take dumps waiting longer (i.e. 60s, 120s, etc.) if you think they may help.

Any suggestion?