It may be that I’ve found the solution to my problem:
In my java_opts i had “-XX:+UseConcMarkSweepGC”. This collecter is deprecated since Java 9 and mostly for single-core machines. I switched to -XX:+UseParallelGC like suggested here https://www.xwiki.org/xwiki/bin/view/Documentation/AdminGuide/Performances/#HMemory and til now it runs very great. At last I’ll see in the coming days.
Hahahaha, oh, wait, you are being serious. 
I think that is an oxymoron - simple and Apache Tomcat for Windows.
Anyway, I’ve read the post you sent as well as the guide, but so far, I can’t figure out how to make it work.
What have you tried that doesn’t work? I don’t use windows but the explanations seem clear to me. The page at Add JVM options in Tomcat - Stack Overflow gives a lot of options (I’ve counted 3). Which did you try with what value and what did you get?
The following seems clear to me:
Apache Tomcat 8 (8.5.100) - Windows Service How-To (this is for 8.5 but you can find the same for 9.x I guess and the principle has probably not changed).
More generally speaking @laurin1 , it would help if you could explain what you’ve done in your questions so that we know 1) that you’ve tried something 2) what you’ve tried exactly (and the detailed result) so that we can help with deeper answers.
Thanks
I try so many things and troubleshooting an issue like this is extremely time-consuming already. To document every single thing I try is too much, I just don’t have time (I’m so overloaded as it is), so I’ll try to add details as I can, but I can’t always provide more detailed answers. Even just trying to provide a well thought out response to @vmassol took me 30 minutes and doesn’t document half of what I probably need to provide.
The issue seems to be that there is very little experience with XWiki running on Apache Tomcat on Windows. Take the inability for anyone to tell us how to get the logs for XWiki. How many times have I been asked to provide logs, but nobody seems to know how to do it on Windows or can tell me how to do it. So we may just have to start looking for other options for our Wiki. We work to get the platform stable, and then there’s some whole new problem that is extremely time-consuming to resolve. We love the product, but maintaining it at times seems beyond our ability (time allowing).
None of those solutions on either of the pages are even remotely clear to me on how to implement them. For example, tried this:
Where do I set CATALINA_OPTS? This is Windows so we don’t use catalina.sh - there is a catalina.bat, but I don’t see how to set variables in that batch file (or if that is even used since we are running Tomcat as a service).
I don’t even know what that means. Do I need run XWiki from the console to do this? How does that help if we are running it as a service normally?
I can’t figure out what I am supposed to do with this here.
Seems that I need to use the “–JvmOptions9” parameter, but I can’t find an example of how to do that without overwriting the options we have in use now or does that even matter.
So tried adding this to the registry here, which I know is what the command line above is supposed to update anyway, so I should be able to do that manually.
As you can see, the Options9 format is very different from what glowroot says I need to add (-javaagent:path/to/glowroot.jar).
So I added it like this, but it does not appear to work. I really don’t know, since the only way I know to test it is to open http://localhost:4000.
If you don’t have the time then be sure that I have even less of it (I don’t even have windows nor use it!). I’d just advise to re-read carefully the stackoverflow URL + the doc link I have you. I believe they provide all the info. Can’t help you more.
Note that all this thread (the part about configuring tomcat on windows) even has nothing to do here on the xwiki forum. It’s purely a tomcat config issue and you should ask on the tomcat mailing lists or forum. I think you might be lacking knowledge in both tomcat and windows which makes it harder for you.
And yes almost nobody here (and in general in the xwiki user ecosystem) is using windows so we can’t help you much except point you to documentation and ask you to follow it.
Last: you might want to consider taking professional service from a company who can help you.
I wish you good luck
I finally got Glowroot working:
Where are my application server’s JVM args? · glowroot/glowroot Wiki (github.com)
You should have a file
<tomcat>/bin/tomcat8w.exe. Run that executable and add-javaagent:path/to/glowroot.jarto theJava Optionsunder theJavatab.
Now, the next time I have the high CPU issue, I will get a thread dump from here:
FYI, this has not occurred since we installed 13.4.
Never mind, happening right now.
Restarted and 45 minutes later, already doing it again. I was able to get a thread dump from Glowroot.xwiki_thread_dump_2021-06-18.txt (77.0 KB)
So from what I see in this thread dump XWiki itself is not doing anything and the only things officially running are the following
"h2sc-6-thread-1" #44
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll0(Native Method)
at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll(WindowsSelectorImpl.java:296)
at sun.nio.ch.WindowsSelectorImpl$SubSelector.access$400(WindowsSelectorImpl.java:278)
at sun.nio.ch.WindowsSelectorImpl.doSelect(WindowsSelectorImpl.java:159)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
- locked on sun.nio.ch.Util$3@222c6e69
- locked on java.util.Collections$UnmodifiableSet@256ad237
- locked on sun.nio.ch.WindowsSelectorImpl@328967a4
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
at org.eclipse.jetty.io.ManagedSelector.nioSelect(ManagedSelector.java:149)
at org.eclipse.jetty.io.ManagedSelector.select(ManagedSelector.java:156)
at org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:572)
at org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:509)
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produceTask(EatWhatYouKill.java:360)
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:184)
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:171)
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produce(EatWhatYouKill.java:135)
at org.eclipse.jetty.io.ManagedSelector$$Lambda$302/1894981305.run()
at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:218)
at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$$Lambda$303/1589127750.run()
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
"h2sc-8-thread-1" #47
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll0(Native Method)
at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll(WindowsSelectorImpl.java:296)
at sun.nio.ch.WindowsSelectorImpl$SubSelector.access$400(WindowsSelectorImpl.java:278)
at sun.nio.ch.WindowsSelectorImpl.doSelect(WindowsSelectorImpl.java:159)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
- locked on sun.nio.ch.Util$3@748e79f2
- locked on java.util.Collections$UnmodifiableSet@6f9f0d6b
- locked on sun.nio.ch.WindowsSelectorImpl@388bec2e
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
at org.eclipse.jetty.io.ManagedSelector.nioSelect(ManagedSelector.java:149)
at org.eclipse.jetty.io.ManagedSelector.select(ManagedSelector.java:156)
at org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:572)
at org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:509)
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produceTask(EatWhatYouKill.java:360)
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:184)
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:171)
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produce(EatWhatYouKill.java:135)
at org.eclipse.jetty.io.ManagedSelector$$Lambda$302/1894981305.run()
at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:218)
at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$$Lambda$303/1589127750.run()
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
----------------------------------------
"https-openssl-nio-8443-exec-3" #97
java.lang.Thread.State: RUNNABLE
at sun.misc.Unsafe.allocateMemory(Native Method)
at java.nio.DirectByteBuffer.<init>(DirectByteBuffer.java:127)
at java.nio.ByteBuffer.allocateDirect(ByteBuffer.java:311)
at org.apache.tomcat.util.net.openssl.OpenSSLEngine.readPlaintextData(OpenSSLEngine.java:331)
at org.apache.tomcat.util.net.openssl.OpenSSLEngine.unwrap(OpenSSLEngine.java:584)
- locked on org.apache.tomcat.util.net.openssl.OpenSSLEngine@6cbe2f10
at javax.net.ssl.SSLEngine.unwrap(SSLEngine.java:624)
at org.apache.tomcat.util.net.SecureNioChannel.read(SecureNioChannel.java:617)
at org.apache.tomcat.util.net.NioEndpoint$NioSocketWrapper.fillReadBuffer(NioEndpoint.java:1228)
at org.apache.tomcat.util.net.NioEndpoint$NioSocketWrapper.read(NioEndpoint.java:1140)
at org.apache.coyote.http11.Http11InputBuffer.fill(Http11InputBuffer.java:780)
at org.apache.coyote.http11.Http11InputBuffer.parseRequestLine(Http11InputBuffer.java:356)
at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:260)
at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:65)
at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:868)
at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1589)
at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)
- locked on org.apache.tomcat.util.net.NioEndpoint$NioSocketWrapper@554ecac1
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
at java.lang.Thread.run(Thread.java:748)
"https-openssl-nio-8443-exec-5" #99
java.lang.Thread.State: RUNNABLE
at sun.misc.Unsafe.allocateMemory(Native Method)
at java.nio.DirectByteBuffer.<init>(DirectByteBuffer.java:127)
at java.nio.ByteBuffer.allocateDirect(ByteBuffer.java:311)
at org.apache.tomcat.util.net.openssl.OpenSSLEngine.readPlaintextData(OpenSSLEngine.java:331)
at org.apache.tomcat.util.net.openssl.OpenSSLEngine.unwrap(OpenSSLEngine.java:584)
- locked on org.apache.tomcat.util.net.openssl.OpenSSLEngine@7a58f573
at javax.net.ssl.SSLEngine.unwrap(SSLEngine.java:624)
at org.apache.tomcat.util.net.SecureNioChannel.read(SecureNioChannel.java:617)
at org.apache.tomcat.util.net.NioEndpoint$NioSocketWrapper.fillReadBuffer(NioEndpoint.java:1228)
at org.apache.tomcat.util.net.NioEndpoint$NioSocketWrapper.read(NioEndpoint.java:1140)
at org.apache.coyote.http11.Http11InputBuffer.fill(Http11InputBuffer.java:780)
at org.apache.coyote.http11.Http11InputBuffer.parseRequestLine(Http11InputBuffer.java:356)
at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:260)
at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:65)
at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:868)
at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1589)
at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)
- locked on org.apache.tomcat.util.net.NioEndpoint$NioSocketWrapper@7f348e74
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
at java.lang.Thread.run(Thread.java:748)
The first 2 are Solr background executors, but they don’t seem to actually execute any task when this thread dump was taken (maybe just checking if there is something to do).
The other 2 are Tomcat threads. What is strange is that it seems related to SSL, are you accessing Tomcat in HTTPS directly without an HTTP proxy in front of it ? I doubt it would cause the problem you describe, but it’s surprising.
To summarize, it really does not look like a JVM using all the CPU.
What is strange is that it seems related to SSL, are you accessing Tomcat in HTTPS directly without an HTTP proxy in front of it ? I doubt it would cause the problem you describe, but it’s surprising.
Yes.