This is what the user sees on the My Activity Stream. I’m not sure when this started being broken.
org.xwiki.rendering.RenderingException: Error while retrieving the event status
at org.xwiki.notifications.notifiers.internal.DefaultAsyncNotificationRenderer.render(DefaultAsyncNotificationRenderer.java:143)
at org.xwiki.rendering.async.internal.AsyncRendererJob.runInternal(AsyncRendererJob.java:101)
at org.xwiki.job.AbstractJob.runInContext(AbstractJob.java:246)
at org.xwiki.job.AbstractJob.run(AbstractJob.java:223)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: org.xwiki.eventstream.EventStreamException: Failed to execute Solr query
at org.xwiki.eventstream.store.solr.internal.SolrEventStore.getEventStatuses(SolrEventStore.java:428)
at org.xwiki.eventstream.internal.DefaultEventStore.getEventStatuses(DefaultEventStore.java:446)
at org.xwiki.notifications.internal.DefaultCompositeEventStatusManager.getEventStatuses(DefaultCompositeEventStatusManager.java:82)
at org.xwiki.notifications.internal.DefaultCompositeEventStatusManager.getCompositeEventStatuses(DefaultCompositeEventStatusManager.java:69)
at org.xwiki.notifications.notifiers.internal.DefaultAsyncNotificationRenderer.render(DefaultAsyncNotificationRenderer.java:138)
... 6 more
Caused by: java.lang.NullPointerException
at org.xwiki.eventstream.store.solr.internal.SolrEventStore.getEventStatuses(SolrEventStore.java:426)
... 10 more
Here is a catalina.out from a restart of tomcat9. A lot of errors, hard for me to tell what is significant.
This is what catalina.out spits out when going to a user page:
OK tried this, did not seem to help. I am still seeing stack traces in the application, and this is what the catalina.out looks like for the current run.
I know nothing about SOLR so I’m just flying blind, but I notice a lot of references to “falling back to the legacy store” - is it possible something is misconfigured somewhere? I’ve been tracking changes to the various configuration files, but it is possible I missed an update somewhere.
This is not really a Solr thing, the reason you see it is that since the configured events store (which happen to be based on Solr but it could be anything) initialization failed it fallback on the old one (based on a dedicated table in the database). So again just a consequence of that failed init.
But I don’t understand why you have this failed init, as I cannot reproduce with 15.2. You are really sure you stopped Tomcat, deleted /var/lib/xwiki/data/store/solr/events and restarted Tomcat ?
Another thing you could try is deleted the whole /var/lib/xwiki/data/store/solr (and associated /var/lib/xwiki/data/cache/solr) to really completely reset the embedded Solr, but that means loosing page likes.
[2023-04-06 17:34:32] [info] Server version name: Apache Tomcat/9.0.58 (Ubuntu)
[2023-04-06 17:34:32] [info] Server built: Jan 6 1970 15:09:28 UTC
[2023-04-06 17:34:32] [info] Server version number: 9.0.58.0
[2023-04-06 17:34:32] [info] OS Name: Linux
[2023-04-06 17:34:32] [info] OS Version: 5.15.0-69-generic
[2023-04-06 17:34:32] [info] Architecture: amd64
[2023-04-06 17:34:32] [info] Java Home: /usr/lib/jvm/java-11-openjdk-amd64
[2023-04-06 17:34:32] [info] JVM Version: 11.0.18+10-post-Ubuntu-0ubuntu122.04
[2023-04-06 17:34:32] [info] JVM Vendor: Ubuntu
[2023-04-06 17:34:32] [info] CATALINA_BASE: /var/lib/tomcat9
[2023-04-06 17:34:32] [info] CATALINA_HOME: /usr/share/tomcat9
[2023-04-06 17:34:33] [info] Command line argument: --add-opens=java.base/java.lang=ALL-UNNAMED
[2023-04-06 17:34:33] [info] Command line argument: --add-opens=java.base/java.io=ALL-UNNAMED
[2023-04-06 17:34:33] [info] Command line argument: --add-opens=java.base/java.util=ALL-UNNAMED
[2023-04-06 17:34:33] [info] Command line argument: --add-opens=java.base/java.util.concurrent=ALL-UNNAMED
[2023-04-06 17:34:33] [info] Command line argument: --add-opens=java.rmi/sun.rmi.transport=ALL-UNNAMED
[2023-04-06 17:34:33] [info] Command line argument: -Djava.util.logging.config.file=/var/lib/tomcat9/conf/logging.properties
[2023-04-06 17:34:33] [info] Command line argument: -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager
[2023-04-06 17:34:33] [info] Command line argument: -Xms800m
[2023-04-06 17:34:33] [info] Command line argument: -Xmx1600m
[2023-04-06 17:34:33] [info] Command line argument: -Dfile.encoding=utf-8
[2023-04-06 17:34:33] [info] Command line argument: -Djava.awt.headless=true
[2023-04-06 17:34:33] [info] Command line argument: -XX:+UseParallelGC
[2023-04-06 17:34:33] [info] Command line argument: -XX:MaxGCPauseMillis=100
[2023-04-06 17:34:33] [info] Command line argument: -javaagent:/usr/share/java/glowroot/glowroot.jar
[2023-04-06 17:34:33] [info] Command line argument: -Dorg.apache.tomcat.util.buf.UDecoder.ALLOW_ENCODED_SLASH=true
[2023-04-06 17:34:33] [info] Command line argument: -Dorg.apache.catalina.connector.CoyoteAdapter.ALLOW_BACKSLASH=true
[2023-04-06 17:34:33] [info] Command line argument: -Djdk.tls.ephemeralDHKeySize=2048
[2023-04-06 17:34:33] [info] Command line argument: -Djava.protocol.handler.pkgs=org.apache.catalina.webresources
[2023-04-06 17:34:33] [info] Command line argument: -Dorg.apache.catalina.security.SecurityListener.UMASK=0027
[2023-04-06 17:34:33] [info] Command line argument: -Dignore.endorsed.dirs=
[2023-04-06 17:34:33] [info] Command line argument: -Dcatalina.base=/var/lib/tomcat9
[2023-04-06 17:34:33] [info] Command line argument: -Dcatalina.home=/usr/share/tomcat9
[2023-04-06 17:34:33] [info] Command line argument: -Djava.io.tmpdir=/tmp
[2023-04-06 17:34:33] [info] Loaded Apache Tomcat Native library [1.2.31] using APR version [1.7.0].
[2023-04-06 17:34:33] [info] APR capabilities: IPv6 [true], sendfile [true], accept filters [false], random [true], UDS [true].
[2023-04-06 17:34:33] [info] APR/OpenSSL configuration: useAprConnector [false], useOpenSSL [true]
[2023-04-06 17:34:33] [info] OpenSSL successfully initialized [OpenSSL 3.0.2 15 Mar 2022]
[2023-04-06 17:34:33] [info] Initializing ProtocolHandler ["http-nio-8080"]
[2023-04-06 17:34:33] [info] Server initialization in [1525] milliseconds
[2023-04-06 17:34:34] [info] Starting service [Catalina]
[2023-04-06 17:34:34] [info] Starting Servlet engine: [Apache Tomcat/9.0.58 (Ubuntu)]
[2023-04-06 17:34:34] [info] Deploying deployment descriptor [/etc/tomcat9/Catalina/localhost/xwiki.xml]
[2023-04-06 17:34:34] [warning] The path attribute with value [/xwiki] in deployment descriptor [/etc/tomcat9/Catalina/localhost/xwiki.xml] has been ignored
[2023-04-06 17:34:40] [info] 2023-04-06 17:34:40,962 [main] INFO iPropertiesConfigurationSource - Loading [xwiki.properties] from default location [/etc/xwiki/xwiki.properties]
[2023-04-06 17:34:41] [info] 2023-04-06 17:34:41,256 [main] INFO o.x.e.i.ServletEnvironment - Using permanent directory [/var/lib/xwiki/data]
[2023-04-06 17:35:00] [info] 2023-04-06 17:35:00,911 [main] INFO o.x.s.s.i.EmbeddedSolr - Generating a new Solr home directory at [/var/lib/xwiki/data/store/solr]
[2023-04-06 17:35:00] [info] 2023-04-06 17:35:00,925 [main] INFO o.x.s.s.i.EmbeddedSolr - Starting embedded Solr server...
[2023-04-06 17:35:00] [info] 2023-04-06 17:35:00,925 [main] INFO o.x.s.s.i.EmbeddedSolr - Using Solr home directory: [/var/lib/xwiki/data/store/solr]
[2023-04-06 17:35:02] [info] 2023-04-06 17:35:02,841 [main] WARN o.e.j.u.s.S.config - Trusting all certificates configured for Client@2d739a6f[provider=null,keyStore=null,trustStore=null]
[2023-04-06 17:35:02] [info] 2023-04-06 17:35:02,842 [main] WARN o.e.j.u.s.S.config - No Client EndPointIdentificationAlgorithm configured for Client@2d739a6f[provider=null,keyStore=null,trustStore=null]
[2023-04-06 17:35:03] [info] 2023-04-06 17:35:03,096 [main] WARN o.e.j.u.s.S.config - Trusting all certificates configured for Client@5ff24b0f[provider=null,keyStore=null,trustStore=null]
[2023-04-06 17:35:03] [info] 2023-04-06 17:35:03,096 [main] WARN o.e.j.u.s.S.config - No Client EndPointIdentificationAlgorithm configured for Client@5ff24b0f[provider=null,keyStore=null,trustStore=null]
[2023-04-06 17:35:03] [info] 2023-04-06 17:35:03,294 [main] WARN o.a.s.c.CoreContainer - Not all security plugins configured! authentication=disabled authorization=disabled. Solr is only as secure as you make it. Consider configuring authentication/authorization before exposing Solr to users internal or external. See https://s.apache.org/solrsecurity for more info
[2023-04-06 17:35:06] [info] 2023-04-06 17:35:06,879 [main] INFO o.x.s.s.i.EmbeddedSolr - Started embedded Solr server.
[2023-04-06 17:35:16] [info] 2023-04-06 17:35:16,158 [main] INFO o.x.s.f.i.FilesystemStoreTools - Using filesystem store directory [/data/xwiki/data/store/file]
[2023-04-06 17:35:16] [info] Deployment of deployment descriptor [/etc/tomcat9/Catalina/localhost/xwiki.xml] has finished in [42,426] ms
[2023-04-06 17:35:16] [info] Deploying web application directory [/var/lib/tomcat9/webapps/ROOT]
[2023-04-06 17:35:17] [info] At least one JAR was scanned for TLDs yet contained no TLDs. Enable debug logging for this logger for a complete list of JARs that were scanned but no TLDs were found in them. Skipping unneeded JARs during scanning can improve startup time and JSP compilation time.
[2023-04-06 17:35:17] [info] Deployment of web application directory [/var/lib/tomcat9/webapps/ROOT] has finished in [1,020] ms
[2023-04-06 17:35:17] [info] Starting ProtocolHandler ["http-nio-8080"]
[2023-04-06 17:35:17] [info] Server startup in [43670] milliseconds
Also and maybe coincidentally, the server started responding to requests immediately without the need for any restarts (see: Xwiki not responding to requests). Which is very exciting.