Solr "Failed to commit" and "Failed to send event" with NullPointerException

tracesbs · January 13, 2023, 2:33pm

Since upgrading to XWiki 14.10.3 (from 5.0 ) I’ve been seeing errors in the logs around Solr events. I’ve had a bit of a look around the forum and Jira but I’ve not seen too much on this topic that seems to help.

XWiki is deployed on Tomcat 9.0.70 with Oracle Java 11.0.17 and MySQL 8.0.31.

I’m aware of the Failed to load license file at [null] - Help / Discuss - XWiki Forum issue with Tomcat 9.0.70, but the post suggests that this issue should not have side-effects.

2023-01-13 14:21:54,056 [Asynchronous handler for event store [solr]] ERROR .o.i.DefaultObservationManager - Failed to send event [org.xwiki.eventstream.events.EventStreamAddedEvent@1e6a5fd0] to listener [org.xwiki.notifications.notifiers.internal.UserEventDispatcherListener@5fcf8217] 
java.lang.NullPointerException: null
	at org.xwiki.notifications.notifiers.internal.UserEventDispatcher.onEvent(UserEventDispatcher.java:150)
	at org.xwiki.notifications.notifiers.internal.UserEventDispatcherListener.onEvent(UserEventDispatcherListener.java:76)
	at org.xwiki.observation.internal.DefaultObservationManager.notify(DefaultObservationManager.java:320)
	at org.xwiki.observation.internal.DefaultObservationManager.notify(DefaultObservationManager.java:285)
	at org.xwiki.observation.internal.DefaultObservationManager.notify(DefaultObservationManager.java:337)
	at org.xwiki.eventstream.internal.AbstractAsynchronousEventStore.complete(AbstractAsynchronousEventStore.java:441)
	at org.xwiki.eventstream.internal.AbstractAsynchronousEventStore.afterTasks(AbstractAsynchronousEventStore.java:523)
	at org.xwiki.eventstream.store.solr.internal.SolrEventStore.afterTasks(SolrEventStore.java:174)
	at org.xwiki.eventstream.internal.AbstractAsynchronousEventStore.processTasks(AbstractAsynchronousEventStore.java:347)
	at org.xwiki.eventstream.internal.AbstractAsynchronousEventStore.run(AbstractAsynchronousEventStore.java:325)
	at java.base/java.lang.Thread.run(Thread.java:834)

Also, presumably related I’m seeing:

2023-01-13 14:21:54,056 [Asynchronous handler for event store [solr]] ERROR .o.i.DefaultObservationManager - Failed to send event [org.xwiki.eventstream.events.EventStreamAddedEvent@43d5eef] to listener [org.xwiki.notifications.notifiers.internal.UserEventDispatcherListener@5fcf8217] 
java.lang.NullPointerException: null
	at org.xwiki.notifications.notifiers.internal.UserEventDispatcher.onEvent(UserEventDispatcher.java:150)
	at org.xwiki.notifications.notifiers.internal.UserEventDispatcherListener.onEvent(UserEventDispatcherListener.java:76)
	at org.xwiki.observation.internal.DefaultObservationManager.notify(DefaultObservationManager.java:320)
	at org.xwiki.observation.internal.DefaultObservationManager.notify(DefaultObservationManager.java:285)
	at org.xwiki.observation.internal.DefaultObservationManager.notify(DefaultObservationManager.java:337)
	at org.xwiki.eventstream.internal.AbstractAsynchronousEventStore.complete(AbstractAsynchronousEventStore.java:441)
	at org.xwiki.eventstream.internal.AbstractAsynchronousEventStore.afterTasks(AbstractAsynchronousEventStore.java:523)
	at org.xwiki.eventstream.store.solr.internal.SolrEventStore.afterTasks(SolrEventStore.java:174)
	at org.xwiki.eventstream.internal.AbstractAsynchronousEventStore.processTasks(AbstractAsynchronousEventStore.java:347)
	at org.xwiki.eventstream.internal.AbstractAsynchronousEventStore.run(AbstractAsynchronousEventStore.java:325)
	at java.base/java.lang.Thread.run(Thread.java:834)

Is there something I can look at that would help diagnose this issue further?

Thanks,
Stuart

vmassol · January 13, 2023, 2:37pm

Big jump! I hope that you read all the release notes between these 2 versions and especially the “backward compatibility” sections of them since they contain things you need to do manually to upgrade.

We don’t have a way to aggregate that info over a range of version ATM, so I fear you need to read them one by one for now. Note that most releases don’t have anything to do re backward compatibility.

About solr, the format has changed over the years, and several release notes mentioned remove the solr directory from the permanent directory. See also Configuration (XWiki.org)

tmortagne · January 13, 2023, 2:47pm

Would be interesting to know if you have any other error in the log before that, for example at init. The error you get are really strange and are actually probably the consequence of something else.

tracesbs · January 13, 2023, 2:54pm

Thanks for the swift replies. Much appreciated.

I had another scan of the logs and I missed a couple of entries because of the multiple license stack traces. I found these which look like the culprits, hopefully something closer to the cause…

2023-01-13 14:14:44,967 [main] INFO  o.x.s.s.i.EmbeddedSolr         - Updating Solr home directory at [D:\XWiki\XWikiPermanent\store\solr] 
2023-01-13 14:14:44,998 [main] INFO  o.x.s.s.i.EmbeddedSolr         - Starting embedded Solr server... 
2023-01-13 14:14:44,998 [main] INFO  o.x.s.s.i.EmbeddedSolr         - Using Solr home directory: [D:\XWiki\XWikiPermanent\store\solr] 
2023-01-13 14:14:46,208 [main] WARN  o.e.j.u.s.S.config             - Trusting all certificates configured for Client@77d86aba[provider=null,keyStore=null,trustStore=null] 
2023-01-13 14:14:46,208 [main] WARN  o.e.j.u.s.S.config             - No Client EndPointIdentificationAlgorithm configured for Client@77d86aba[provider=null,keyStore=null,trustStore=null] 
2023-01-13 14:14:46,489 [main] WARN  o.e.j.u.s.S.config             - Trusting all certificates configured for Client@42ca6733[provider=null,keyStore=null,trustStore=null] 
2023-01-13 14:14:46,489 [main] WARN  o.e.j.u.s.S.config             - No Client EndPointIdentificationAlgorithm configured for Client@42ca6733[provider=null,keyStore=null,trustStore=null] 
2023-01-13 14:14:46,567 [main] WARN  o.a.s.c.CoreContainer          - Not all security plugins configured!  authentication=disabled authorization=disabled.  Solr is only as secure as you make it. Consider configuring authentication/authorization before exposing Solr to users internal or external.  See https://s.apache.org/solrsecurity for more info 
2023-01-13 14:14:50,147 [main] INFO  o.x.s.s.i.EmbeddedSolr         - Started embedded Solr server. 
2023-01-13 14:14:50,147 [main] ERROR o.x.s.s.i.EmbeddedSolr         - Failed to create the Solr client for core with name [events] 
org.xwiki.search.solr.SolrException: Failed to prepare the Solr core storage
	at org.xwiki.search.solr.internal.EmbeddedSolr.createCore(EmbeddedSolr.java:178)
	at org.xwiki.search.solr.internal.AbstractSolr.getSynchronizedClient(AbstractSolr.java:91)
	at java.base/java.util.concurrent.ConcurrentHashMap.computeIfAbsent(ConcurrentHashMap.java:1705)
	at org.xwiki.search.solr.internal.AbstractSolr.getClient(AbstractSolr.java:71)
	at org.xwiki.search.solr.internal.DefaultSolr.getClient(DefaultSolr.java:67)
	at org.xwiki.eventstream.store.solr.internal.SolrEventStore.initialize(SolrEventStore.java:159)
	at org.xwiki.component.embed.InitializableLifecycleHandler.handle(InitializableLifecycleHandler.java:39)
	at org.xwiki.component.embed.EmbeddableComponentManager.createInstance(EmbeddableComponentManager.java:365)
	at org.xwiki.component.embed.EmbeddableComponentManager.getComponentInstance(EmbeddableComponentManager.java:451)
	at org.xwiki.component.embed.EmbeddableComponentManager.getInstance(EmbeddableComponentManager.java:201)
	at org.xwiki.eventstream.internal.DefaultEventStore.initialize(DefaultEventStore.java:102)
	at org.xwiki.component.embed.InitializableLifecycleHandler.handle(InitializableLifecycleHandler.java:39)
	at org.xwiki.component.embed.EmbeddableComponentManager.createInstance(EmbeddableComponentManager.java:365)
	at org.xwiki.component.embed.EmbeddableComponentManager.getComponentInstance(EmbeddableComponentManager.java:451)
	at org.xwiki.component.embed.EmbeddableComponentManager.getInstance(EmbeddableComponentManager.java:201)
	at org.xwiki.component.embed.EmbeddableComponentManager.getDependencyInstance(EmbeddableComponentManager.java:406)
	at org.xwiki.component.embed.EmbeddableComponentManager.createInstance(EmbeddableComponentManager.java:355)
	at org.xwiki.component.embed.EmbeddableComponentManager.getComponentInstance(EmbeddableComponentManager.java:451)
	at org.xwiki.component.embed.EmbeddableComponentManager.getInstance(EmbeddableComponentManager.java:201)
	at org.xwiki.component.embed.EmbeddableComponentManager.getDependencyInstance(EmbeddableComponentManager.java:406)
	at org.xwiki.component.embed.EmbeddableComponentManager.createInstance(EmbeddableComponentManager.java:355)
	at org.xwiki.component.embed.EmbeddableComponentManager.getComponentInstance(EmbeddableComponentManager.java:451)
	at org.xwiki.component.embed.EmbeddableComponentManager.getInstanceMap(EmbeddableComponentManager.java:242)
	at org.xwiki.component.embed.EmbeddableComponentManager.getInstanceList(EmbeddableComponentManager.java:225)
	at org.xwiki.observation.internal.DefaultObservationManager.initializeListeners(DefaultObservationManager.java:166)
	at org.xwiki.observation.internal.DefaultObservationManager.getListenersByEvent(DefaultObservationManager.java:132)
	at org.xwiki.observation.internal.DefaultObservationManager.notify(DefaultObservationManager.java:283)
	at org.xwiki.component.internal.StackingComponentEventManager.sendEvent(StackingComponentEventManager.java:151)
	at org.xwiki.component.internal.StackingComponentEventManager.flushEvents(StackingComponentEventManager.java:92)
	at org.xwiki.container.servlet.XWikiServletContextListener.contextInitialized(XWikiServletContextListener.java:133)
	at org.apache.catalina.core.StandardContext.listenerStart(StandardContext.java:4764)
	at org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:5222)
	at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:183)
	at org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:726)
	at org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:698)
	at org.apache.catalina.core.StandardHost.addChild(StandardHost.java:696)
	at org.apache.catalina.startup.HostConfig.deployDirectory(HostConfig.java:1184)
	at org.apache.catalina.startup.HostConfig$DeployDirectory.run(HostConfig.java:1932)
	at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
	at org.apache.tomcat.util.threads.InlineExecutorService.execute(InlineExecutorService.java:75)
	at java.base/java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:118)
	at org.apache.catalina.startup.HostConfig.deployDirectories(HostConfig.java:1094)
	at org.apache.catalina.startup.HostConfig.deployApps(HostConfig.java:476)
	at org.apache.catalina.startup.HostConfig.start(HostConfig.java:1617)
	at org.apache.catalina.startup.HostConfig.lifecycleEvent(HostConfig.java:318)
	at org.apache.catalina.util.LifecycleBase.fireLifecycleEvent(LifecycleBase.java:123)
	at org.apache.catalina.util.LifecycleBase.setStateInternal(LifecycleBase.java:423)
	at org.apache.catalina.util.LifecycleBase.setState(LifecycleBase.java:366)
	at org.apache.catalina.core.ContainerBase.startInternal(ContainerBase.java:943)
	at org.apache.catalina.core.StandardHost.startInternal(StandardHost.java:835)
	at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:183)
	at org.apache.catalina.core.ContainerBase$StartChild.call(ContainerBase.java:1393)
	at org.apache.catalina.core.ContainerBase$StartChild.call(ContainerBase.java:1383)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
	at org.apache.tomcat.util.threads.InlineExecutorService.execute(InlineExecutorService.java:75)
	at java.base/java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:140)
	at org.apache.catalina.core.ContainerBase.startInternal(ContainerBase.java:916)
	at org.apache.catalina.core.StandardEngine.startInternal(StandardEngine.java:265)
	at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:183)
	at org.apache.catalina.core.StandardService.startInternal(StandardService.java:430)
	at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:183)
	at org.apache.catalina.core.StandardServer.startInternal(StandardServer.java:930)
	at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:183)
	at org.apache.catalina.startup.Catalina.start(Catalina.java:772)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:566)
	at org.apache.catalina.startup.Bootstrap.start(Bootstrap.java:347)
	at org.apache.catalina.startup.Bootstrap.main(Bootstrap.java:478)
Caused by: java.nio.file.FileAlreadyExistsException: D:\XWiki\XWikiPermanent\store\solr\events
	at java.base/sun.nio.fs.WindowsException.translateToIOException(WindowsException.java:87)
	at java.base/sun.nio.fs.WindowsException.rethrowAsIOException(WindowsException.java:103)
	at java.base/sun.nio.fs.WindowsException.rethrowAsIOException(WindowsException.java:108)
	at java.base/sun.nio.fs.WindowsFileSystemProvider.createDirectory(WindowsFileSystemProvider.java:509)
	at java.base/java.nio.file.Files.createDirectory(Files.java:690)
	at org.xwiki.search.solr.internal.EmbeddedSolr.prepareCore(EmbeddedSolr.java:204)
	at org.xwiki.search.solr.internal.EmbeddedSolr.createCore(EmbeddedSolr.java:176)
	... 70 common frames omitted
2023-01-13 14:14:51,800 [main] ERROR o.x.s.s.i.EmbeddedSolr         - Failed to create the Solr client for core with name [extension_index] 
org.xwiki.search.solr.SolrException: Failed to prepare the Solr core storage
	at org.xwiki.search.solr.internal.EmbeddedSolr.createCore(EmbeddedSolr.java:178)
	at org.xwiki.search.solr.internal.AbstractSolr.getSynchronizedClient(AbstractSolr.java:91)
	at java.base/java.util.concurrent.ConcurrentHashMap.computeIfAbsent(ConcurrentHashMap.java:1705)
	at org.xwiki.search.solr.internal.AbstractSolr.getClient(AbstractSolr.java:71)
	at org.xwiki.search.solr.internal.DefaultSolr.getClient(DefaultSolr.java:67)
	at org.xwiki.extension.index.internal.ExtensionIndexStore.initialize(ExtensionIndexStore.java:176)
	at org.xwiki.component.embed.InitializableLifecycleHandler.handle(InitializableLifecycleHandler.java:39)
	at org.xwiki.component.embed.EmbeddableComponentManager.createInstance(EmbeddableComponentManager.java:365)
	at org.xwiki.component.embed.EmbeddableComponentManager.getComponentInstance(EmbeddableComponentManager.java:451)
	at org.xwiki.component.embed.EmbeddableComponentManager.getInstance(EmbeddableComponentManager.java:201)
	at org.xwiki.component.embed.EmbeddableComponentManager.getDependencyInstance(EmbeddableComponentManager.java:406)
	at org.xwiki.component.embed.EmbeddableComponentManager.createInstance(EmbeddableComponentManager.java:355)
	at org.xwiki.component.embed.EmbeddableComponentManager.getComponentInstance(EmbeddableComponentManager.java:451)
	at org.xwiki.component.embed.EmbeddableComponentManager.getInstanceMap(EmbeddableComponentManager.java:242)
	at org.xwiki.component.embed.EmbeddableComponentManager.getInstanceList(EmbeddableComponentManager.java:225)
	at org.xwiki.observation.internal.DefaultObservationManager.initializeListeners(DefaultObservationManager.java:166)
	at org.xwiki.observation.internal.DefaultObservationManager.getListenersByEvent(DefaultObservationManager.java:132)
	at org.xwiki.observation.internal.DefaultObservationManager.notify(DefaultObservationManager.java:283)
	at org.xwiki.component.internal.StackingComponentEventManager.sendEvent(StackingComponentEventManager.java:151)
	at org.xwiki.component.internal.StackingComponentEventManager.flushEvents(StackingComponentEventManager.java:92)
	at org.xwiki.container.servlet.XWikiServletContextListener.contextInitialized(XWikiServletContextListener.java:133)
	at org.apache.catalina.core.StandardContext.listenerStart(StandardContext.java:4764)
	at org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:5222)
	at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:183)
	at org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:726)
	at org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:698)
	at org.apache.catalina.core.StandardHost.addChild(StandardHost.java:696)
	at org.apache.catalina.startup.HostConfig.deployDirectory(HostConfig.java:1184)
	at org.apache.catalina.startup.HostConfig$DeployDirectory.run(HostConfig.java:1932)
	at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
	at org.apache.tomcat.util.threads.InlineExecutorService.execute(InlineExecutorService.java:75)
	at java.base/java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:118)
	at org.apache.catalina.startup.HostConfig.deployDirectories(HostConfig.java:1094)
	at org.apache.catalina.startup.HostConfig.deployApps(HostConfig.java:476)
	at org.apache.catalina.startup.HostConfig.start(HostConfig.java:1617)
	at org.apache.catalina.startup.HostConfig.lifecycleEvent(HostConfig.java:318)
	at org.apache.catalina.util.LifecycleBase.fireLifecycleEvent(LifecycleBase.java:123)
	at org.apache.catalina.util.LifecycleBase.setStateInternal(LifecycleBase.java:423)
	at org.apache.catalina.util.LifecycleBase.setState(LifecycleBase.java:366)
	at org.apache.catalina.core.ContainerBase.startInternal(ContainerBase.java:943)
	at org.apache.catalina.core.StandardHost.startInternal(StandardHost.java:835)
	at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:183)
	at org.apache.catalina.core.ContainerBase$StartChild.call(ContainerBase.java:1393)
	at org.apache.catalina.core.ContainerBase$StartChild.call(ContainerBase.java:1383)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
	at org.apache.tomcat.util.threads.InlineExecutorService.execute(InlineExecutorService.java:75)
	at java.base/java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:140)
	at org.apache.catalina.core.ContainerBase.startInternal(ContainerBase.java:916)
	at org.apache.catalina.core.StandardEngine.startInternal(StandardEngine.java:265)
	at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:183)
	at org.apache.catalina.core.StandardService.startInternal(StandardService.java:430)
	at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:183)
	at org.apache.catalina.core.StandardServer.startInternal(StandardServer.java:930)
	at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:183)
	at org.apache.catalina.startup.Catalina.start(Catalina.java:772)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:566)
	at org.apache.catalina.startup.Bootstrap.start(Bootstrap.java:347)
	at org.apache.catalina.startup.Bootstrap.main(Bootstrap.java:478)
Caused by: java.nio.file.FileAlreadyExistsException: D:\XWiki\XWikiPermanent\store\solr\extension_index
	at java.base/sun.nio.fs.WindowsException.translateToIOException(WindowsException.java:87)
	at java.base/sun.nio.fs.WindowsException.rethrowAsIOException(WindowsException.java:103)
	at java.base/sun.nio.fs.WindowsException.rethrowAsIOException(WindowsException.java:108)
	at java.base/sun.nio.fs.WindowsFileSystemProvider.createDirectory(WindowsFileSystemProvider.java:509)
	at java.base/java.nio.file.Files.createDirectory(Files.java:690)
	at org.xwiki.search.solr.internal.EmbeddedSolr.prepareCore(EmbeddedSolr.java:204)
	at org.xwiki.search.solr.internal.EmbeddedSolr.createCore(EmbeddedSolr.java:176)
	... 61 common frames omitted
2023-01-13 14:14:51,925 [main] INFO  o.x.s.f.i.FilesystemStoreTools - Using filesystem store directory [D:\XWiki\XWikiPermanent\store\file]

tracesbs · January 13, 2023, 3:04pm

It seems to stem from:

and

Tomcat is running as “Local Service” and was created by Tomcat, so I think it should have the correct rights?

PS D:\XWiki\XWikiPermanent\store\solr> Get-Acl | Format-List


Path   : Microsoft.PowerShell.Core\FileSystem::D:\XWiki\XWikiPermanent\store\solr
Owner  : NT AUTHORITY\LOCAL SERVICE
Group  : NT AUTHORITY\LOCAL SERVICE
Access : BUILTIN\Administrators Allow  FullControl
         NT AUTHORITY\SYSTEM Allow  FullControl
         NT AUTHORITY\LOCAL SERVICE Allow  FullControl
         CREATOR OWNER Allow  268435456
         BUILTIN\Users Allow  ReadAndExecute, Synchronize
         BUILTIN\Users Allow  AppendData
         BUILTIN\Users Allow  CreateFiles
Audit  :
Sddl   : O:LSG:LSD:AI(A;OICIID;FA;;;BA)(A;OICIID;FA;;;SY)(A;ID;FA;;;LS)(A;OICIIOID;GA;;;CO)(A;OICIID;0x1200a9;;;BU)(A;C
         IID;LC;;;BU)(A;CIID;DC;;;BU)

Is it safe to clear the contents of the D:\XWiki\XWikiPermanent\store\solr folder?

tmortagne · January 13, 2023, 3:06pm

Definitely, looks like it failed to properly create the storage for the events and the other errors you reported are about events to store.

I would suggest trying:

stop XWiki
delete D:\XWiki\XWikiPermanent\store\solr\
delete D:\XWiki\XWikiPermanent\cache (might not exist)
start again XWiki but don’t access it with the browser and see if the Solr init went OK this time (warnings about not configured security plugins and certificates complains are “normal”)

tracesbs · January 13, 2023, 3:13pm

Thanks, I’ve done that. I can see that it’s having issues creating the store, so it appears to be half-baked which probably explains why we were where we were. I’m not quite sure what to make of the stack below…

2023-01-13 15:07:24,821 [main] INFO  o.x.s.s.i.EmbeddedSolr         - Generating a new Solr home directory at [D:\XWiki\XWikiPermanent\store\solr] 
2023-01-13 15:07:24,930 [main] INFO  o.x.s.s.i.EmbeddedSolr         - Starting embedded Solr server... 
2023-01-13 15:07:24,930 [main] INFO  o.x.s.s.i.EmbeddedSolr         - Using Solr home directory: [D:\XWiki\XWikiPermanent\store\solr] 
2023-01-13 15:07:26,602 [main] WARN  o.e.j.u.s.S.config             - Trusting all certificates configured for Client@5e5e680b[provider=null,keyStore=null,trustStore=null] 
2023-01-13 15:07:26,602 [main] WARN  o.e.j.u.s.S.config             - No Client EndPointIdentificationAlgorithm configured for Client@5e5e680b[provider=null,keyStore=null,trustStore=null] 
2023-01-13 15:07:26,799 [main] WARN  o.e.j.u.s.S.config             - Trusting all certificates configured for Client@5e5af20c[provider=null,keyStore=null,trustStore=null] 
2023-01-13 15:07:26,799 [main] WARN  o.e.j.u.s.S.config             - No Client EndPointIdentificationAlgorithm configured for Client@5e5af20c[provider=null,keyStore=null,trustStore=null] 
2023-01-13 15:07:26,885 [main] WARN  o.a.s.c.CoreContainer          - Not all security plugins configured!  authentication=disabled authorization=disabled.  Solr is only as secure as you make it. Consider configuring authentication/authorization before exposing Solr to users internal or external.  See https://s.apache.org/solrsecurity for more info 
2023-01-13 15:07:30,427 [main] INFO  o.x.s.s.i.EmbeddedSolr         - Started embedded Solr server. 
2023-01-13 15:07:37,209 [main] ERROR .o.i.DefaultObservationManager - Failed to lookup listeners 
org.xwiki.component.manager.ComponentLookupException: Failed to lookup component with type [interface org.xwiki.observation.EventListener] and hint [org.xwiki.extension.index.internal.ExtensionInstallListener]
	at org.xwiki.component.embed.EmbeddableComponentManager.getInstanceMap(EmbeddableComponentManager.java:245)
	at org.xwiki.component.embed.EmbeddableComponentManager.getInstanceList(EmbeddableComponentManager.java:225)
	at org.xwiki.observation.internal.DefaultObservationManager.initializeListeners(DefaultObservationManager.java:166)
	at org.xwiki.observation.internal.DefaultObservationManager.getListenersByEvent(DefaultObservationManager.java:132)
	at org.xwiki.observation.internal.DefaultObservationManager.notify(DefaultObservationManager.java:283)
	at org.xwiki.component.internal.StackingComponentEventManager.sendEvent(StackingComponentEventManager.java:151)
	at org.xwiki.component.internal.StackingComponentEventManager.flushEvents(StackingComponentEventManager.java:92)
	at org.xwiki.container.servlet.XWikiServletContextListener.contextInitialized(XWikiServletContextListener.java:133)
	at org.apache.catalina.core.StandardContext.listenerStart(StandardContext.java:4764)
	at org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:5222)
	at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:183)
	at org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:726)
	at org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:698)
	at org.apache.catalina.core.StandardHost.addChild(StandardHost.java:696)
	at org.apache.catalina.startup.HostConfig.deployDirectory(HostConfig.java:1184)
	at org.apache.catalina.startup.HostConfig$DeployDirectory.run(HostConfig.java:1932)
	at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
	at org.apache.tomcat.util.threads.InlineExecutorService.execute(InlineExecutorService.java:75)
	at java.base/java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:118)
	at org.apache.catalina.startup.HostConfig.deployDirectories(HostConfig.java:1094)
	at org.apache.catalina.startup.HostConfig.deployApps(HostConfig.java:476)
	at org.apache.catalina.startup.HostConfig.start(HostConfig.java:1617)
	at org.apache.catalina.startup.HostConfig.lifecycleEvent(HostConfig.java:318)
	at org.apache.catalina.util.LifecycleBase.fireLifecycleEvent(LifecycleBase.java:123)
	at org.apache.catalina.util.LifecycleBase.setStateInternal(LifecycleBase.java:423)
	at org.apache.catalina.util.LifecycleBase.setState(LifecycleBase.java:366)
	at org.apache.catalina.core.ContainerBase.startInternal(ContainerBase.java:943)
	at org.apache.catalina.core.StandardHost.startInternal(StandardHost.java:835)
	at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:183)
	at org.apache.catalina.core.ContainerBase$StartChild.call(ContainerBase.java:1393)
	at org.apache.catalina.core.ContainerBase$StartChild.call(ContainerBase.java:1383)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
	at org.apache.tomcat.util.threads.InlineExecutorService.execute(InlineExecutorService.java:75)
	at java.base/java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:140)
	at org.apache.catalina.core.ContainerBase.startInternal(ContainerBase.java:916)
	at org.apache.catalina.core.StandardEngine.startInternal(StandardEngine.java:265)
	at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:183)
	at org.apache.catalina.core.StandardService.startInternal(StandardService.java:430)
	at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:183)
	at org.apache.catalina.core.StandardServer.startInternal(StandardServer.java:930)
	at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:183)
	at org.apache.catalina.startup.Catalina.start(Catalina.java:772)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:566)
	at org.apache.catalina.startup.Bootstrap.start(Bootstrap.java:347)
	at org.apache.catalina.startup.Bootstrap.main(Bootstrap.java:478)
Caused by: org.xwiki.component.manager.ComponentLookupException: Failed to lookup component [org.xwiki.extension.index.internal.ExtensionIndexStore] identified by type [class org.xwiki.extension.index.internal.ExtensionIndexStore] and hint [default]
	at org.xwiki.component.embed.EmbeddableComponentManager.getInstance(EmbeddableComponentManager.java:204)
	at org.xwiki.component.embed.EmbeddableComponentManager.getDependencyInstance(EmbeddableComponentManager.java:406)
	at org.xwiki.component.embed.EmbeddableComponentManager.createInstance(EmbeddableComponentManager.java:355)
	at org.xwiki.component.embed.EmbeddableComponentManager.getComponentInstance(EmbeddableComponentManager.java:451)
	at org.xwiki.component.embed.EmbeddableComponentManager.getInstanceMap(EmbeddableComponentManager.java:242)
	... 48 common frames omitted
Caused by: org.apache.solr.common.SolrException: Error CREATEing SolrCore 'extension_index': Unable to create core [extension_index] Caused by: no segments* file found in LockValidatingDirectoryWrapper(NRTCachingDirectory(MMapDirectory@D:\XWiki\XWikiPermanent\cache\solr\extension_index\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@f6f5188; maxCacheMB=48.0 maxMergeSizeMB=4.0)): files: [_12q.fdt, _12q.fdx, _12q.kdd, _12q.kdi, _12q.nvd, _12q_Lucene80_0.dvd, _12q_Lucene84_0.doc, _12q_Lucene84_0.pos, _12q_Lucene84_0.tim, _12q_Lucene84_0.tip, _12r.fdt, _12r.fdx, _12r.kdd, _12r.kdi, _12r.nvd, _12r_Lucene80_0.dvd, _12r_Lucene84_0.doc, _12r_Lucene84_0.pos, _12r_Lucene84_0.tim, _12r_Lucene84_0.tip, _12s.fdt, _12s.fdx, _12s.kdd, _12s.kdi, _12s.nvd, _12s_Lucene80_0.dvd, _12s_Lucene84_0.doc, _12s_Lucene84_0.pos, _12s_Lucene84_0.tim, _12s_Lucene84_0.tip, write.lock]
	at org.apache.solr.core.CoreContainer.create(CoreContainer.java:1381)
	at org.apache.solr.core.CoreContainer.create(CoreContainer.java:1289)
	at org.xwiki.search.solr.internal.EmbeddedSolr.createCore(EmbeddedSolr.java:193)
	at org.xwiki.search.solr.internal.AbstractSolr.getSynchronizedClient(AbstractSolr.java:91)
	at java.base/java.util.concurrent.ConcurrentHashMap.computeIfAbsent(ConcurrentHashMap.java:1705)
	at org.xwiki.search.solr.internal.AbstractSolr.getClient(AbstractSolr.java:71)
	at org.xwiki.search.solr.internal.DefaultSolr.getClient(DefaultSolr.java:67)
	at org.xwiki.extension.index.internal.ExtensionIndexStore.initialize(ExtensionIndexStore.java:176)
	at org.xwiki.component.embed.InitializableLifecycleHandler.handle(InitializableLifecycleHandler.java:39)
	at org.xwiki.component.embed.EmbeddableComponentManager.createInstance(EmbeddableComponentManager.java:365)
	at org.xwiki.component.embed.EmbeddableComponentManager.getComponentInstance(EmbeddableComponentManager.java:451)
	at org.xwiki.component.embed.EmbeddableComponentManager.getInstance(EmbeddableComponentManager.java:201)
	... 52 common frames omitted
Caused by: org.apache.solr.common.SolrException: Unable to create core [extension_index]
	at org.apache.solr.core.CoreContainer.createFromDescriptor(CoreContainer.java:1482)
	at org.apache.solr.core.CoreContainer.create(CoreContainer.java:1342)
	... 63 common frames omitted
Caused by: org.apache.solr.common.SolrException: Error opening new searcher
	at org.apache.solr.core.SolrCore.<init>(SolrCore.java:1086)
	at org.apache.solr.core.SolrCore.<init>(SolrCore.java:921)
	at org.apache.solr.core.CoreContainer.createFromDescriptor(CoreContainer.java:1461)
	... 64 common frames omitted
Caused by: org.apache.solr.common.SolrException: Error opening new searcher
	at org.apache.solr.core.SolrCore.openNewSearcher(SolrCore.java:2216)
	at org.apache.solr.core.SolrCore.getSearcher(SolrCore.java:2339)
	at org.apache.solr.core.SolrCore.initSearcher(SolrCore.java:1145)
	at org.apache.solr.core.SolrCore.<init>(SolrCore.java:1027)
	... 66 common frames omitted
Caused by: org.apache.lucene.index.IndexNotFoundException: no segments* file found in LockValidatingDirectoryWrapper(NRTCachingDirectory(MMapDirectory@D:\XWiki\XWikiPermanent\cache\solr\extension_index\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@f6f5188; maxCacheMB=48.0 maxMergeSizeMB=4.0)): files: [_12q.fdt, _12q.fdx, _12q.kdd, _12q.kdi, _12q.nvd, _12q_Lucene80_0.dvd, _12q_Lucene84_0.doc, _12q_Lucene84_0.pos, _12q_Lucene84_0.tim, _12q_Lucene84_0.tip, _12r.fdt, _12r.fdx, _12r.kdd, _12r.kdi, _12r.nvd, _12r_Lucene80_0.dvd, _12r_Lucene84_0.doc, _12r_Lucene84_0.pos, _12r_Lucene84_0.tim, _12r_Lucene84_0.tip, _12s.fdt, _12s.fdx, _12s.kdd, _12s.kdi, _12s.nvd, _12s_Lucene80_0.dvd, _12s_Lucene84_0.doc, _12s_Lucene84_0.pos, _12s_Lucene84_0.tim, _12s_Lucene84_0.tip, write.lock]
	at org.apache.lucene.index.IndexWriter.<init>(IndexWriter.java:1032)
	at org.apache.solr.update.SolrIndexWriter.<init>(SolrIndexWriter.java:126)
	at org.apache.solr.update.SolrIndexWriter.create(SolrIndexWriter.java:100)
	at org.apache.solr.update.DefaultSolrCoreState.createMainIndexWriter(DefaultSolrCoreState.java:261)
	at org.apache.solr.update.DefaultSolrCoreState.getIndexWriter(DefaultSolrCoreState.java:135)
	at org.apache.solr.core.SolrCore.openNewSearcher(SolrCore.java:2177)
	... 69 common frames omitted

tracesbs · January 13, 2023, 3:26pm

Ok, I figured it out. I think at some point the XWiki permanent was configured at the wrong level and so there was another level of folders one level deeper. I tried the deletion again and it seems much better now, so user error on the previous attempt.

The logs are calmer now, so thank you for persisting. You’ve been very helpful once again.

All the best,
Stuart