Solr index directory '/var/lib/xwiki/data/store/solr/events/data/index/' is locked (lockType=native) - Throwing exception

Hi,
Environment: CentOS 7.x and Tomcat.

Followed this link to install: Centos7.2 install XWIKI 11.10.5 - Programmer Sought

Found another error to, but not captured log file.

[main] ERROR o.a.s.c.SolrCore - [extension_index] Solr index directory '/var/lib/xwiki/data/store/solr/extension_index/../../../cache/solr/extension_index/index/' is locked (lockType=native). Throwing exception.

Deleting this /var/lib/xwiki/data/store/solr/events/data/index/ and
/var/lib/xwiki/data/store/solr/extension_index/…/…/…/cache/solr/extension_index/index directories

multiple times for every shut and start tomcat service.

Is there any better work-around for this?

Catalina.out log:

2021-07-07 05:21:48,979 [main] ERROR o.a.s.c.SolrCore - [events] Solr index directory '/var/lib/xwiki/data/store/solr/events/data/index/' is locked (lockType=native).  Throwing exception.
2021-07-07 05:21:48,994 [main] ERROR .o.i.DefaultObservationManager - Failed to lookup listeners
org.xwiki.component.manager.ComponentLookupException: Failed to lookup component with type [interface org.xwiki.observation.EventListener] and hint [EventStreamStoreListener]
        at org.xwiki.component.embed.EmbeddableComponentManager.getInstanceMap(EmbeddableComponentManager.java:245)
        at org.xwiki.component.embed.EmbeddableComponentManager.getInstanceList(EmbeddableComponentManager.java:225)
        at org.xwiki.observation.internal.DefaultObservationManager.initializeListeners(DefaultObservationManager.java:166)
        at org.xwiki.observation.internal.DefaultObservationManager.getListenersByEvent(DefaultObservationManager.java:132)
        at org.xwiki.observation.internal.DefaultObservationManager.notify(DefaultObservationManager.java:283)
        at org.xwiki.component.internal.StackingComponentEventManager.sendEvent(StackingComponentEventManager.java:151)
        at org.xwiki.component.internal.StackingComponentEventManager.flushEvents(StackingComponentEventManager.java:92)  at org.xwiki.container.servlet.XWikiServletContextListener.contextInitialized(XWikiServletContextListener.java:124)
        at org.apache.catalina.core.StandardContext.listenerStart(StandardContext.java:4768)
        at org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:5230)
        at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:183)
        at org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:728)
        at org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:700)
        at org.apache.catalina.core.StandardHost.addChild(StandardHost.java:696)
        at org.apache.catalina.startup.HostConfig.deployWAR(HostConfig.java:1024)
        at org.apache.catalina.startup.HostConfig$DeployWar.run(HostConfig.java:1911)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at org.apache.tomcat.util.threads.InlineExecutorService.execute(InlineExecutorService.java:75)
        at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112)
        at org.apache.catalina.startup.HostConfig.deployWARs(HostConfig.java:825)
        at org.apache.catalina.startup.HostConfig.deployApps(HostConfig.java:475)
        at org.apache.catalina.startup.HostConfig.start(HostConfig.java:1618)
        at org.apache.catalina.startup.HostConfig.lifecycleEvent(HostConfig.java:319)
        at org.apache.catalina.util.LifecycleBase.fireLifecycleEvent(LifecycleBase.java:123)
        at org.apache.catalina.util.LifecycleBase.setStateInternal(LifecycleBase.java:423)
        at org.apache.catalina.util.LifecycleBase.setState(LifecycleBase.java:366)
        at org.apache.catalina.core.ContainerBase.startInternal(ContainerBase.java:948)
        at org.apache.catalina.core.StandardHost.startInternal(StandardHost.java:835)
        at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:183)
        at org.apache.catalina.core.ContainerBase$StartChild.call(ContainerBase.java:1398)
        at org.apache.catalina.core.ContainerBase$StartChild.call(ContainerBase.java:1388)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at org.apache.tomcat.util.threads.InlineExecutorService.execute(InlineExecutorService.java:75)
        at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:134)
        at org.apache.catalina.core.ContainerBase.startInternal(ContainerBase.java:921)
        at org.apache.catalina.core.StandardEngine.startInternal(StandardEngine.java:263)
        at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:183)
        at org.apache.catalina.core.StandardService.startInternal(StandardService.java:437)
        at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:183)
        at org.apache.catalina.core.StandardServer.startInternal(StandardServer.java:934)
        at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:183)
        at org.apache.catalina.startup.Catalina.start(Catalina.java:772)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.apache.catalina.startup.Bootstrap.start(Bootstrap.java:345)
        at org.apache.catalina.startup.Bootstrap.main(Bootstrap.java:476)
Caused by: org.xwiki.component.manager.ComponentLookupException: Failed to lookup component [org.xwiki.eventstream.store.internal.DocumentEventRecorder] identified by type [class org.xwiki.eventstream.store.internal.DocumentEventRecorder] and hint [default]
        at org.xwiki.component.embed.EmbeddableComponentManager.getInstance(EmbeddableComponentManager.java:204)
        at org.xwiki.component.embed.EmbeddableComponentManager.getDependencyInstance(EmbeddableComponentManager.java:406)
        at org.xwiki.component.embed.EmbeddableComponentManager.createInstance(EmbeddableComponentManager.java:355)
        at org.xwiki.component.embed.EmbeddableComponentManager.getComponentInstance(EmbeddableComponentManager.java:451)
        at org.xwiki.component.embed.EmbeddableComponentManager.getInstanceMap(EmbeddableComponentManager.java:242)
        ... 48 common frames omitted
Caused by: org.xwiki.component.manager.ComponentLookupException: Failed to lookup component [org.xwiki.eventstream.internal.DefaultEventStore] identified by type [interface org.xwiki.eventstream.EventStore] and hint [default]
        at org.xwiki.component.embed.EmbeddableComponentManager.getInstance(EmbeddableComponentManager.java:204)
        at org.xwiki.component.embed.EmbeddableComponentManager.getDependencyInstance(EmbeddableComponentManager.java:406)
         at org.xwiki.component.embed.EmbeddableComponentManager.createInstance(EmbeddableComponentManager.java:355)
        at org.xwiki.component.embed.EmbeddableComponentManager.getComponentInstance(EmbeddableComponentManager.java:451)
        at org.xwiki.component.embed.EmbeddableComponentManager.getInstance(EmbeddableComponentManager.java:201)
        ... 52 common frames omitted
Caused by: org.xwiki.component.phase.InitializationException: Failed to get the configured store
        at org.xwiki.eventstream.internal.DefaultEventStore.initialize(DefaultEventStore.java:101)
        at org.xwiki.component.embed.InitializableLifecycleHandler.handle(InitializableLifecycleHandler.java:39)
        at org.xwiki.component.embed.EmbeddableComponentManager.createInstance(EmbeddableComponentManager.java:365)
        at org.xwiki.component.embed.EmbeddableComponentManager.getComponentInstance(EmbeddableComponentManager.java:451)
        at org.xwiki.component.embed.EmbeddableComponentManager.getInstance(EmbeddableComponentManager.java:201)
        ... 56 common frames omitted
Caused by: org.xwiki.component.manager.ComponentLookupException: Failed to lookup component [org.xwiki.eventstream.store.solr.internal.SolrEventStore] identified by type [interface org.xwiki.eventstream.EventStore] and hint [solr]
        at org.xwiki.component.embed.EmbeddableComponentManager.getInstance(EmbeddableComponentManager.java:204)
        at org.xwiki.eventstream.internal.DefaultEventStore.initialize(DefaultEventStore.java:99)
        ... 60 common frames omitted
Caused by: org.apache.solr.common.SolrException: Unable to create core [events]
        at org.apache.solr.core.CoreContainer.createFromDescriptor(CoreContainer.java:1327)
        at org.apache.solr.core.CoreContainer.getCore(CoreContainer.java:1811)
        at org.xwiki.search.solr.internal.EmbeddedSolr.getInternalSolrClient(EmbeddedSolr.java:150)
        at org.xwiki.search.solr.internal.AbstractSolr.getSynchronizedClient(AbstractSolr.java:78)
        at java.util.concurrent.ConcurrentHashMap.computeIfAbsent(ConcurrentHashMap.java:1660)
        at org.xwiki.search.solr.internal.AbstractSolr.getClient(AbstractSolr.java:71)
        at org.xwiki.search.solr.internal.DefaultSolr.getClient(DefaultSolr.java:67)
        at org.xwiki.eventstream.store.solr.internal.SolrEventStore.initialize(SolrEventStore.java:154)
        at org.xwiki.component.embed.InitializableLifecycleHandler.handle(InitializableLifecycleHandler.java:39)
        at org.xwiki.component.embed.EmbeddableComponentManager.createInstance(EmbeddableComponentManager.java:365)   
        at org.xwiki.component.embed.EmbeddableComponentManager.getComponentInstance(EmbeddableComponentManager.java:451)
        at org.xwiki.component.embed.EmbeddableComponentManager.getInstance(EmbeddableComponentManager.java:201)
        ... 61 common frames omitted
Caused by: org.apache.solr.common.SolrException: Index dir '/var/lib/xwiki/data/store/solr/events/data/index/' of core 'events' is already locked. The most likely cause is another Solr server (or another solr core in this server) also configured to use this directory; other possible causes may be specific to lockType: native
        at org.apache.solr.core.SolrCore.<init>(SolrCore.java:1072)
        at org.apache.solr.core.SolrCore.<init>(SolrCore.java:901)
        at org.apache.solr.core.CoreContainer.createFromDescriptor(CoreContainer.java:1306)
        ... 72 common frames omitted
Caused by: org.apache.lucene.store.LockObtainFailedException: Index dir '/var/lib/xwiki/data/store/solr/events/data/index/' of core 'events' is already locked. The most likely cause is another Solr server (or another solr core in this server) also configured to use this directory; other possible causes may be specific to lockType: native
        at org.apache.solr.core.SolrCore.initIndex(SolrCore.java:781)
        at org.apache.solr.core.SolrCore.<init>(SolrCore.java:989)
        ... 74 common frames omitted
07-Jul-2021 05:21:49.627 INFO [main] org.apache.catalina.startup.HostConfig.deployWAR Deployment of web application archive [/opt/tomcat/apache-tomcat-9.0.48/webapps/xwiki.war] has finished in [17,984] ms
07-Jul-2021 05:21:49.628 INFO [main] org.apache.catalina.startup.HostConfig.deployDirectory Deploying web application directory [/opt/tomcat/apache-tomcat-9.0.48/webapps/ROOT]
07-Jul-2021 05:21:49.666 INFO [main] org.apache.catalina.startup.HostConfig.deployDirectory Deployment of web application directory [/opt/tomcat/apache-tomcat-9.0.48/webapps/ROOT] has finished in [38] ms
07-Jul-2021 05:21:49.666 INFO [main] org.apache.catalina.startup.HostConfig.deployDirectory Deploying web application directory [/opt/tomcat/apache-tomcat-9.0.48/webapps/docs]
07-Jul-2021 05:21:49.676 INFO [main] org.apache.catalina.startup.HostConfig.deployDirectory Deployment of web application directory [/opt/tomcat/apache-tomcat-9.0.48/webapps/docs] has finished in [9] ms

This error suggests that several Solr servers manipulate this file. Maybe you have several instances of XWiki running with the same permanent directory configuration.

Thank you @tmortagne . There are no other xwiki apps are running on this server. I did everything from the scratch again - right from installing java-1.8.0-open-jdk, tomcat-9.0.48 and xwiki 12.10.8. But still the issue persists. Any more work-arounds for this. Thank you again in advance.

Latest catalina.out error log.

 2021-07-08 13:50:12,449 [coreLoadExecutor-9-thread-1] ERROR o.a.s.c.SolrCore  - [search] Solr index directory '/var/lib/xwiki/data/store/solr/search/../../../cache/solr/search/index/' is locked (lockType=native).  Throwing exception.
    2021-07-08 13:50:12,463 [main] ERROR .o.i.DefaultObservationManager - Failed to lookup listeners
    org.xwiki.component.manager.ComponentLookupException: Failed to lookup component with type [interface org.xwiki.observation.EventListener] and hint [EventStreamStoreListener]
            at org.xwiki.component.embed.EmbeddableComponentManager.getInstanceMap(EmbeddableComponentManager.java:245)
            at org.xwiki.component.embed.EmbeddableComponentManager.getInstanceList(EmbeddableComponentManager.java:225)
            at org.xwiki.observation.internal.DefaultObservationManager.initializeListeners(DefaultObservationManager.java:166)
            at org.xwiki.observation.internal.DefaultObservationManager.getListenersByEvent(DefaultObservationManager.java:132)
            at org.xwiki.observation.internal.DefaultObservationManager.notify(DefaultObservationManager.java:283)
            at org.xwiki.observation.internal.DefaultObservationManager.notify(DefaultObservationManager.java:337)
            at org.xwiki.container.servlet.XWikiServletContextListener.contextInitialized(XWikiServletContextListener.java:127)
            at org.apache.catalina.core.StandardContext.listenerStart(StandardContext.java:4768)
            at org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:5230)
            at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:183)
            at org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:728)
            at org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:700)
            at org.apache.catalina.core.StandardHost.addChild(StandardHost.java:696)
            at org.apache.catalina.startup.HostConfig.deployDirectory(HostConfig.java:1185)
            at org.apache.catalina.startup.HostConfig$DeployDirectory.run(HostConfig.java:1933)
            at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
            at java.util.concurrent.FutureTask.run(FutureTask.java:266)
            at org.apache.tomcat.util.threads.InlineExecutorService.execute(InlineExecutorService.java:75)
            at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112)
            at org.apache.catalina.startup.HostConfig.deployDirectories(HostConfig.java:1095)
            at org.apache.catalina.startup.HostConfig.deployApps(HostConfig.java:477)
            at org.apache.catalina.startup.HostConfig.start(HostConfig.java:1618)
            at org.apache.catalina.startup.HostConfig.lifecycleEvent(HostConfig.java:319)
            at org.apache.catalina.util.LifecycleBase.fireLifecycleEvent(LifecycleBase.java:123)
            at org.apache.catalina.util.LifecycleBase.setStateInternal(LifecycleBase.java:423)
            at org.apache.catalina.util.LifecycleBase.setState(LifecycleBase.java:366)
            at org.apache.catalina.core.ContainerBase.startInternal(ContainerBase.java:948)
            at org.apache.catalina.core.StandardHost.startInternal(StandardHost.java:835)
            at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:183)
            at org.apache.catalina.core.ContainerBase$StartChild.call(ContainerBase.java:1398)
            at org.apache.catalina.core.ContainerBase$StartChild.call(ContainerBase.java:1388)
            at java.util.concurrent.FutureTask.run(FutureTask.java:266)
            at org.apache.tomcat.util.threads.InlineExecutorService.execute(InlineExecutorService.java:75)
            at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:134)
            at org.apache.catalina.core.ContainerBase.startInternal(ContainerBase.java:921)
            at org.apache.catalina.core.StandardEngine.startInternal(StandardEngine.java:263)
            at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:183)
            at org.apache.catalina.core.StandardService.startInternal(StandardService.java:437)
            at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:183)
            at org.apache.catalina.core.StandardServer.startInternal(StandardServer.java:934)
            at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:183)
            at org.apache.catalina.startup.Catalina.start(Catalina.java:772)
            at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
            at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
            at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
            at java.lang.reflect.Method.invoke(Method.java:498)
            at org.apache.catalina.startup.Bootstrap.start(Bootstrap.java:345)
            at org.apache.catalina.startup.Bootstrap.main(Bootstrap.java:476)
    Caused by: org.xwiki.component.manager.ComponentLookupException: Failed to lookup component [org.xwiki.eventstream.store.internal.DocumentEventRecorder] identified by type [class org.xwiki.eventstream.store.internal.DocumentEventRecorder] and hint [default]
            at org.xwiki.component.embed.EmbeddableComponentManager.getInstance(EmbeddableComponentManager.java:204)
            at org.xwiki.component.embed.EmbeddableComponentManager.getDependencyInstance(EmbeddableComponentManager.java:406)
            at org.xwiki.component.embed.EmbeddableComponentManager.createInstance(EmbeddableComponentManager.java:355)
            at org.xwiki.component.embed.EmbeddableComponentManager.getComponentInstance(EmbeddableComponentManager.java:451)
            at org.xwiki.component.embed.EmbeddableComponentManager.getInstanceMap(EmbeddableComponentManager.java:242)
            ... 47 common frames omitted
    Caused by: org.xwiki.component.manager.ComponentLookupException: Failed to lookup component [org.xwiki.eventstream.internal.DefaultEventStore] identified by type [interface org.xwiki.eventstream.EventStore] and hint [default]
            at org.xwiki.component.embed.EmbeddableComponentManager.getInstance(EmbeddableComponentManager.java:204)
            at org.xwiki.component.embed.EmbeddableComponentManager.getDependencyInstance(EmbeddableComponentManager.java:406)
            at org.xwiki.component.embed.EmbeddableComponentManager.createInstance(EmbeddableComponentManager.java:355)
            at org.xwiki.component.embed.EmbeddableComponentManager.getComponentInstance(EmbeddableComponentManager.java:451)
            at org.xwiki.component.embed.EmbeddableComponentManager.getInstance(EmbeddableComponentManager.java:201)
            ... 51 common frames omitted
    Caused by: org.xwiki.component.phase.InitializationException: Failed to get the configured store
            at org.xwiki.eventstream.internal.DefaultEventStore.initialize(DefaultEventStore.java:101)
            at org.xwiki.component.embed.InitializableLifecycleHandler.handle(InitializableLifecycleHandler.java:39)
            at org.xwiki.component.embed.EmbeddableComponentManager.createInstance(EmbeddableComponentManager.java:365)
            at org.xwiki.component.embed.EmbeddableComponentManager.getComponentInstance(EmbeddableComponentManager.java:451)
            at org.xwiki.component.embed.EmbeddableComponentManager.getInstance(EmbeddableComponentManager.java:201)
            ... 55 common frames omitted
    Caused by: org.xwiki.component.manager.ComponentLookupException: Failed to lookup component [org.xwiki.eventstream.store.solr.internal.SolrEventStore] identified by type [interface org.xwiki.eventstream.EventStore] and hint [solr]
            at org.xwiki.component.embed.EmbeddableComponentManager.getInstance(EmbeddableComponentManager.java:204)
            at org.xwiki.eventstream.internal.DefaultEventStore.initialize(DefaultEventStore.java:99)
            ... 59 common frames omitted
    Caused by: org.xwiki.component.manager.ComponentLookupException: Failed to lookup component [org.xwiki.search.solr.internal.DefaultSolr] identified by type [interface org.xwiki.search.solr.Solr] and hint [default]
            at org.xwiki.component.embed.EmbeddableComponentManager.getInstance(EmbeddableComponentManager.java:204)
            at org.xwiki.component.embed.EmbeddableComponentManager.getDependencyInstance(EmbeddableComponentManager.java:406)
            at org.xwiki.component.embed.EmbeddableComponentManager.createInstance(EmbeddableComponentManager.java:355)
            at org.xwiki.component.embed.EmbeddableComponentManager.getComponentInstance(EmbeddableComponentManager.java:451)
            at org.xwiki.component.embed.EmbeddableComponentManager.getInstance(EmbeddableComponentManager.java:201)
            ... 60 common frames omitted
    Caused by: org.xwiki.component.phase.InitializationException: Failed to lookup configured Solr type [embedded]
            at org.xwiki.search.solr.internal.DefaultSolr.initialize(DefaultSolr.java:60)
            at org.xwiki.component.embed.InitializableLifecycleHandler.handle(InitializableLifecycleHandler.java:39)
            at org.xwiki.component.embed.EmbeddableComponentManager.createInstance(EmbeddableComponentManager.java:365)
            at org.xwiki.component.embed.EmbeddableComponentManager.getComponentInstance(EmbeddableComponentManager.java:451)
            at org.xwiki.component.embed.EmbeddableComponentManager.getInstance(EmbeddableComponentManager.java:201)
            ... 64 common frames omitted
    Caused by: org.xwiki.component.manager.ComponentLookupException: Failed to lookup component [org.xwiki.search.solr.internal.EmbeddedSolr] identified by type [interface org.xwiki.search.solr.Solr] and hint [embedded]
            at org.xwiki.component.embed.EmbeddableComponentManager.getInstance(EmbeddableComponentManager.java:204)
            at org.xwiki.search.solr.internal.DefaultSolr.initialize(DefaultSolr.java:58)
            ... 68 common frames omitted
    Caused by: org.xwiki.component.phase.InitializationException: Failed to initialize the Solr embedded server with home directory set to [/var/lib/xwiki/data/store/solr]
            at org.xwiki.search.solr.internal.EmbeddedSolr.initialize(EmbeddedSolr.java:130)
            at org.xwiki.component.embed.InitializableLifecycleHandler.handle(InitializableLifecycleHandler.java:39)
            at org.xwiki.component.embed.EmbeddableComponentManager.createInstance(EmbeddableComponentManager.java:365)
            at org.xwiki.component.embed.EmbeddableComponentManager.getComponentInstance(EmbeddableComponentManager.java:451)
            at org.xwiki.component.embed.EmbeddableComponentManager.getInstance(EmbeddableComponentManager.java:201)
            ... 69 common frames omitted
    Caused by: org.apache.solr.client.solrj.SolrServerException: Failed to initialize the Solr core. Please check the configuration and log messages.
            at org.xwiki.search.solr.internal.EmbeddedSolr.createCoreContainer(EmbeddedSolr.java:140)
            at org.xwiki.search.solr.internal.EmbeddedSolr.initialize(EmbeddedSolr.java:126)
            ... 73 common frames omitted
    08-Jul-2021 13:50:13.027 INFO [main] org.apache.catalina.startup.HostConfig.deployDirectory Deployment of web application directory [/opt/tomcat/apache-tomcat-9.0.48/webapps/xwiki] has finished in [15,205] ms
    08-Jul-2021 13:50:13.031 INFO [main] org.apache.coyote.AbstractProtocol.start Starting ProtocolHandler ["http-nio-8080"]
    08-Jul-2021 13:50:13.040 INFO [main] org.apache.catalina.startup.Catalina.start Server startup in [15679] milliseconds