When Xwiki 16.4.0 change solr tokenizer it shows exception: java.lang.ClassNotFoundException

gshtime · June 20, 2024, 9:43am

I install a solr’s chinese tokenizer in Xwiki 16.4.0 branch. it doesn’t work.

I do it by these steps:

put ik-analyzer-8.3.0.jar in tomcat/webapps/xwiki/WEB-INF/lib ( ik-analyzer is GitHub - magese/ik-analyzer-solr: ik-analyzer for solr 7.x-8.x)
put five file in tomcat/webapps/xwiki/WEB-INF/classes including ① IKAnalyzer.cfg.xml ② ext.dic③ stopword.dic④ ik.conf⑤ dynamicdic.txt

3.vim xwiki/data/store/solr/search_9/conf/managed-schema.xml and add :

<!-- ik tokenizer  -->
<dynamicField name="*_zh"  type="text_ik"    indexed="true"  stored="true" multiValued="true" />
<dynamicField name="*_zh_CN"  type="text_ik"    indexed="true"  stored="true" multiValued="true" />
<dynamicField name="*_zh_TW"  type="text_ik"    indexed="true"  stored="true" multiValued="true" />
  
<!-- ik tokenizer  -->
<fieldType name="text_ik" class="solr.TextField">
  <analyzer type="index">
      <tokenizer class="org.wltea.analyzer.lucene.IKTokenizerFactory" useSmart="false" conf="ik.conf"/>
      <filter class="solr.LowerCaseFilterFactory"/>
  </analyzer>
  <analyzer type="query">
      <tokenizer class="org.wltea.analyzer.lucene.IKTokenizerFactory" useSmart="true" conf="ik.conf"/>
      <filter class="solr.LowerCaseFilterFactory"/>
  </analyzer>
</fieldType>

when i restart tomcat and xwiki, exception shows java.lang.ClassNotFoundException: org.apache.lucene.analysis.util.ResourceLoaderAware

2024-06-20 17:28:18,133 [main] INFO  o.x.s.s.i.EmbeddedSolr         - Updating Solr home directory at [/home/xwiki/ap/xwiki-16.4.0-data/store/solr] 
2024-06-20 17:28:18,152 [main] INFO  o.x.s.s.i.EmbeddedSolr         - Starting embedded Solr server... 
2024-06-20 17:28:18,152 [main] INFO  o.x.s.s.i.EmbeddedSolr         - Using Solr home directory: [/home/xwiki/ap/xwiki-16.4.0-data/store/solr] 
2024-06-20 17:28:20,860 [main] WARN  o.a.s.c.CoreContainer          - Not all security plugins configured!  authentication=disabled authorization=disabled.  Solr is only as secure as you make it. Consider configuring authentication/authorization before exposing Solr to users internal or external.  See https://s.apache.org/solrsecurity for more info 
2024-06-20 17:28:24,351 [main] ERROR o.x.s.s.i.EmbeddedSolr         - Failed to initialize Solr core with id [search_9] 
org.apache.solr.common.SolrException: JVM Error creating core [search_9]: org/apache/lucene/analysis/util/ResourceLoaderAware
	at org.apache.solr.core.CoreContainer.createFromDescriptor(CoreContainer.java:1746)
	at org.apache.solr.core.CoreContainer.lambda$loadInternal$12(CoreContainer.java:1044)
	at com.codahale.metrics.InstrumentedExecutorService$InstrumentedCallable.call(InstrumentedExecutorService.java:234)
	at java.base/java.util.concurrent.FutureTask.run(Unknown Source)
	at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:294)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
	at java.base/java.lang.Thread.run(Unknown Source)
Caused by: java.lang.NoClassDefFoundError: org/apache/lucene/analysis/util/ResourceLoaderAware
	at java.base/java.lang.ClassLoader.defineClass1(Native Method)
	at java.base/java.lang.ClassLoader.defineClass(Unknown Source)
	at java.base/java.security.SecureClassLoader.defineClass(Unknown Source)
	at java.base/java.net.URLClassLoader.defineClass(Unknown Source)
	at java.base/java.net.URLClassLoader$1.run(Unknown Source)
	at java.base/java.net.URLClassLoader$1.run(Unknown Source)
	at java.base/java.security.AccessController.doPrivileged(Unknown Source)
	at java.base/java.net.URLClassLoader.findClass(Unknown Source)
	at java.base/java.lang.ClassLoader.loadClass(Unknown Source)
	at java.base/java.net.FactoryURLClassLoader.loadClass(Unknown Source)
	at java.base/java.lang.ClassLoader.loadClass(Unknown Source)
	at java.base/java.lang.Class.forName0(Native Method)
	at java.base/java.lang.Class.forName(Unknown Source)
	at org.apache.solr.core.SolrResourceLoader.findClass(SolrResourceLoader.java:534)
	at org.apache.solr.core.SolrResourceLoader.newInstance(SolrResourceLoader.java:612)
	at org.apache.solr.schema.FieldTypePluginLoader$2.create(FieldTypePluginLoader.java:333)
	at org.apache.solr.schema.FieldTypePluginLoader$2.create(FieldTypePluginLoader.java:307)
	at org.apache.solr.util.plugin.AbstractPluginLoader.load(AbstractPluginLoader.java:144)
	at org.apache.solr.schema.FieldTypePluginLoader.readAnalyzer(FieldTypePluginLoader.java:365)
	at org.apache.solr.schema.FieldTypePluginLoader.create(FieldTypePluginLoader.java:78)
	at org.apache.solr.schema.FieldTypePluginLoader.create(FieldTypePluginLoader.java:43)
	at org.apache.solr.util.plugin.AbstractPluginLoader.load(AbstractPluginLoader.java:144)
	at org.apache.solr.schema.IndexSchema.readSchema(IndexSchema.java:537)
	at org.apache.solr.schema.IndexSchema.<init>(IndexSchema.java:192)
	at org.apache.solr.schema.ManagedIndexSchema.<init>(ManagedIndexSchema.java:112)
	at org.apache.solr.schema.ManagedIndexSchemaFactory.create(ManagedIndexSchemaFactory.java:287)
	at org.apache.solr.schema.ManagedIndexSchemaFactory.create(ManagedIndexSchemaFactory.java:54)
	at org.apache.solr.core.ConfigSetService.createIndexSchema(ConfigSetService.java:357)
	at org.apache.solr.core.ConfigSetService.lambda$loadConfigSet$0(ConfigSetService.java:268)
	at org.apache.solr.core.ConfigSet.<init>(ConfigSet.java:49)
	at org.apache.solr.core.ConfigSetService.loadConfigSet(ConfigSetService.java:264)
	at org.apache.solr.core.CoreContainer.createFromDescriptor(CoreContainer.java:1696)
	... 7 common frames omitted
Caused by: java.lang.ClassNotFoundException: org.apache.lucene.analysis.util.ResourceLoaderAware
	at java.base/java.net.URLClassLoader.findClass(Unknown Source)
	at java.base/java.lang.ClassLoader.loadClass(Unknown Source)
	at java.base/java.net.FactoryURLClassLoader.loadClass(Unknown Source)
	at java.base/java.lang.ClassLoader.loadClass(Unknown Source)
	... 39 common frames omitted
2024-06-20 17:28:24,355 [main] ERROR .o.i.DefaultObservationManager - Failed to lookup listener with role hint [solr.availablelocales] 
org.xwiki.component.manager.ComponentLookupException: Failed to lookup component [org.xwiki.search.solr.internal.SolrIndexAvailableLocalesListener] identified by type [interface org.xwiki.observation.EventListener] and hint [solr.availablelocales]
	at org.xwiki.component.embed.EmbeddableComponentManager.getInstance(EmbeddableComponentManager.java:332)
	at org.xwiki.component.embed.EmbeddableComponentManager.getInstance(EmbeddableComponentManager.java:320)
	at org.xwiki.observation.internal.DefaultObservationManager.initializeListeners(DefaultObservationManager.java:178)
	at org.xwiki.observation.internal.DefaultObservationManager.getListenersByEvent(DefaultObservationManager.java:143)
	at org.xwiki.observation.internal.DefaultObservationManager.notify(DefaultObservationManager.java:301)
	at org.xwiki.component.internal.StackingComponentEventManager.sendEvent(StackingComponentEventManager.java:151)
	at org.xwiki.component.internal.StackingComponentEventManager.flushEvents(StackingComponentEventManager.java:92)
	at org.xwiki.container.servlet.XWikiServletContextListener.contextInitialized(XWikiServletContextListener.java:133)
	at org.apache.catalina.core.StandardContext.listenerStart(StandardContext.java:4007)
	at org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:4448)
	at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:171)
	at org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:599)
	at org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:571)
	at org.apache.catalina.core.StandardHost.addChild(StandardHost.java:603)
	at org.apache.catalina.startup.HostConfig.deployDirectory(HostConfig.java:1175)
	at org.apache.catalina.startup.HostConfig$DeployDirectory.run(HostConfig.java:1888)
	at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
	at java.base/java.util.concurrent.FutureTask.run(Unknown Source)
	at org.apache.tomcat.util.threads.InlineExecutorService.execute(InlineExecutorService.java:75)
	at java.base/java.util.concurrent.AbstractExecutorService.submit(Unknown Source)
	at org.apache.catalina.startup.HostConfig.deployDirectories(HostConfig.java:1086)
	at org.apache.catalina.startup.HostConfig.deployApps(HostConfig.java:470)
	at org.apache.catalina.startup.HostConfig.start(HostConfig.java:1584)
	at org.apache.catalina.startup.HostConfig.lifecycleEvent(HostConfig.java:312)
	at org.apache.catalina.util.LifecycleBase.fireLifecycleEvent(LifecycleBase.java:114)
	at org.apache.catalina.util.LifecycleBase.setStateInternal(LifecycleBase.java:402)
	at org.apache.catalina.util.LifecycleBase.setState(LifecycleBase.java:345)
	at org.apache.catalina.core.ContainerBase.startInternal(ContainerBase.java:776)
	at org.apache.catalina.core.StandardHost.startInternal(StandardHost.java:721)
	at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:171)
	at org.apache.catalina.core.ContainerBase$StartChild.call(ContainerBase.java:1203)
	at org.apache.catalina.core.ContainerBase$StartChild.call(ContainerBase.java:1193)
	at java.base/java.util.concurrent.FutureTask.run(Unknown Source)
	at org.apache.tomcat.util.threads.InlineExecutorService.execute(InlineExecutorService.java:75)
	at java.base/java.util.concurrent.AbstractExecutorService.submit(Unknown Source)
	at org.apache.catalina.core.ContainerBase.startInternal(ContainerBase.java:749)
	at org.apache.catalina.core.StandardEngine.startInternal(StandardEngine.java:211)
	at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:171)
	at org.apache.catalina.core.StandardService.startInternal(StandardService.java:415)
	at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:171)
	at org.apache.catalina.core.StandardServer.startInternal(StandardServer.java:878)
	at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:171)
	at org.apache.catalina.startup.Catalina.start(Catalina.java:735)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
	at java.base/java.lang.reflect.Method.invoke(Unknown Source)
	at org.apache.catalina.startup.Bootstrap.start(Bootstrap.java:345)
	at org.apache.catalina.startup.Bootstrap.main(Bootstrap.java:473)
Caused by: org.xwiki.component.manager.ComponentLookupException: Failed to lookup component [org.xwiki.search.solr.internal.SolrClientInstance] identified by type [interface org.xwiki.search.solr.internal.api.SolrInstance] and hint [default]
	at org.xwiki.component.embed.EmbeddableComponentManager.getInstance(EmbeddableComponentManager.java:332)
	at org.xwiki.component.embed.EmbeddableComponentManager.getInstance(EmbeddableComponentManager.java:320)
	at org.xwiki.component.embed.EmbeddableComponentManager.getDependencyInstance(EmbeddableComponentManager.java:591)
	at org.xwiki.component.embed.EmbeddableComponentManager.createInstance(EmbeddableComponentManager.java:540)
	at org.xwiki.component.embed.EmbeddableComponentManager.getComponentInstance(EmbeddableComponentManager.java:636)
	at org.xwiki.component.embed.EmbeddableComponentManager.getInstance(EmbeddableComponentManager.java:329)
	... 48 common frames omitted
Caused by: org.xwiki.component.manager.ComponentLookupException: Failed to lookup component [org.xwiki.search.solr.internal.DefaultSolr] identified by type [interface org.xwiki.search.solr.Solr] and hint [default]
	at org.xwiki.component.embed.EmbeddableComponentManager.getInstance(EmbeddableComponentManager.java:332)
	at org.xwiki.component.embed.EmbeddableComponentManager.getInstance(EmbeddableComponentManager.java:320)
	at org.xwiki.component.embed.EmbeddableComponentManager.getDependencyInstance(EmbeddableComponentManager.java:591)
	at org.xwiki.component.embed.EmbeddableComponentManager.createInstance(EmbeddableComponentManager.java:540)
	at org.xwiki.component.embed.EmbeddableComponentManager.getComponentInstance(EmbeddableComponentManager.java:636)
	at org.xwiki.component.embed.EmbeddableComponentManager.getInstance(EmbeddableComponentManager.java:329)
	... 53 common frames omitted
Caused by: org.xwiki.component.phase.InitializationException: Failed to lookup configured Solr type [embedded]
	at org.xwiki.search.solr.internal.DefaultSolr.initialize(DefaultSolr.java:61)
	at org.xwiki.component.embed.InitializableLifecycleHandler.handle(InitializableLifecycleHandler.java:39)
	at org.xwiki.component.embed.EmbeddableComponentManager.createInstance(EmbeddableComponentManager.java:550)
	at org.xwiki.component.embed.EmbeddableComponentManager.getComponentInstance(EmbeddableComponentManager.java:636)
	at org.xwiki.component.embed.EmbeddableComponentManager.getInstance(EmbeddableComponentManager.java:329)
	... 58 common frames omitted
Caused by: org.xwiki.component.manager.ComponentLookupException: Failed to lookup component [org.xwiki.search.solr.internal.EmbeddedSolr] identified by type [interface org.xwiki.search.solr.Solr] and hint [embedded]
	at org.xwiki.component.embed.EmbeddableComponentManager.getInstance(EmbeddableComponentManager.java:332)
	at org.xwiki.component.embed.EmbeddableComponentManager.getInstance(EmbeddableComponentManager.java:320)
	at org.xwiki.search.solr.internal.DefaultSolr.initialize(DefaultSolr.java:59)
	... 62 common frames omitted
Caused by: org.xwiki.component.phase.InitializationException: Failed to initialize the Solr embedded server with home directory set to [/home/xwiki/ap/xwiki-16.4.0-data/store/solr]
	at org.xwiki.search.solr.internal.EmbeddedSolr.initialize(EmbeddedSolr.java:148)
	at org.xwiki.component.embed.InitializableLifecycleHandler.handle(InitializableLifecycleHandler.java:39)
	at org.xwiki.component.embed.EmbeddableComponentManager.createInstance(EmbeddableComponentManager.java:550)
	at org.xwiki.component.embed.EmbeddableComponentManager.getComponentInstance(EmbeddableComponentManager.java:636)
	at org.xwiki.component.embed.EmbeddableComponentManager.getInstance(EmbeddableComponentManager.java:329)
	... 64 common frames omitted
Caused by: org.apache.solr.client.solrj.SolrServerException: Failed to initialize the Solr core. Please check previous log messages.
	at org.xwiki.search.solr.internal.EmbeddedSolr.createCoreContainer(EmbeddedSolr.java:164)
	at org.xwiki.search.solr.internal.EmbeddedSolr.initialize(EmbeddedSolr.java:144)
	... 68 common frames omitted

but this solution for Chinese tokenizer works well in Xwiki 15.10.10. I don’t know whats difference between Xwiki 15.10.10 and Xwiki 16.4.0.

tmortagne · June 20, 2024, 9:56am

XWiki moved to Solr 9 (and Lucene 9) in 16.2. Apparently the setup you added is more designed for Lucene 8 and, among other things, one class it’s using (org.apache.lucene.analysis.util.ResourceLoaderAware) seems to have moved (to org.apache.lucene.util.ResourceLoaderAware) in Lucene 9.

gshtime · June 20, 2024, 10:54am

thank you for reply.

solr official website says:

Solr modules are not included in the `slim` variant of the Solr binary release. Please download/install the full Solr binary release in order to use Solr modules.

I want to know whether the Xwiki 16.4.0 contains the slim variant of the Solr. and how can know the embbed slor version?

pjeanjean · June 20, 2024, 11:31am

Hi,

It seems that a fork of the project you linked has added support for Solr 9: feat: 升级lucene版本为 9.8.0 · alona789/ik-analyzer-solr@925cfe1 · GitHub
I haven’t tested it, but it might be the easiest solution in your case.

gshtime · June 21, 2024, 1:27am

I think answer you tell would work.
And I tested org/apache/lucene/lucene-analysis-smartcn/9.8.0 of lucene, it work well for Chinese tokenizer too.

@tmortagne Could Xwiki officially integrate the function in future Xwiki.
just insert the smartcn jar in …/solr/search_9/lib/ and add some setting in managed-schema.xml, like:

    <!-- smartcn tokenizer -->
    <dynamicField name="*_zh"  type="text_smartcn"    indexed="true"  stored="true" multiValued="true" />
    <dynamicField name="*_zh_CN"  type="text_smartcn"    indexed="true"  stored="true" multiValued="true" />
    <dynamicField name="*_zh_TW"  type="text_smartcn"    indexed="true"  stored="true" multiValued="true" />
  
    <!-- smartcn tokenizer -->
    <fieldType name="text_smartcn" class="solr.TextField" positionIncrementGap="100">
      <analyzer type="index">
        <tokenizer class="org.apache.lucene.analysis.cn.smart.HMMChineseTokenizerFactory"/>
        <filter class="solr.StopFilterFactory" ignoreCase="true" words="stopwords.txt" />
        <filter class="solr.LowerCaseFilterFactory"/>
      </analyzer>
      <analyzer type="query">
        <tokenizer class="org.apache.lucene.analysis.cn.smart.HMMChineseTokenizerFactory"/>
        <filter class="solr.StopFilterFactory" ignoreCase="true" words="stopwords.txt" />
        <filter class="solr.SynonymFilterFactory" synonyms="synonyms.txt" ignoreCase="true" expand="true"/>
        <filter class="solr.LowerCaseFilterFactory"/>
       </analyzer>
    </fieldType>

More infrormation for pepple in stuck of chinese tokenizer:
Xwiki 16.4.0 use Solr 9.4.1 and Lucene 9.8.0

tmortagne · June 21, 2024, 7:39am

@gshtime could you indicate those details on one of Loading... or Loading....

And if you feel like it, the fastest and most accurate way to have this be integrated in XWiki Standard would be to provide a pull request (that way you are more clearly credited for this improvement).

gshtime · June 21, 2024, 8:38am

ok I have completed a comment in：
https://jira.xwiki.org/browse/XWIKI-18419