Search broken after upgrade to 16.8.0

after upgrading to 16.8.0 on two xwiki instances search has stopped working and just delivers this error:

(shortened)

org.xwiki.rendering.macro.MacroExecutionException: Failed to evaluate Velocity Macro for content [#handleSolrSearchRequest]
 at org.xwiki.rendering.internal.macro.velocity.VelocityMacro.evaluateString(VelocityMacro.java:186)
 at org.xwiki.rendering.internal.macro.velocity.VelocityMacro.evaluateString(VelocityMacro.java:60)
 at org.xwiki.rendering.macro.script.AbstractScriptMacro.evaluateBlock(AbstractScriptMacro.java:279)
 at org.xwiki.rendering.macro.script.AbstractScriptMacro.execute(AbstractScriptMacro.java:182)
 at org.xwiki.rendering.macro.script.AbstractScriptMacro.execute(AbstractScriptMacro.java:58)
 at org.xwiki.rendering.internal.transformation.macro.MacroTransformation.transform(MacroTransformation.java:311)
 at org.xwiki.rendering.internal.transformation.DefaultRenderingContext.transformInContext(DefaultRenderingContext.java:183)
 at org.xwiki.rendering.internal.transformation.DefaultTransformationManager.performTransformations(DefaultTransformationManager.java:88)
 at org.xwiki.display.internal.DocumentContentAsyncExecutor.executeInCurrentExecutionContext(DocumentContentAsyncExecutor.java:396)
 at org.xwiki.display.internal.DocumentContentAsyncExecutor.execute(DocumentContentAsyncExecutor.java:269)
 at org.xwiki.display.internal.DocumentContentAsyncRenderer.execute(DocumentContentAsyncRenderer.java:112)
 at org.xwiki.rendering.async.internal.block.AbstractBlockAsyncRenderer.render(AbstractBlockAsyncRenderer.java:157)
 at org.xwiki.rendering.async.internal.block.AbstractBlockAsyncRenderer.render(AbstractBlockAsyncRenderer.java:54)
 at org.xwiki.rendering.async.internal.DefaultAsyncRendererExecutor.syncRender(DefaultAsyncRendererExecutor.java:290)
 at org.xwiki.rendering.async.internal.DefaultAsyncRendererExecutor.render(DefaultAsyncRendererExecutor.java:267)
 at org.xwiki.rendering.async.internal.block.DefaultBlockAsyncRendererExecutor.execute(DefaultBlockAsyncRendererExecutor.java:125)
 at org.xwiki.display.internal.DocumentContentDisplayer.display(DocumentContentDisplayer.java:67)
 at org.xwiki.display.internal.DocumentContentDisplayer.display(DocumentContentDisplayer.java:43)
 at org.xwiki.display.internal.DefaultDocumentDisplayer.display(DefaultDocumentDisplayer.java:96)
 at org.xwiki.display.internal.DefaultDocumentDisplayer.display(DefaultDocumentDisplayer.java:39)
 at org.xwiki.sheet.internal.SheetDocumentDisplayer.display(SheetDocumentDisplayer.java:123)
 at org.xwiki.sheet.internal.SheetDocumentDisplayer.display(SheetDocumentDisplayer.java:52)
 at org.xwiki.display.internal.ConfiguredDocumentDisplayer.display(ConfiguredDocumentDisplayer.java:68)
 at org.xwiki.display.internal.ConfiguredDocumentDisplayer.display(ConfiguredDocumentDisplayer.java:42)
 at com.xpn.xwiki.doc.XWikiDocument.display(XWikiDocument.java:1388)
 at com.xpn.xwiki.doc.XWikiDocument.getRenderedContent(XWikiDocument.java:1524)
 at com.xpn.xwiki.doc.XWikiDocument.displayDocument(XWikiDocument.java:1474)
 at com.xpn.xwiki.doc.XWikiDocument.displayDocument(XWikiDocument.java:1443)
 at com.xpn.xwiki.api.Document.displayDocument(Document.java:788)
 at jdk.internal.reflect.GeneratedMethodAccessor599.invoke(Unknown Source)
 at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
 at java.base/java.lang.reflect.Method.invoke(Unknown Source)
 at org.apache.velocity.util.introspection.UberspectImpl$VelMethodImpl.doInvoke(UberspectImpl.java:571)
 at org.apache.velocity.util.introspection.UberspectImpl$VelMethodImpl.invoke(UberspectImpl.java:554)
 at org.apache.velocity.runtime.parser.node.ASTMethod.execute(ASTMethod.java:221)
 at org.apache.velocity.runtime.parser.node.ASTReference.execute(ASTReference.java:368)
 at org.apache.velocity.runtime.parser.node.ASTReference.value(ASTReference.java:704)
 at org.apache.velocity.runtime.parser.node.ASTExpression.value(ASTExpression.java:75)
 at org.apache.velocity.runtime.parser.node.ASTSetDirective.render(ASTSetDirective.java:242)
 at org.apache.velocity.runtime.parser.node.ASTBlock.render(ASTBlock.java:147)
 at org.apache.velocity.runtime.parser.node.SimpleNode.render(SimpleNode.java:439)
 at org.apache.velocity.runtime.parser.node.ASTIfStatement.render(ASTIfStatement.java:190)
 at org.apache.velocity.runtime.parser.node.ASTBlock.render(ASTBlock.java:147)
 at org.xwiki.velocity.internal.directive.TryCatchDirective.render(TryCatchDirective.java:86)
 at org.apache.velocity.runtime.parser.node.ASTDirective.render(ASTDirective.java:304)
 at org.apache.velocity.runtime.parser.node.SimpleNode.render(SimpleNode.java:439)
 at org.apache.velocity.Template.merge(Template.java:358)
 at org.apache.velocity.Template.merge(Template.java:262)
 at org.xwiki.velocity.internal.InternalVelocityEngine.evaluate(InternalVelocityEngine.java:225)
 at com.xpn.xwiki.internal.template.VelocityTemplateEvaluator.evaluateContent(VelocityTemplateEvaluator.java:105)
 at com.xpn.xwiki.internal.template.TemplateAsyncRenderer.evaluateContent(TemplateAsyncRenderer.java:219)
 at com.xpn.xwiki.internal.template.TemplateAsyncRenderer.renderVelocity(TemplateAsyncRenderer.java:174)
 at com.xpn.xwiki.internal.template.TemplateAsyncRenderer.render(TemplateAsyncRenderer.java:135)
 at com.xpn.xwiki.internal.template.TemplateAsyncRenderer.render(TemplateAsyncRenderer.java:54)
 at org.xwiki.rendering.async.internal.DefaultAsyncRendererExecutor.lambda$syncRender$0(DefaultAsyncRendererExecutor.java:284)
 at com.xpn.xwiki.internal.security.authorization.DefaultAuthorExecutor.call(DefaultAuthorExecutor.java:98)
 at org.xwiki.rendering.async.internal.DefaultAsyncRendererExecutor.syncRender(DefaultAsyncRendererExecutor.java:284)
 at org.xwiki.rendering.async.internal.DefaultAsyncRendererExecutor.render(DefaultAsyncRendererExecutor.java:267)
 at org.xwiki.rendering.async.internal.block.DefaultBlockAsyncRendererExecutor.render(DefaultBlockAsyncRendererExecutor.java:154)
 at com.xpn.xwiki.internal.template.InternalTemplateManager.render(InternalTemplateManager.java:904)
 at com.xpn.xwiki.internal.template.InternalTemplateManager.renderFromSkin(InternalTemplateManager.java:866)
 at com.xpn.xwiki.internal.template.InternalTemplateManager.render(InternalTemplateManager.java:853)
 at com.xpn.xwiki.internal.template.InternalTemplateManager.renderNoException(InternalTemplateManager.java:808)
 at com.xpn.xwiki.internal.template.InternalTemplateManager.renderNoException(InternalTemplateManager.java:800)
 at com.xpn.xwiki.internal.template.DefaultTemplateManager.renderNoException(DefaultTemplateManager.java:79)
 at com.xpn.xwiki.internal.template.DefaultTemplateManager.renderNoException(DefaultTemplateManager.java:73)
 at org.xwiki.template.script.TemplateScriptService.render(TemplateScriptService.java:54)
 at jdk.internal.reflect.GeneratedMethodAccessor246.invoke(Unknown Source)
 at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
 at java.base/java.lang.reflect.Method.invoke(Unknown Source)
 at org.apache.velocity.util.introspection.UberspectImpl$VelMethodImpl.doInvoke(UberspectImpl.java:571)
 at org.apache.velocity.util.introspection.UberspectImpl$VelMethodImpl.invoke(UberspectImpl.java:554)
 at org.apache.velocity.runtime.parser.node.ASTMethod.execute(ASTMethod.java:221)
 at org.apache.velocity.runtime.parser.node.ASTReference.execute(ASTReference.java:368)
 at org.apache.velocity.runtime.parser.node.ASTReference.render(ASTReference.java:492)
 at org.apache.velocity.runtime.parser.node.ASTBlock.render(ASTBlock.java:147)
 at org.apache.velocity.runtime.directive.VelocimacroProxy.render(VelocimacroProxy.java:218)
 at org.apache.velocity.runtime.directive.RuntimeMacro.render(RuntimeMacro.java:331)
 at org.apache.velocity.runtime.directive.RuntimeMacro.render(RuntimeMacro.java:261)
 at org.apache.velocity.runtime.parser.node.ASTDirective.render(ASTDirective.java:304)
 at org.apache.velocity.runtime.parser.node.SimpleNode.render(SimpleNode.java:439)
 at org.apache.velocity.Template.merge(Template.java:358)
 at org.apache.velocity.Template.merge(Template.java:262)
 at org.xwiki.velocity.internal.InternalVelocityEngine.evaluate(InternalVelocityEngine.java:225)
 at com.xpn.xwiki.internal.template.VelocityTemplateEvaluator.evaluateContent(VelocityTemplateEvaluator.java:105)
 at com.xpn.xwiki.internal.template.TemplateAsyncRenderer.evaluateContent(TemplateAsyncRenderer.java:219)
 at com.xpn.xwiki.internal.template.TemplateAsyncRenderer.renderVelocity(TemplateAsyncRenderer.java:174)
 at com.xpn.xwiki.internal.template.TemplateAsyncRenderer.render(TemplateAsyncRenderer.java:135)
 at com.xpn.xwiki.internal.template.TemplateAsyncRenderer.render(TemplateAsyncRenderer.java:54)
 at org.xwiki.rendering.async.internal.DefaultAsyncRendererExecutor.lambda$syncRender$0(DefaultAsyncRendererExecutor.java:284)
 at com.xpn.xwiki.internal.security.authorization.DefaultAuthorExecutor.call(DefaultAuthorExecutor.java:98)
 at org.xwiki.rendering.async.internal.DefaultAsyncRendererExecutor.syncRender(DefaultAsyncRendererExecutor.java:284)
 at org.xwiki.rendering.async.internal.DefaultAsyncRendererExecutor.render(DefaultAsyncRendererExecutor.java:267)
 at org.xwiki.rendering.async.internal.block.DefaultBlockAsyncRendererExecutor.render(DefaultBlockAsyncRendererExecutor.java:154)
 at com.xpn.xwiki.internal.template.InternalTemplateManager.render(InternalTemplateManager.java:904)
 at com.xpn.xwiki.internal.template.InternalTemplateManager.renderFromSkin(InternalTemplateManager.java:866)
 at com.xpn.xwiki.internal.template.InternalTemplateManager.render(InternalTemplateManager.java:853)
 at com.xpn.xwiki.internal.template.InternalTemplateManager.renderNoException(InternalTemplateManager.java:808)
 at com.xpn.xwiki.internal.template.InternalTemplateManager.renderNoException(InternalTemplateManager.java:800)
 at com.xpn.xwiki.internal.template.DefaultTemplateManager.renderNoException(DefaultTemplateManager.java:79)
 at com.xpn.xwiki.internal.template.DefaultTemplateManager.renderNoException(DefaultTemplateManager.java:73)
 at org.xwiki.template.script.TemplateScriptService.render(TemplateScriptService.java:54)
 at jdk.internal.reflect.GeneratedMethodAccessor246.invoke(Unknown Source)
 at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
 at java.base/java.lang.reflect.Method.invoke(Unknown Source)
 at org.apache.velocity.util.introspection.UberspectImpl$VelMethodImpl.doInvoke(UberspectImpl.java:571)
 at org.apache.velocity.util.introspection.UberspectImpl$VelMethodImpl.invoke(UberspectImpl.java:554)
 at org.apache.velocity.runtime.parser.node.ASTMethod.execute(ASTMethod.java:221)
 at org.apache.velocity.runtime.parser.node.ASTReference.execute(ASTReference.java:368)
 at org.apache.velocity.runtime.parser.node.ASTReference.render(ASTReference.java:492)
 at org.apache.velocity.runtime.parser.node.ASTBlock.render(ASTBlock.java:147)
 at org.apache.velocity.runtime.directive.VelocimacroProxy.render(VelocimacroProxy.java:218)
 at org.apache.velocity.runtime.directive.RuntimeMacro.render(RuntimeMacro.java:331)
 at org.apache.velocity.runtime.directive.RuntimeMacro.render(RuntimeMacro.java:261)
 at org.apache.velocity.runtime.parser.node.ASTDirective.render(ASTDirective.java:304)
 at org.apache.velocity.runtime.parser.node.ASTBlock.render(ASTBlock.java:147)
 at org.apache.velocity.runtime.parser.node.ASTIfStatement.render(ASTIfStatement.java:171)
 at org.apache.velocity.runtime.parser.node.ASTBlock.render(ASTBlock.java:147)
 at org.apache.velocity.runtime.parser.node.SimpleNode.render(SimpleNode.java:439)
 at org.apache.velocity.runtime.parser.node.ASTIfStatement.render(ASTIfStatement.java:190)
 at org.apache.velocity.runtime.parser.node.SimpleNode.render(SimpleNode.java:439)
 at org.apache.velocity.Template.merge(Template.java:358)
 at org.apache.velocity.Template.merge(Template.java:262)
 at org.xwiki.velocity.internal.InternalVelocityEngine.evaluate(InternalVelocityEngine.java:225)
 at com.xpn.xwiki.internal.template.VelocityTemplateEvaluator.evaluateContent(VelocityTemplateEvaluator.java:105)
 at com.xpn.xwiki.internal.template.TemplateAsyncRenderer.evaluateContent(TemplateAsyncRenderer.java:219)
 at com.xpn.xwiki.internal.template.TemplateAsyncRenderer.renderVelocity(TemplateAsyncRenderer.java:174)
 at com.xpn.xwiki.internal.template.TemplateAsyncRenderer.render(TemplateAsyncRenderer.java:135)
 at com.xpn.xwiki.internal.template.TemplateAsyncRenderer.render(TemplateAsyncRenderer.java:54)
 at org.xwiki.rendering.async.internal.DefaultAsyncRendererExecutor.lambda$syncRender$0(DefaultAsyncRendererExecutor.java:284)
 at com.xpn.xwiki.internal.security.authorization.DefaultAuthorExecutor.call(DefaultAuthorExecutor.java:98)
 at org.xwiki.rendering.async.internal.DefaultAsyncRendererExecutor.syncRender(DefaultAsyncRendererExecutor.java:284)
 at org.xwiki.rendering.async.internal.DefaultAsyncRendererExecutor.render(DefaultAsyncRendererExecutor.java:267)
 at org.xwiki.rendering.async.internal.block.DefaultBlockAsyncRendererExecutor.render(DefaultBlockAsyncRendererExecutor.java:154)
 at com.xpn.xwiki.internal.template.InternalTemplateManager.render(InternalTemplateManager.java:904)
 at com.xpn.xwiki.internal.template.InternalTemplateManager.renderFromSkin(InternalTemplateManager.java:866)
 at com.xpn.xwiki.internal.template.InternalTemplateManager.renderFromSkin(InternalTemplateManager.java:846)
 at com.xpn.xwiki.internal.template.InternalTemplateManager.render(InternalTemplateManager.java:832)
 at com.xpn.xwiki.internal.template.DefaultTemplateManager.render(DefaultTemplateManager.java:91)
 at com.xpn.xwiki.internal.template.DefaultTemplateManager.render(DefaultTemplateManager.java:85)
 at com.xpn.xwiki.XWiki.evaluateTemplate(XWiki.java:2564)
 at com.xpn.xwiki.web.Utils.parseTemplate(Utils.java:180)
 at com.xpn.xwiki.web.XWikiAction.execute(XWikiAction.java:651)
 at com.xpn.xwiki.web.XWikiAction.execute(XWikiAction.java:339)
 at com.xpn.xwiki.web.LegacyActionServlet.service(LegacyActionServlet.java:108)
 at javax.servlet.http.HttpServlet.service(HttpServlet.java:623)
 at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:199)
 at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:144)
 at com.xpn.xwiki.web.ActionFilter.doFilter(ActionFilter.java:122)
 at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:168)
 at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:144)
 at org.xwiki.wysiwyg.filter.ConversionFilter.doFilter(ConversionFilter.java:61)
 at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:168)
 at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:144)
 at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:51)
 at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:168)
 at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:144)
 at org.xwiki.container.servlet.filters.internal.SetHTTPHeaderFilter.doFilter(SetHTTPHeaderFilter.java:63)
 at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:168)
 at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:144)
 at org.xwiki.resource.servlet.RoutingFilter.doFilter(RoutingFilter.java:132)
 at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:168)
 at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:144)
 at org.xwiki.container.servlet.filters.internal.SavedRequestRestorerFilter.doFilter(SavedRequestRestorerFilter.java:208)
 at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:168)
 at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:144)
 at org.xwiki.container.servlet.filters.internal.SetCharacterEncodingFilter.doFilter(SetCharacterEncodingFilter.java:117)
 at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:168)
 at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:144)
 at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:168)
 at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:90)
 at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:482)
 at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:130)
 at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:93)
 at org.apache.catalina.valves.AbstractAccessLogValve.invoke(AbstractAccessLogValve.java:660)
 at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:74)
 at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:346)
 at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:383)
 at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:63)
 at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:937)
 at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1791)
 at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:52)
 at org.apache.tomcat.util.threads.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1190)
 at org.apache.tomcat.util.threads.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:659)
 at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:63)
 at java.base/java.lang.Thread.run(Unknown Source)
Caused by: org.xwiki.velocity.XWikiVelocityException: Failed to evaluate content with namespace [xwiki:Main.Search]
 at org.xwiki.velocity.internal.InternalVelocityEngine.evaluate(InternalVelocityEngine.java:227)
 at org.xwiki.rendering.internal.macro.velocity.VelocityMacro.evaluateString(VelocityMacro.java:168)
 ... 185 more
Caused by: org.apache.velocity.exception.MethodInvocationException: Invocation of method 'execute' in  class org.xwiki.query.internal.ScriptQuery threw exception org.xwiki.query.QueryException: Fail to lookup query executor. Query statement = [suche] at xwiki:Main.SolrSearchMacros[line 589, column 32]
 at org.apache.velocity.runtime.parser.node.ASTMethod.handleInvocationException(ASTMethod.java:308)
 at org.apache.velocity.runtime.parser.node.ASTMethod.execute(ASTMethod.java:235)
 at org.apache.velocity.runtime.parser.node.ASTReference.execute(ASTReference.java:368)
 at org.apache.velocity.runtime.parser.node.ASTReference.value(ASTReference.java:704)
 at org.apache.velocity.runtime.parser.node.ASTExpression.value(ASTExpression.java:75)
 at org.apache.velocity.runtime.parser.node.ASTSetDirective.render(ASTSetDirective.java:242)
 at org.apache.velocity.runtime.parser.node.ASTBlock.render(ASTBlock.java:147)
 at org.apache.velocity.runtime.directive.VelocimacroProxy.render(VelocimacroProxy.java:218)
 at org.apache.velocity.runtime.directive.RuntimeMacro.render(RuntimeMacro.java:331)
 at org.apache.velocity.runtime.directive.RuntimeMacro.render(RuntimeMacro.java:261)
 at org.apache.velocity.runtime.parser.node.ASTDirective.render(ASTDirective.java:304)
 at org.apache.velocity.runtime.parser.node.ASTBlock.render(ASTBlock.java:147)
 at org.apache.velocity.runtime.parser.node.ASTIfStatement.render(ASTIfStatement.java:171)
 at org.apache.velocity.runtime.parser.node.ASTBlock.render(ASTBlock.java:147)
 at org.apache.velocity.runtime.directive.VelocimacroProxy.render(VelocimacroProxy.java:218)
 at org.apache.velocity.runtime.directive.RuntimeMacro.render(RuntimeMacro.java:331)
 at org.apache.velocity.runtime.directive.RuntimeMacro.render(RuntimeMacro.java:261)
 at org.apache.velocity.runtime.parser.node.ASTDirective.render(ASTDirective.java:304)
 at org.apache.velocity.runtime.parser.node.ASTBlock.render(ASTBlock.java:147)
 at org.apache.velocity.runtime.parser.node.ASTElseIfStatement.render(ASTElseIfStatement.java:1)

the xwiki container shows this message:

2024-10-16 16:55:49,388 [http-nio-8080-exec-72 - http://192.168.5.6:8100/bin/view/Main/Search?text=suche&f_type=DOCUMENT&f_locale=de&f_locale=&r=1] INFO  o.x.s.s.i.EmbeddedSolr         - Updating Solr home directory at [/usr/local/xwiki/data/store/solr]
2024-10-16 16:55:49,389 [http-nio-8080-exec-49 - http://192.168.5.6:8100/bin/get/XWiki/SuggestSolrService?query=fq%3Dtype%3AATTACHMENT%0Aqf%3Dfilename&nb=3&outputSyntax=plain&input=suche] ERROR o.a.v.d.null                   - Exception in macro #getSearchSuggestResults called at xwi
ki:XWiki.SuggestSolrService[line 2, column 1]
2024-10-16 16:55:49,391 [http-nio-8080-exec-72 - http://192.168.5.6:8100/bin/view/Main/Search?text=suche&f_type=DOCUMENT&f_locale=de&f_locale=&r=1] INFO  o.x.s.s.i.EmbeddedSolr         - Starting embedded Solr server...
2024-10-16 16:55:49,392 [http-nio-8080-exec-72 - http://192.168.5.6:8100/bin/view/Main/Search?text=suche&f_type=DOCUMENT&f_locale=de&f_locale=&r=1] INFO  o.x.s.s.i.EmbeddedSolr         - Using Solr home directory: [/usr/local/xwiki/data/store/solr]
2024-10-16 16:55:49,491 [http-nio-8080-exec-72 - http://192.168.5.6:8100/bin/view/Main/Search?text=suche&f_type=DOCUMENT&f_locale=de&f_locale=&r=1] WARN  o.a.s.c.CoreContainer          - Not all security plugins configured!  authentication=disabled authorization=disabled.  Solr is on
ly as secure as you make it. Consider configuring authentication/authorization before exposing Solr to users internal or external.  See https://s.apache.org/solrsecurity for more info
2024-10-16 16:55:49,563 [http-nio-8080-exec-72 - http://192.168.5.6:8100/bin/view/Main/Search?text=suche&f_type=DOCUMENT&f_locale=de&f_locale=&r=1] ERROR o.x.s.s.i.EmbeddedSolr         - Failed to initialize Solr core with id [search_9]
org.apache.solr.common.SolrException: Could not load conf for core search_9: Error loading parsing schema
        at org.apache.solr.core.ConfigSetService.loadConfigSet(ConfigSetService.java:278)
        at org.apache.solr.core.CoreContainer.createFromDescriptor(CoreContainer.java:1696)
        at org.apache.solr.core.CoreContainer.lambda$loadInternal$12(CoreContainer.java:1044)
        at com.codahale.metrics.InstrumentedExecutorService$InstrumentedCallable.call(InstrumentedExecutorService.java:234)
        at java.base/java.util.concurrent.FutureTask.run(Unknown Source)
        at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:294)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
        at java.base/java.lang.Thread.run(Unknown Source)
Caused by: org.apache.solr.common.SolrException: Error loading parsing schema
        at org.apache.solr.schema.ManagedIndexSchemaFactory.create(ManagedIndexSchemaFactory.java:289)
        at org.apache.solr.schema.ManagedIndexSchemaFactory.create(ManagedIndexSchemaFactory.java:54)
        at org.apache.solr.core.ConfigSetService.createIndexSchema(ConfigSetService.java:357)
        at org.apache.solr.core.ConfigSetService.lambda$loadConfigSet$0(ConfigSetService.java:268)
        at org.apache.solr.core.ConfigSet.<init>(ConfigSet.java:49)
        at org.apache.solr.core.ConfigSetService.loadConfigSet(ConfigSetService.java:264)
        ... 8 common frames omitted
Caused by: org.apache.solr.common.SolrException: Can't load schema /usr/local/xwiki/data/store/solr/search_9/conf/managed-schema.xml: org.apache.solr.core.SolrResourceNotFoundException: Can't find resource 'org/languagetool/resource/uk/ukrainian.dict' in classpath or '/usr/local/xwik
i/data/store/solr/search_9'
        at org.apache.solr.schema.IndexSchema.readSchema(IndexSchema.java:662)
        at org.apache.solr.schema.IndexSchema.<init>(IndexSchema.java:192)
        at org.apache.solr.schema.ManagedIndexSchema.<init>(ManagedIndexSchema.java:112)
        at org.apache.solr.schema.ManagedIndexSchemaFactory.create(ManagedIndexSchemaFactory.java:287)
        ... 13 common frames omitted
Caused by: org.apache.solr.common.SolrException: org.apache.solr.core.SolrResourceNotFoundException: Can't find resource 'org/languagetool/resource/uk/ukrainian.dict' in classpath or '/usr/local/xwiki/data/store/solr/search_9'
        at org.apache.solr.schema.IndexSchema.informResourceLoaderAwareObjectsInChain(IndexSchema.java:2097)
        at org.apache.solr.schema.IndexSchema.informResourceLoaderAwareObjectsForFieldType(IndexSchema.java:2007)
        at java.base/java.util.HashMap$Values.forEach(Unknown Source)
        at org.apache.solr.schema.IndexSchema.postReadInform(IndexSchema.java:684)
        at org.apache.solr.schema.IndexSchema.readSchema(IndexSchema.java:658)
        ... 16 common frames omitted
Caused by: org.apache.solr.core.SolrResourceNotFoundException: Can't find resource 'org/languagetool/resource/uk/ukrainian.dict' in classpath or '/usr/local/xwiki/data/store/solr/search_9'
        at org.apache.solr.core.SolrResourceLoader.openResource(SolrResourceLoader.java:382)
        at org.apache.lucene.analysis.morfologik.MorfologikFilterFactory.inform(MorfologikFilterFactory.java:104)
        at org.apache.solr.core.SolrResourceLoader.informAware(SolrResourceLoader.java:765)
        at org.apache.solr.schema.IndexSchema.informResourceLoaderAwareObjectsInChain(IndexSchema.java:2095)
        ... 20 common frames omitted

What I noticed was this line:

Caused by: org.apache.solr.common.SolrException: Can't load schema /usr/local/xwiki/data/store/solr/search_9/conf/managed-schema.xml: org.apache.solr.core.SolrResourceNotFoundException: Can't find resource 'org/languagetool/resource/uk/ukrainian.dict' in classpath or '/usr/local/xwik
i/data/store/solr/search_9'

and a quick search showed me this, I have no clue about solr so I don’t know how helpful this is.

In Solr version 7, the ukrainian.dict , which enables the analysis of the Ukrainian language, has been removed. The following steps describe the necessary changes to adapt the schema accordingly

Setup

We’re running xwiki in docker with this compose file

version: '3.4'
services:
  db:
    environment:
      - POSTGRES_ROOT_PASSWORD=${POSTGRES_ROOT_PASSWORD}
      - POSTGRES_PASSWORD=${POSTGRES_ROOT_PASSWORD}
      - POSTGRES_USER=xwiki
      - POSTGRES_DB=xwiki
      - POSTGRES_INITDB_ARGS="--encoding=UTF8"
    image: postgres:15
    restart: unless-stopped
    networks:
      - xwiki
    volumes:
      - /opt/docker/xwiki/postgres:/var/lib/postgresql/data
  xwiki:
    depends_on:
      - db
    environment:
      - DB_USER=xwiki
      - DB_PASSWORD=${POSTGRES_ROOT_PASSWORD}
      - DB_HOST=db
      - JAVA_OPTS=-Xmx2048m -Dlog4j2.formatMsgNoLookups=true
    image: xwiki:stable-postgres-tomcat
    restart: unless-stopped
    networks:
      - xwiki
      - proxy
    ports:
      - 8100:8080
    volumes:
      - /opt/docker/xwiki/xwiki:/usr/local/xwiki

networks:
  xwiki: null
  proxy:
    external: true

ok, I wen’t down that rabbit hole.

I copied the file /usr/local/xwiki/data/store/solr/search_9/conf/managed-schema.xml out of the container, commented out the ukranian language:

    <!-- XWIKI: Ukrainian -->
    <!--
    <dynamicField name="*_uk" type="text_uk"  indexed="true"  stored="true" multiValued="true"/>
    <fieldType name="text_uk" class="solr.TextField" positionIncrementGap="100">
      <analyzer>
        <tokenizer name="standard"/>
        <filter name="stop" words="org/apache/lucene/analysis/uk/stopwords.txt"/>
        <filter name="lowercase"/>
        <filter name="morfologik" dictionary="org/languagetool/resource/uk/ukrainian.dict"/>
      </analyzer>
    </fieldType>
    -->

then bind mounted this file:

        volumes:
        - /opt/docker/xwiki/xwiki:/usr/local/xwiki
        - /opt/docker/xwiki/server.xml:/usr/local/tomcat/conf/server.xml
        - /opt/docker/xwiki/managed-schema.xml:/usr/local/xwiki/data/store/solr/search_9/conf/managed-schema.xml

and search is now working again :tada:

and it seems to go to this commit: XWIKI-18416: Ukrainian content is not supported by the Solr search · xwiki/xwiki-platform@9694062 · GitHub

You seem to have a missing JAR in your setup, this file is supposed to be located in morfologik-ukrainian-lt-6.5.1.jar which is supposed to be in <permdir>/store/solr/search_9/lib.

I suspect you are only going to find an outdated xwiki-platform-search-solr-server-plugin jar file. I think we are missing a migration to reset the search core. To fix this for now what you can do is stop XWiki and delete the store/solr/search_9 and cache/solr/search_9 (i.e. reset the search core so that a clean one is re-created).

ok, I understand now, thank you very much.
since I have the /usr/local/xwiki bind mounted outside of the container, the data still is there when upgrading xwiki.
And yes indeed under that directory I have a xwiki-platform-search-solr-server-plugin-16.6.0.jar
I will check with one instance to fix this by deleting the directories.
Was suspecting that something might have been left over but couldn’t identify any informations in the release notes. I suspect other people using docker and upgrading will face the same issue.

What is still not very clear to me is how you ended up with a 16.8.0 version of store/solr/search_9/conf/managed-schema.xml but an outdated store/solr/search_9/lib/. Have you updated it yourself by hand when you upgraded XWiki to 16.8.0 ? Without doing anything you should have an a 16.6.0 store/solr/search_9/ (which should work fine, even with XWiki 16.8.0, you just don’t have Ukrainian support).

all I did was:

  • docker compose pull which pulls the latest xwiki:stable-postgres-tomcat
  • docker compose down
  • docker compose up -d

after that the search was broken as mentioned above. I did absolutely nothing.

But here is what happens.
The official documentations for the docker installation mentions to set up a volume for the xwiki data:

 volumes:
      - xwiki-data:/usr/local/xwiki

But this means that the solr data is also saved there!
/usr/local/xwiki/data/store/solr/search_9/conf/managed-schema.xml from the container is probably copied to the new location, but store/solr/search_9/lib/ won’t be cleared and it is persisted from the old version?
Do I even need to persist everything out of /usr/local/xwiki? or should one only put /usr/local/xwiki/store in a volume?

/usr/local/xwiki/data/ is only data which is supposed to be created at runtime, I don’t really see why the container would contain any file that would go in there.
XWiki does not touch store/solr/search_9/ if the version of Solr does not change right now, and when it does, it deletes the whole store/solr/search_9/ and recreate it. Currently, I don’t have any explanation on XWiki side on how to end up with a partially updated store/solr/search_9/ core.

Any idea @vmassol ?

let me try to reproduce. I will set up an older xwiki, upgrade and see if I can reproduce the issue and also if there are any logs or something I might have missed.
Need to get some work done before that, I’ve spend quite some time going down this rabbit hole.

1 Like

Thanks @precut-banked !

of course I could not reproduce it :joy: Tried installing fresh 16.4.x and then upgraded to 16.8.0, search was still working.
Not sure if I should go back to older version and try that way or just clean up the 16.8 install as you mentioned and call it a day.