Solr Indexer Crashed

localby · June 24, 2019, 5:14pm

The bug is described at Loading...

Solr Indexer crashed when index the article like this:

== 1. Header 1 ==
Text by header 1
== 2.1. Header 2 ==
Text by header 2.1

When we modify it to (correct 2.1. to 2.) it start to work correct:

== 1. Header 1 ==
Text by header 1
== 2. Header 2 ==
Text by header 2

The xwiki 11.4 Tomcat Docker + Mysql 5.7 Docker

2019-06-24 17:20:24,843 [XWiki Solr index thread] ERROR o.a.s.h.RequestHandlerBase - org.apache.solr.common.SolrException: Exception writing document id xwiki:Sandbox.ArticleTest.WebHome_ru to the index; possible analysis error: startOffset must be non-negative, and endOffset must be >= startOffset, and offsets must not go backwards startOffset=40,endOffset=41,lastStartOffset=41 for field ‘doccontentraw_ru’
at org.apache.solr.update.DirectUpdateHandler2.addDoc(DirectUpdateHandler2.java:243)
at org.apache.solr.update.processor.RunUpdateProcessor.processAdd(RunUpdateProcessorFactory.java:67)
at org.apache.solr.update.processor.UpdateRequestProcessor.processAdd(UpdateRequestProcessor.java:55)
at org.apache.solr.update.processor.DistributedUpdateProcessor.doLocalAdd(DistributedUpdateProcessor.java:1001)
at org.apache.solr.update.processor.DistributedUpdateProcessor.versionAdd(DistributedUpdateProcessor.java:1222)
at org.apache.solr.update.processor.DistributedUpdateProcessor.processAdd(DistributedUpdateProcessor.java:693)
at org.apache.solr.handler.loader.JavabinLoader$1.update(JavabinLoader.java:110)
at org.apache.solr.client.solrj.request.JavaBinUpdateRequestCodec$StreamingCodec.readOuterMostDocIterator(JavaBinUpdateRequestCodec.java:327)
at org.apache.solr.client.solrj.request.JavaBinUpdateRequestCodec$StreamingCodec.readIterator(JavaBinUpdateRequestCodec.java:280)
at org.apache.solr.common.util.JavaBinCodec.readObject(JavaBinCodec.java:333)
at org.apache.solr.common.util.JavaBinCodec.readVal(JavaBinCodec.java:278)
at org.apache.solr.client.solrj.request.JavaBinUpdateRequestCodec$StreamingCodec.readNamedList(JavaBinUpdateRequestCodec.java:235)
at org.apache.solr.common.util.JavaBinCodec.readObject(JavaBinCodec.java:298)
at org.apache.solr.common.util.JavaBinCodec.readVal(JavaBinCodec.java:278)
at org.apache.solr.common.util.JavaBinCodec.unmarshal(JavaBinCodec.java:191)
at org.apache.solr.client.solrj.request.JavaBinUpdateRequestCodec.unmarshal(JavaBinUpdateRequestCodec.java:126)
at org.apache.solr.handler.loader.JavabinLoader.parseAndLoadDocs(JavabinLoader.java:123)
at org.apache.solr.handler.loader.JavabinLoader.load(JavabinLoader.java:70)
at org.apache.solr.handler.UpdateRequestHandler$1.load(UpdateRequestHandler.java:97)
at org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(ContentStreamHandlerBase.java:68)
at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:199)
at org.apache.solr.core.SolrCore.execute(SolrCore.java:2551)
at org.apache.solr.client.solrj.embedded.EmbeddedSolrServer.request(EmbeddedSolrServer.java:191)
at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:194)
at org.apache.solr.client.solrj.SolrClient.add(SolrClient.java:177)
at org.apache.solr.client.solrj.SolrClient.add(SolrClient.java:138)
at org.apache.solr.client.solrj.SolrClient.add(SolrClient.java:156)
at org.xwiki.search.solr.internal.AbstractSolrInstance.add(AbstractSolrInstance.java:61)
at org.xwiki.search.solr.internal.DefaultSolrIndexer.processBatch(DefaultSolrIndexer.java:413)
at org.xwiki.search.solr.internal.DefaultSolrIndexer.run(DefaultSolrIndexer.java:377)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.IllegalArgumentException: startOffset must be non-negative, and endOffset must be >= startOffset, and offsets must not go backwards startOffset=40,endOffset=41,lastStartOffset=41 for field ‘doccontentraw_ru’
at org.apache.lucene.index.DefaultIndexingChain$PerField.invert(DefaultIndexingChain.java:824)
at org.apache.lucene.index.DefaultIndexingChain.processField(DefaultIndexingChain.java:430)
at org.apache.lucene.index.DefaultIndexingChain.processDocument(DefaultIndexingChain.java:394)
at org.apache.lucene.index.DocumentsWriterPerThread.updateDocument(DocumentsWriterPerThread.java:251)
at org.apache.lucene.index.DocumentsWriter.updateDocument(DocumentsWriter.java:494)
at org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:1616)
at org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:1608)
at org.apache.solr.update.DirectUpdateHandler2.updateDocOrDocValues(DirectUpdateHandler2.java:969)
at org.apache.solr.update.DirectUpdateHandler2.doNormalUpdate(DirectUpdateHandler2.java:341)
at org.apache.solr.update.DirectUpdateHandler2.addDoc0(DirectUpdateHandler2.java:288)
at org.apache.solr.update.DirectUpdateHandler2.addDoc(DirectUpdateHandler2.java:235)
… 30 more

2019-06-24 17:20:24,843 [XWiki Solr index thread] ERROR o.x.s.s.i.DefaultSolrIndexer - Failed to process entry [INDEX xwiki:Sandbox.ArticleTest.WebHome]
org.apache.solr.common.SolrException: Exception writing document id xwiki:Sandbox.ArticleTest.WebHome_ru to the index; possible analysis error: startOffset must be non-negative, and endOffset must be >= startOffset, and offsets must not go backwards startOffset=40,endOffset=41,lastStartOffset=41 for field ‘doccontentraw_ru’
at org.apache.solr.update.DirectUpdateHandler2.addDoc(DirectUpdateHandler2.java:243)
at org.apache.solr.update.processor.RunUpdateProcessor.processAdd(RunUpdateProcessorFactory.java:67)
at org.apache.solr.update.processor.UpdateRequestProcessor.processAdd(UpdateRequestProcessor.java:55)
at org.apache.solr.update.processor.DistributedUpdateProcessor.doLocalAdd(DistributedUpdateProcessor.java:1001)
at org.apache.solr.update.processor.DistributedUpdateProcessor.versionAdd(DistributedUpdateProcessor.java:1222)
at org.apache.solr.update.processor.DistributedUpdateProcessor.processAdd(DistributedUpdateProcessor.java:693)
at org.apache.solr.handler.loader.JavabinLoader$1.update(JavabinLoader.java:110)
at org.apache.solr.client.solrj.request.JavaBinUpdateRequestCodec$StreamingCodec.readOuterMostDocIterator(JavaBinUpdateRequestCodec.java:327)
at org.apache.solr.client.solrj.request.JavaBinUpdateRequestCodec$StreamingCodec.readIterator(JavaBinUpdateRequestCodec.java:280)
at org.apache.solr.common.util.JavaBinCodec.readObject(JavaBinCodec.java:333)
at org.apache.solr.common.util.JavaBinCodec.readVal(JavaBinCodec.java:278)
at org.apache.solr.client.solrj.request.JavaBinUpdateRequestCodec$StreamingCodec.readNamedList(JavaBinUpdateRequestCodec.java:235)
at org.apache.solr.common.util.JavaBinCodec.readObject(JavaBinCodec.java:298)
at org.apache.solr.common.util.JavaBinCodec.readVal(JavaBinCodec.java:278)
at org.apache.solr.common.util.JavaBinCodec.unmarshal(JavaBinCodec.java:191)
at org.apache.solr.client.solrj.request.JavaBinUpdateRequestCodec.unmarshal(JavaBinUpdateRequestCodec.java:126)
at org.apache.solr.handler.loader.JavabinLoader.parseAndLoadDocs(JavabinLoader.java:123)
at org.apache.solr.handler.loader.JavabinLoader.load(JavabinLoader.java:70)
at org.apache.solr.handler.UpdateRequestHandler$1.load(UpdateRequestHandler.java:97)
at org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(ContentStreamHandlerBase.java:68)
at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:199)
at org.apache.solr.core.SolrCore.execute(SolrCore.java:2551)
at org.apache.solr.client.solrj.embedded.EmbeddedSolrServer.request(EmbeddedSolrServer.java:191)
at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:194)
at org.apache.solr.client.solrj.SolrClient.add(SolrClient.java:177)
at org.apache.solr.client.solrj.SolrClient.add(SolrClient.java:138)
at org.apache.solr.client.solrj.SolrClient.add(SolrClient.java:156)
at org.xwiki.search.solr.internal.AbstractSolrInstance.add(AbstractSolrInstance.java:61)
at org.xwiki.search.solr.internal.DefaultSolrIndexer.processBatch(DefaultSolrIndexer.java:413)
at org.xwiki.search.solr.internal.DefaultSolrIndexer.run(DefaultSolrIndexer.java:377)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.IllegalArgumentException: startOffset must be non-negative, and endOffset must be >= startOffset, and offsets must not go backwards startOffset=40,endOffset=41,lastStartOffset=41 for field ‘doccontentraw_ru’
at org.apache.lucene.index.DefaultIndexingChain$PerField.invert(DefaultIndexingChain.java:824)
at org.apache.lucene.index.DefaultIndexingChain.processField(DefaultIndexingChain.java:430)
at org.apache.lucene.index.DefaultIndexingChain.processDocument(DefaultIndexingChain.java:394)
at org.apache.lucene.index.DocumentsWriterPerThread.updateDocument(DocumentsWriterPerThread.java:251)
at org.apache.lucene.index.DocumentsWriter.updateDocument(DocumentsWriter.java:494)
at org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:1616)
at org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:1608)
at org.apache.solr.update.DirectUpdateHandler2.updateDocOrDocValues(DirectUpdateHandler2.java:969)
at org.apache.solr.update.DirectUpdateHandler2.doNormalUpdate(DirectUpdateHandler2.java:341)
at org.apache.solr.update.DirectUpdateHandler2.addDoc0(DirectUpdateHandler2.java:288)
at org.apache.solr.update.DirectUpdateHandler2.addDoc(DirectUpdateHandler2.java:235)
… 30 common frames omitted

localby · June 25, 2019, 8:02am

Word Delimiter Filter

https://lucene.apache.org/solr/guide/6_6/filter-descriptions.html

Word Delimiter Filter has been deprecated in favor of Word Delimiter Graph Filter, which is required to produce a correct token graph so that e.g. phrase queries can work correctly.

Solution: use Word Delimiter Graph Filter instead of Word Delimiter Filter in your config file.

mflorea · June 25, 2019, 8:06am

Solution: use Word Delimiter Graph Filter instead of Word Delimiter Filter in your config file.

This has been fixed in XWiki 11.4RC1 by the upgrade to Solr 7.7.1. Did you upgrade to XWiki 11.4 or is it a clean install? Because the default Solr index schema in XWiki 11.4 doesn’t use Word Delimiter Filter, see xwiki-platform/xwiki-platform-core/xwiki-platform-search/xwiki-platform-search-solr/xwiki-platform-search-solr-server/xwiki-platform-search-solr-server-data/src/main/resources/xwiki/conf/managed-schema at xwiki-platform-11.4 · xwiki/xwiki-platform · GitHub . Only Word Delimiter Graph Filter is used.

localby · June 25, 2019, 12:41pm

Thanks!
I have the old Scheme file with Word Delimiter and I edit it manually to fix the problem.