Xwiki 11.3 to 11.6.1 upgrade fail [almost solved]

pdwalker · August 26, 2019, 9:16am

Normally, after upgrading xwiki and loading the page for the first time, the updater will run and update all the extensions.

This time, it didn’t and I was left with a wiki lots of incompatible extensions and nothing was really accessible.

Fortunately, I had a page open to the extension updater, so I could run the updater from there. So far, the manual update seems to be working (it’s still running and figuring out what to do), so hopefully this is a non-issue.

I have no idea why the extension updater didn’t run. That’s the first time I’ve seen that happen.

tmortagne · August 26, 2019, 9:32am

Hard to say. Do you have any kind of error in the log ?

pdwalker · August 26, 2019, 9:36am

Good question:

This was the first error to appear after starting tomcat8 after the upgrade

root@wiki:/var/log/tomcat8# service tomcat8 start && tail -f catalina.out
Aug 26, 2019 4:53:43 PM org.apache.catalina.loader.WebappClassLoaderBase checkThreadLocalMapForLeaks
SEVERE: The web application [xwiki] created a ThreadLocal with key of type [com.xpn.xwiki.store.migration.AbstractDataMigrationManager.ThreadLock] (value [com.xpn.xwiki.store.migration.AbstractDataMigrationManager$ThreadLock@1e7e018d]) and a value of type [java.lang.Integer] (value [0]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
Aug 26, 2019 4:53:43 PM org.apache.catalina.loader.WebappClassLoaderBase checkThreadLocalMapForLeaks
SEVERE: The web application [xwiki] created a ThreadLocal with key of type [com.xpn.xwiki.store.migration.AbstractDataMigrationManager.ThreadLock] (value [com.xpn.xwiki.store.migration.AbstractDataMigrationManager$ThreadLock@1e7e018d]) and a value of type [java.lang.Integer] (value [0]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
Aug 26, 2019 4:53:43 PM org.apache.coyote.AbstractProtocol stop
INFO: Stopping ProtocolHandler ["http-nio-80"]
Aug 26, 2019 4:53:43 PM org.apache.coyote.AbstractProtocol destroy
INFO: Destroying ProtocolHandler ["http-nio-80"]

tmortagne · August 26, 2019, 9:52am

Those are part of the stop log actually, not the start one.

pdwalker · August 26, 2019, 9:54am

The next warning was

INFO: Deploying configuration descriptor /etc/tomcat8/Catalina/localhost/xwiki.xml
2019-08-26 16:54:32,332 [localhost-startStop-1] INFO  iPropertiesConfigurationSource - loading xwiki.properties from default location /etc/xwiki/xwiki.properties
2019-08-26 16:54:32,655 [localhost-startStop-1] INFO  o.x.e.i.ServletEnvironment     - Using permanent directory [/var/lib/xwiki/data]
2019-08-26 16:54:38,939 [localhost-startStop-1] WARN  ltInstalledExtensionRepository - Invalid extension [org.xwiki.platform:xwiki-platform-distribution-flavor-tour/11.4] on namespace [wiki:xwiki] (InvalidExtensionException: No compatible extension is installed for dependency [org.xwiki.platform:xwiki-platform-appwithinminutes-ui-11.4])
2019-08-26 16:54:38,948 [localhost-startStop-1] WARN  ltInstalledExtensionRepository - Invalid extension [org.xwiki.platform:xwiki-platform-distribution-flavor-tour/11.4] on namespace [wiki:it] (InvalidExtensionException: No compatible extension is installed for dependency [org.xwiki.platform:xwiki-platform-appwithinminutes-ui-11.4])
2019-08-26 16:54:38,967 [localhost-startStop-1] WARN  ltInstalledExtensionRepository - Invalid extension [org.xwiki.platform:xwiki-platform-distribution-flavor-wiki/11.3] on namespace [wiki:it] (InvalidExtensionException: Dependency [org.xwiki.platform:xwiki-platform-oldcore-[11.3]] is incompatible with the core extension [org.xwiki.platform:xwiki-platform-legacy-oldcore/11.6.1])
2019-08-26 16:54:38,984 [localhost-startStop-1] WARN  ltInstalledExtensionRepository - Invalid extension [org.xwiki.platform:xwiki-platform-distribution-flavor-wiki/11.3] on namespace [wiki:archive] (InvalidExtensionException: Dependency [org.xwiki.platform:xwiki-platform-oldcore-[11.3]] is incompatible with the core extension [org.xwiki.platform:xwiki-platform-legacy-oldcore/11.6.1])
2019-08-26 16:54:38,999 [localhost-startStop-1] WARN  ltInstalledExtensionRepository - Invalid extension [org.xwiki.platform:xwiki-platform-distribution-flavor-mainwiki/11.3] on namespace [wiki:xwiki] (InvalidExtensionException: Dependency [org.xwiki.platform:xwiki-platform-oldcore-[11.3]] is incompatible with the core extension [org.xwiki.platform:xwiki-platform-legacy-oldcore/11.6.1])
2019-08-26 16:54:39,001 [localhost-startStop-1] WARN  ltInstalledExtensionRepository - Invalid extension [org.xwiki.contrib.confluence:confluence-xml/9.2.3] on namespace [wiki:xwiki] (InvalidExtensionException: No compatible extension is installed for dependency [commons-configuration:commons-configuration-1.10])

pdwalker · August 26, 2019, 9:58am

You know what? Don’t worry about it. It seems to be updating correctly now, so hopefully my problems go away after it completes. It might have just been something weird that caused the updater to not run.

It’s probably not worth going after.

tmortagne · August 26, 2019, 9:59am

All that looks OK, EM just identified that those extensions are not valid anymore because of the upgrade and that’s what the distribution wizard is about.

tmortagne · August 26, 2019, 10:03am

Do you have any customization in the skin ? Do you see something when you directly go to the Distribution Wizard (https://myhost/xwiki/bin/distribution/XWiki/Distribution).

pdwalker · August 26, 2019, 10:16am

I get this:

edit: I’m running the updater in another window.

tmortagne · August 26, 2019, 10:26am

So it does not seems to be a missing redirect issue. The DW is not started but no idea why, an invalid flavor and current user being admin is supposed to be all it takes.

Could you check what you have in <permdir>/jobs/status/distribution/status.xml file

pdwalker · August 26, 2019, 10:33am

status.xml.txt (8.3 KB)

tmortagne · August 26, 2019, 10:42am

That’s weird, this is the status of a 11.3 instance, not a 11.6.1 one. As if the DW was not executed at all.

Do you have a /usr/lib/xwiki/WEB-INF/lib//xwiki-platform-extension-distribution-11.6.1.jar file ?

pdwalker · August 26, 2019, 10:47am

Correct. The DW didn’t run at all after upgrading xwiki.

root@wiki:~# ls -l /usr/lib/xwiki/WEB-INF/lib//xwiki-platform-extension-distribution*
-rw-r--r-- 1 root root 75649 Aug  9 00:07 /usr/lib/xwiki/WEB-INF/lib//xwiki-platform-extension-distribution-11.6.1.jar
-rw-r--r-- 1 root root 37394 Aug  9 00:07 /usr/lib/xwiki/WEB-INF/lib//xwiki-platform-extension-distribution-11.6.1.xed

I’m still running the updater. I’ll keep running it until there are no more updates shown.

pdwalker · August 26, 2019, 11:04am

ok, no more updates show. I’ve restarted tomcat, and this is the first warning that looks weird that I see

2019-08-26 19:02:32,319 [localhost-startStop-1] WARN ltInstalledExtensionRepository - Invalid extension [org.xwiki.platform:xwiki-platform-distribution-flavor-wiki/11.3] on namespace [wiki:archive] (InvalidExtensionException: Dependency [org.xwiki.platform:xwiki-platform-oldcore-[11.3]] is incompatible with the core extension [org.xwiki.platform:xwiki-platform-legacy-oldcore/11.6.1])

pdwalker · August 26, 2019, 11:06am

The first big error is a solr error. (not sure if this is related or not):

2019-08-26 19:02:42,817 [localhost-startStop-1] ERROR .o.i.DefaultObservationManager - Failed to lookup listeners
org.xwiki.component.manager.ComponentLookupException: Failed to lookup component with type [interface org.xwiki.observation.EventListener] and hint [solr.availablelocales]
	at org.xwiki.component.embed.EmbeddableComponentManager.getInstanceMap(EmbeddableComponentManager.java:245)
	at org.xwiki.component.embed.EmbeddableComponentManager.getInstanceList(EmbeddableComponentManager.java:225)
	at org.xwiki.observation.internal.DefaultObservationManager.initializeListeners(DefaultObservationManager.java:166)
	at org.xwiki.observation.internal.DefaultObservationManager.getListenersByEvent(DefaultObservationManager.java:132)
	at org.xwiki.observation.internal.DefaultObservationManager.notify(DefaultObservationManager.java:283)
	at org.xwiki.component.internal.StackingComponentEventManager.sendEvent(StackingComponentEventManager.java:142)
	at org.xwiki.component.internal.StackingComponentEventManager.flushEvents(StackingComponentEventManager.java:92)
	at org.xwiki.extension.jar.internal.handler.JarExtensionHandler.loadComponents(JarExtensionHandler.java:237)
	at org.xwiki.extension.jar.internal.handler.JarExtensionHandler.install(JarExtensionHandler.java:165)
	at org.xwiki.extension.jar.internal.handler.JarExtensionHandler.initialize(JarExtensionHandler.java:143)
	at org.xwiki.extension.handler.internal.DefaultExtensionHandlerManager.initialize(DefaultExtensionHandlerManager.java:154)
	at org.xwiki.extension.handler.internal.DefaultExtensionInitializer.initializeExtensionInNamespace(DefaultExtensionInitializer.java:182)
	at org.xwiki.extension.handler.internal.DefaultExtensionInitializer.initializeExtensionInNamespace(DefaultExtensionInitializer.java:161)
	at org.xwiki.extension.handler.internal.DefaultExtensionInitializer.initializeExtensionDependencyInNamespace(DefaultExtensionInitializer.java:206)
	at org.xwiki.extension.handler.internal.DefaultExtensionInitializer.initializeExtensionInNamespace(DefaultExtensionInitializer.java:177)
	at org.xwiki.extension.handler.internal.DefaultExtensionInitializer.initializeExtensionInNamespace(DefaultExtensionInitializer.java:161)
	at org.xwiki.extension.handler.internal.DefaultExtensionInitializer.initializeExtension(DefaultExtensionInitializer.java:133)
	at org.xwiki.extension.handler.internal.DefaultExtensionInitializer.initialize(DefaultExtensionInitializer.java:105)
	at org.xwiki.extension.handler.internal.DefaultExtensionInitializer.initialize(DefaultExtensionInitializer.java:81)
	at org.xwiki.component.embed.InitializableLifecycleHandler.handle(InitializableLifecycleHandler.java:39)
	at org.xwiki.component.embed.EmbeddableComponentManager.createInstance(EmbeddableComponentManager.java:365)
	at org.xwiki.component.embed.EmbeddableComponentManager.getComponentInstance(EmbeddableComponentManager.java:451)
	at org.xwiki.component.embed.EmbeddableComponentManager.getInstance(EmbeddableComponentManager.java:201)
	at org.xwiki.component.embed.EmbeddableComponentManager.getInstance(EmbeddableComponentManager.java:189)
	at org.xwiki.container.servlet.XWikiServletContextListener.contextInitialized(XWikiServletContextListener.java:120)
	at org.apache.catalina.core.StandardContext.listenerStart(StandardContext.java:4821)
	at org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:5264)
	at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:147)
	at org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:725)
	at org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:701)
	at org.apache.catalina.core.StandardHost.addChild(StandardHost.java:717)
	at org.apache.catalina.startup.HostConfig.deployDescriptor(HostConfig.java:585)
	at org.apache.catalina.startup.HostConfig$DeployDescriptor.run(HostConfig.java:1794)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
Caused by: org.xwiki.component.manager.ComponentLookupException: Failed to lookup component [org.xwiki.search.solr.internal.SolrInstanceProvider] identified by type [javax.inject.Provider<org.xwiki.search.solr.internal.api.SolrInstance>] and hint [default]
	at org.xwiki.component.embed.EmbeddableComponentManager.getInstance(EmbeddableComponentManager.java:204)
	at org.xwiki.component.embed.EmbeddableComponentManager.getDependencyInstance(EmbeddableComponentManager.java:399)
	at org.xwiki.component.embed.EmbeddableComponentManager.createInstance(EmbeddableComponentManager.java:355)
	at org.xwiki.component.embed.EmbeddableComponentManager.getComponentInstance(EmbeddableComponentManager.java:451)
	at org.xwiki.component.embed.EmbeddableComponentManager.getInstanceMap(EmbeddableComponentManager.java:242)
	... 37 common frames omitted
Caused by: org.xwiki.component.phase.InitializationException: Failed to lookup configured Solr instance type [embedded]
	at org.xwiki.search.solr.internal.SolrInstanceProvider.initialize(SolrInstanceProvider.java:67)
	at org.xwiki.component.embed.InitializableLifecycleHandler.handle(InitializableLifecycleHandler.java:39)
	at org.xwiki.component.embed.EmbeddableComponentManager.createInstance(EmbeddableComponentManager.java:365)
	at org.xwiki.component.embed.EmbeddableComponentManager.getComponentInstance(EmbeddableComponentManager.java:451)
	at org.xwiki.component.embed.EmbeddableComponentManager.getInstance(EmbeddableComponentManager.java:201)
	... 41 common frames omitted
Caused by: org.xwiki.component.manager.ComponentLookupException: Failed to lookup component [org.xwiki.search.solr.internal.EmbeddedSolrInstance] identified by type [interface org.xwiki.search.solr.internal.api.SolrInstance] and hint [embedded]
	at org.xwiki.component.embed.EmbeddableComponentManager.getInstance(EmbeddableComponentManager.java:204)
	at org.xwiki.search.solr.internal.SolrInstanceProvider.initialize(SolrInstanceProvider.java:64)
	... 45 common frames omitted
Caused by: org.xwiki.component.phase.InitializationException: Failed to initialize the Solr embedded server with home directory set to [/var/lib/xwiki/data/solr]
	at org.xwiki.search.solr.internal.EmbeddedSolrInstance.initialize(EmbeddedSolrInstance.java:102)
	at org.xwiki.component.embed.InitializableLifecycleHandler.handle(InitializableLifecycleHandler.java:39)
	at org.xwiki.component.embed.EmbeddableComponentManager.createInstance(EmbeddableComponentManager.java:365)
	at org.xwiki.component.embed.EmbeddableComponentManager.getComponentInstance(EmbeddableComponentManager.java:451)
	at org.xwiki.component.embed.EmbeddableComponentManager.getInstance(EmbeddableComponentManager.java:201)
	... 46 common frames omitted
Caused by: org.apache.solr.client.solrj.SolrServerException: Failed to initialize the Solr core. Please check the configuration and log messages.
	at org.xwiki.search.solr.internal.EmbeddedSolrInstance.createCoreContainer(EmbeddedSolrInstance.java:112)
	at org.xwiki.search.solr.internal.EmbeddedSolrInstance.initialize(EmbeddedSolrInstance.java:95)
	... 50 common frames omitted
2019-08-26 19:02:42,832 [Finalizer] ERROR o.a.s.u.SolrIndexWriter        - SolrIndexWriter was not closed prior to finalize(), indicates a bug -- POSSIBLE RESOURCE LEAK!!!
2019-08-26 19:02:42,835 [Finalizer] ERROR o.a.s.u.SolrIndexWriter        - Error closing IndexWriter
java.lang.NullPointerException: null
	at org.apache.lucene.index.IndexWriter.doFlush(IndexWriter.java:3605)
	at org.apache.lucene.index.IndexWriter.flush(IndexWriter.java:3565)
	at org.apache.lucene.index.IndexWriter.shutdown(IndexWriter.java:1044)
	at org.apache.lucene.index.IndexWriter.close(IndexWriter.java:1087)
	at org.apache.solr.update.SolrIndexWriter.close(SolrIndexWriter.java:301)
	at org.apache.solr.update.SolrIndexWriter.finalize(SolrIndexWriter.java:357)
	at java.lang.System$2.invokeFinalize(System.java:1270)
	at java.lang.ref.Finalizer.runFinalizer(Finalizer.java:102)
	at java.lang.ref.Finalizer.access$100(Finalizer.java:34)
	at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:217)
	Suppressed: java.lang.NullPointerException: null
		at org.apache.lucene.index.IndexWriter.rollbackInternalNoCommit(IndexWriter.java:2290)
		at org.apache.lucene.index.IndexWriter.rollbackInternal(IndexWriter.java:2268)
		at org.apache.lucene.index.IndexWriter.shutdown(IndexWriter.java:1051)
		... 7 common frames omitted

pdwalker · August 26, 2019, 11:09am

ok, tomcat reloaded. I try to load the initial wiki home page and see the following error (First of many):

2019-08-26 19:07:05,228 [http://wiki.charltonslaw.local/xwiki/bin/view/Main/] ERROR l.j.i.JARFileTranslationBundle - Failed to parse resource [jar:file:/var/lib/xwiki/data/extension/repository/io%252Enetty%253Anetty-tcnative%253A%2524%257Bos%252Edetected%252Eclassifier%257D/2%252E0%252E25%252EFinal/io%252Enetty%253Anetty-tcnative%253A%2524%257Bos%252Edetected%252Eclassifier%257D-2%252E0%252E25%252EFinal.jar!/ApplicationResources_en_US.properties] as translation bundle
java.util.zip.ZipException: zip file is empty
	at java.util.zip.ZipFile.open(Native Method)
	at java.util.zip.ZipFile.<init>(ZipFile.java:225)
	at java.util.zip.ZipFile.<init>(ZipFile.java:155)
	at java.util.jar.JarFile.<init>(JarFile.java:166)
	at java.util.jar.JarFile.<init>(JarFile.java:103)
	at sun.net.www.protocol.jar.URLJarFile.<init>(URLJarFile.java:93)
	at sun.net.www.protocol.jar.URLJarFile.getJarFile(URLJarFile.java:69)
	at sun.net.www.protocol.jar.JarFileFactory.get(JarFileFactory.java:99)
	at sun.net.www.protocol.jar.JarURLConnection.connect(JarURLConnection.java:122)
	at sun.net.www.protocol.jar.JarURLConnection.getInputStream(JarURLConnection.java:152)
	at java.net.URL.openStream(URL.java:1045)
	at org.xwiki.localization.internal.AbstractURLResourceTranslationBundle.loadResourceLocaleBundle(AbstractURLResourceTranslationBundle.java:143)
	at org.xwiki.localization.internal.AbstractURLResourceTranslationBundle.createBundle(AbstractURLResourceTranslationBundle.java:183)
	at org.xwiki.localization.internal.AbstractCachedTranslationBundle.getSynchLocalizedBundle(AbstractCachedTranslationBundle.java:91)
	at org.xwiki.localization.internal.AbstractCachedTranslationBundle.getLocalizedBundle(AbstractCachedTranslationBundle.java:75)
	at org.xwiki.localization.internal.AbstractCachedTranslationBundle.getTranslation(AbstractCachedTranslationBundle.java:108)
	at org.xwiki.localization.internal.DefaultLocalizationManager.getTranslation(DefaultLocalizationManager.java:76)
	at org.xwiki.localization.script.LocalizationScriptService.render(LocalizationScriptService.java:364)
	at org.xwiki.localization.script.LocalizationScriptService.render(LocalizationScriptService.java:341)
	at org.xwiki.localization.script.LocalizationScriptService.render(LocalizationScriptService.java:314)
	at org.xwiki.localization.script.LocalizationScriptService.render(LocalizationScriptService.java:215)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.apache.velocity.util.introspection.UberspectImpl$VelMethodImpl.doInvoke(UberspectImpl.java:395)
	at org.apache.velocity.util.introspection.UberspectImpl$VelMethodImpl.invoke(UberspectImpl.java:384)
	at org.apache.velocity.runtime.parser.node.ASTMethod.execute(ASTMethod.java:173)
	at org.apache.velocity.runtime.parser.node.ASTReference.execute(ASTReference.java:280)
	at org.apache.velocity.runtime.parser.node.ASTReference.render(ASTReference.java:369)
	at org.apache.velocity.runtime.parser.node.ASTBlock.render(ASTBlock.java:72)
	at org.apache.velocity.runtime.parser.node.SimpleNode.render(SimpleNode.java:342)
	at org.apache.velocity.runtime.parser.node.ASTIfStatement.render(ASTIfStatement.java:106)
	at org.apache.velocity.runtime.parser.node.ASTBlock.render(ASTBlock.java:72)
	at org.apache.velocity.runtime.parser.node.ASTIfStatement.render(ASTIfStatement.java:87)
	at org.apache.velocity.runtime.parser.node.SimpleNode.render(SimpleNode.java:342)
	at org.xwiki.velocity.internal.DefaultVelocityEngine.evaluateInternal(DefaultVelocityEngine.java:259)
	at org.xwiki.velocity.internal.DefaultVelocityEngine.evaluate(DefaultVelocityEngine.java:222)
	at com.xpn.xwiki.render.DefaultVelocityManager.evaluate(DefaultVelocityManager.java:358)
	at com.xpn.xwiki.internal.template.InternalTemplateManager.evaluateContent(InternalTemplateManager.java:884)
	at com.xpn.xwiki.internal.template.InternalTemplateManager.render(InternalTemplateManager.java:755)
	at com.xpn.xwiki.internal.template.InternalTemplateManager.lambda$renderFromSkin$0(InternalTemplateManager.java:730)
	at com.xpn.xwiki.internal.security.authorization.DefaultAuthorExecutor.call(DefaultAuthorExecutor.java:98)
	at com.xpn.xwiki.internal.template.InternalTemplateManager.renderFromSkin(InternalTemplateManager.java:729)
	at com.xpn.xwiki.internal.template.InternalTemplateManager.render(InternalTemplateManager.java:715)
	at com.xpn.xwiki.internal.template.DefaultTemplateManager.render(DefaultTemplateManager.java:108)
	at com.xpn.xwiki.web.XWikiAction.renderInit(XWikiAction.java:656)
	at com.xpn.xwiki.web.XWikiAction.execute(XWikiAction.java:287)
	at com.xpn.xwiki.web.XWikiAction.execute(XWikiAction.java:215)
	at org.apache.struts.action.RequestProcessor.processActionPerform(RequestProcessor.java:425)
	at org.apache.struts.action.RequestProcessor.process(RequestProcessor.java:228)
	at org.apache.struts.action.ActionServlet.process(ActionServlet.java:1913)
	at org.apache.struts.action.ActionServlet.doGet(ActionServlet.java:449)
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:622)
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:729)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:292)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:207)
	at com.xpn.xwiki.web.ActionFilter.doFilter(ActionFilter.java:112)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:240)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:207)
	at org.xwiki.wysiwyg.filter.ConversionFilter.doFilter(ConversionFilter.java:109)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:240)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:207)
	at org.xwiki.container.servlet.filters.internal.SetHTTPHeaderFilter.doFilter(SetHTTPHeaderFilter.java:63)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:240)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:207)
	at org.xwiki.container.servlet.filters.internal.SavedRequestRestorerFilter.doFilter(SavedRequestRestorerFilter.java:208)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:240)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:207)
	at org.xwiki.container.servlet.filters.internal.SetCharacterEncodingFilter.doFilter(SetCharacterEncodingFilter.java:111)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:240)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:207)
	at org.xwiki.resource.servlet.RoutingFilter.doFilter(RoutingFilter.java:132)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:240)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:207)
	at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:213)
	at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:106)
	at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:494)
	at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:141)
	at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:79)
	at org.apache.catalina.valves.AbstractAccessLogValve.invoke(AbstractAccessLogValve.java:616)
	at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:88)
	at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:522)
	at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1095)
	at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:672)
	at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1520)
	at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.run(NioEndpoint.java:1476)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
	at java.lang.Thread.run(Thread.java:748)

pdwalker · August 26, 2019, 11:14am

and sure enough, that jar file is 0 bytes.

ok, so somehow I got a bunch of 0 byte jar files

root@wiki:/var/lib/xwiki/data/extension/repository# find . -type f -size 0 -print
./io%2Enetty%3Anetty-transport-native-epoll%3A%24%7Bos%2Edetected%2Ename%7D-%24%7Bos%2Edetected%2Earch%7D/4%2E0%2E52%2EFinal/io%2Enetty%3Anetty-transport-native-epoll%3A%24%7Bos%2Edetected%2Ename%7D-%24%7Bos%2Edetected%2Earch%7D-4%2E0%2E52%2EFinal.jar
./io%2Enetty%3Anetty-transport-native-epoll%3A%24%7Bos%2Edetected%2Ename%7D-%24%7Bos%2Edetected%2Earch%7D/4%2E1%2E39%2EFinal/io%2Enetty%3Anetty-transport-native-epoll%3A%24%7Bos%2Edetected%2Ename%7D-%24%7Bos%2Edetected%2Earch%7D-4%2E1%2E39%2EFinal.jar
./io%2Enetty%3Anetty-tcnative%3A%24%7Bos%2Edetected%2Eclassifier%7D/2%2E0%2E6%2EFinal/io%2Enetty%3Anetty-tcnative%3A%24%7Bos%2Edetected%2Eclassifier%7D-2%2E0%2E6%2EFinal.jar
./io%2Enetty%3Anetty-tcnative%3A%24%7Bos%2Edetected%2Eclassifier%7D/2%2E0%2E25%2EFinal/io%2Enetty%3Anetty-tcnative%3A%24%7Bos%2Edetected%2Eclassifier%7D-2%2E0%2E25%2EFinal.jar
./io%2Enetty%3Anetty-transport-native-unix-common%3A%24%7Bos%2Edetected%2Ename%7D-%24%7Bos%2Edetected%2Earch%7D/4%2E1%2E39%2EFinal/io%2Enetty%3Anetty-transport-native-unix-common%3A%24%7Bos%2Edetected%2Ename%7D-%24%7Bos%2Edetected%2Earch%7D-4%2E1%2E39%2EFinal.jar

pdwalker · August 26, 2019, 11:40am

So now I have essentially a broken wiki. All the page macros are coming up on my pages as “unknown macro: X” where X is the name of the macro.

Should I uninstall and then reinstall the wiki to attempt resolve this problem? Find replacements for the 0 byte jar files?

Any advice or suggestions would be appreciated.

tmortagne · August 26, 2019, 11:41am

Have you deleted /var/lib/xwiki/data/solr as indicated on https://www.xwiki.org/xwiki/bin/view/ReleaseNotes/Data/XWiki/11.4/#HIssuesspecifictoXWiki11.4 ? If not then it’s most probably the root cause of everything.

pdwalker · August 26, 2019, 11:45am

I missed that instruction in the 11.4 release notes. That’s resolved the solr errors, but not the 0 byte jar file problem.