Xwiki 11.3 to 11.6.1 upgrade fail [almost solved]

ok, no more updates show. I’ve restarted tomcat, and this is the first warning that looks weird that I see

2019-08-26 19:02:32,319 [localhost-startStop-1] WARN ltInstalledExtensionRepository - Invalid extension [org.xwiki.platform:xwiki-platform-distribution-flavor-wiki/11.3] on namespace [wiki:archive] (InvalidExtensionException: Dependency [org.xwiki.platform:xwiki-platform-oldcore-[11.3]] is incompatible with the core extension [org.xwiki.platform:xwiki-platform-legacy-oldcore/11.6.1])

The first big error is a solr error. (not sure if this is related or not):

2019-08-26 19:02:42,817 [localhost-startStop-1] ERROR .o.i.DefaultObservationManager - Failed to lookup listeners
org.xwiki.component.manager.ComponentLookupException: Failed to lookup component with type [interface org.xwiki.observation.EventListener] and hint [solr.availablelocales]
	at org.xwiki.component.embed.EmbeddableComponentManager.getInstanceMap(EmbeddableComponentManager.java:245)
	at org.xwiki.component.embed.EmbeddableComponentManager.getInstanceList(EmbeddableComponentManager.java:225)
	at org.xwiki.observation.internal.DefaultObservationManager.initializeListeners(DefaultObservationManager.java:166)
	at org.xwiki.observation.internal.DefaultObservationManager.getListenersByEvent(DefaultObservationManager.java:132)
	at org.xwiki.observation.internal.DefaultObservationManager.notify(DefaultObservationManager.java:283)
	at org.xwiki.component.internal.StackingComponentEventManager.sendEvent(StackingComponentEventManager.java:142)
	at org.xwiki.component.internal.StackingComponentEventManager.flushEvents(StackingComponentEventManager.java:92)
	at org.xwiki.extension.jar.internal.handler.JarExtensionHandler.loadComponents(JarExtensionHandler.java:237)
	at org.xwiki.extension.jar.internal.handler.JarExtensionHandler.install(JarExtensionHandler.java:165)
	at org.xwiki.extension.jar.internal.handler.JarExtensionHandler.initialize(JarExtensionHandler.java:143)
	at org.xwiki.extension.handler.internal.DefaultExtensionHandlerManager.initialize(DefaultExtensionHandlerManager.java:154)
	at org.xwiki.extension.handler.internal.DefaultExtensionInitializer.initializeExtensionInNamespace(DefaultExtensionInitializer.java:182)
	at org.xwiki.extension.handler.internal.DefaultExtensionInitializer.initializeExtensionInNamespace(DefaultExtensionInitializer.java:161)
	at org.xwiki.extension.handler.internal.DefaultExtensionInitializer.initializeExtensionDependencyInNamespace(DefaultExtensionInitializer.java:206)
	at org.xwiki.extension.handler.internal.DefaultExtensionInitializer.initializeExtensionInNamespace(DefaultExtensionInitializer.java:177)
	at org.xwiki.extension.handler.internal.DefaultExtensionInitializer.initializeExtensionInNamespace(DefaultExtensionInitializer.java:161)
	at org.xwiki.extension.handler.internal.DefaultExtensionInitializer.initializeExtension(DefaultExtensionInitializer.java:133)
	at org.xwiki.extension.handler.internal.DefaultExtensionInitializer.initialize(DefaultExtensionInitializer.java:105)
	at org.xwiki.extension.handler.internal.DefaultExtensionInitializer.initialize(DefaultExtensionInitializer.java:81)
	at org.xwiki.component.embed.InitializableLifecycleHandler.handle(InitializableLifecycleHandler.java:39)
	at org.xwiki.component.embed.EmbeddableComponentManager.createInstance(EmbeddableComponentManager.java:365)
	at org.xwiki.component.embed.EmbeddableComponentManager.getComponentInstance(EmbeddableComponentManager.java:451)
	at org.xwiki.component.embed.EmbeddableComponentManager.getInstance(EmbeddableComponentManager.java:201)
	at org.xwiki.component.embed.EmbeddableComponentManager.getInstance(EmbeddableComponentManager.java:189)
	at org.xwiki.container.servlet.XWikiServletContextListener.contextInitialized(XWikiServletContextListener.java:120)
	at org.apache.catalina.core.StandardContext.listenerStart(StandardContext.java:4821)
	at org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:5264)
	at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:147)
	at org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:725)
	at org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:701)
	at org.apache.catalina.core.StandardHost.addChild(StandardHost.java:717)
	at org.apache.catalina.startup.HostConfig.deployDescriptor(HostConfig.java:585)
	at org.apache.catalina.startup.HostConfig$DeployDescriptor.run(HostConfig.java:1794)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
Caused by: org.xwiki.component.manager.ComponentLookupException: Failed to lookup component [org.xwiki.search.solr.internal.SolrInstanceProvider] identified by type [javax.inject.Provider<org.xwiki.search.solr.internal.api.SolrInstance>] and hint [default]
	at org.xwiki.component.embed.EmbeddableComponentManager.getInstance(EmbeddableComponentManager.java:204)
	at org.xwiki.component.embed.EmbeddableComponentManager.getDependencyInstance(EmbeddableComponentManager.java:399)
	at org.xwiki.component.embed.EmbeddableComponentManager.createInstance(EmbeddableComponentManager.java:355)
	at org.xwiki.component.embed.EmbeddableComponentManager.getComponentInstance(EmbeddableComponentManager.java:451)
	at org.xwiki.component.embed.EmbeddableComponentManager.getInstanceMap(EmbeddableComponentManager.java:242)
	... 37 common frames omitted
Caused by: org.xwiki.component.phase.InitializationException: Failed to lookup configured Solr instance type [embedded]
	at org.xwiki.search.solr.internal.SolrInstanceProvider.initialize(SolrInstanceProvider.java:67)
	at org.xwiki.component.embed.InitializableLifecycleHandler.handle(InitializableLifecycleHandler.java:39)
	at org.xwiki.component.embed.EmbeddableComponentManager.createInstance(EmbeddableComponentManager.java:365)
	at org.xwiki.component.embed.EmbeddableComponentManager.getComponentInstance(EmbeddableComponentManager.java:451)
	at org.xwiki.component.embed.EmbeddableComponentManager.getInstance(EmbeddableComponentManager.java:201)
	... 41 common frames omitted
Caused by: org.xwiki.component.manager.ComponentLookupException: Failed to lookup component [org.xwiki.search.solr.internal.EmbeddedSolrInstance] identified by type [interface org.xwiki.search.solr.internal.api.SolrInstance] and hint [embedded]
	at org.xwiki.component.embed.EmbeddableComponentManager.getInstance(EmbeddableComponentManager.java:204)
	at org.xwiki.search.solr.internal.SolrInstanceProvider.initialize(SolrInstanceProvider.java:64)
	... 45 common frames omitted
Caused by: org.xwiki.component.phase.InitializationException: Failed to initialize the Solr embedded server with home directory set to [/var/lib/xwiki/data/solr]
	at org.xwiki.search.solr.internal.EmbeddedSolrInstance.initialize(EmbeddedSolrInstance.java:102)
	at org.xwiki.component.embed.InitializableLifecycleHandler.handle(InitializableLifecycleHandler.java:39)
	at org.xwiki.component.embed.EmbeddableComponentManager.createInstance(EmbeddableComponentManager.java:365)
	at org.xwiki.component.embed.EmbeddableComponentManager.getComponentInstance(EmbeddableComponentManager.java:451)
	at org.xwiki.component.embed.EmbeddableComponentManager.getInstance(EmbeddableComponentManager.java:201)
	... 46 common frames omitted
Caused by: org.apache.solr.client.solrj.SolrServerException: Failed to initialize the Solr core. Please check the configuration and log messages.
	at org.xwiki.search.solr.internal.EmbeddedSolrInstance.createCoreContainer(EmbeddedSolrInstance.java:112)
	at org.xwiki.search.solr.internal.EmbeddedSolrInstance.initialize(EmbeddedSolrInstance.java:95)
	... 50 common frames omitted
2019-08-26 19:02:42,832 [Finalizer] ERROR o.a.s.u.SolrIndexWriter        - SolrIndexWriter was not closed prior to finalize(), indicates a bug -- POSSIBLE RESOURCE LEAK!!!
2019-08-26 19:02:42,835 [Finalizer] ERROR o.a.s.u.SolrIndexWriter        - Error closing IndexWriter
java.lang.NullPointerException: null
	at org.apache.lucene.index.IndexWriter.doFlush(IndexWriter.java:3605)
	at org.apache.lucene.index.IndexWriter.flush(IndexWriter.java:3565)
	at org.apache.lucene.index.IndexWriter.shutdown(IndexWriter.java:1044)
	at org.apache.lucene.index.IndexWriter.close(IndexWriter.java:1087)
	at org.apache.solr.update.SolrIndexWriter.close(SolrIndexWriter.java:301)
	at org.apache.solr.update.SolrIndexWriter.finalize(SolrIndexWriter.java:357)
	at java.lang.System$2.invokeFinalize(System.java:1270)
	at java.lang.ref.Finalizer.runFinalizer(Finalizer.java:102)
	at java.lang.ref.Finalizer.access$100(Finalizer.java:34)
	at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:217)
	Suppressed: java.lang.NullPointerException: null
		at org.apache.lucene.index.IndexWriter.rollbackInternalNoCommit(IndexWriter.java:2290)
		at org.apache.lucene.index.IndexWriter.rollbackInternal(IndexWriter.java:2268)
		at org.apache.lucene.index.IndexWriter.shutdown(IndexWriter.java:1051)
		... 7 common frames omitted

ok, tomcat reloaded. I try to load the initial wiki home page and see the following error (First of many):

2019-08-26 19:07:05,228 [http://wiki.charltonslaw.local/xwiki/bin/view/Main/] ERROR l.j.i.JARFileTranslationBundle - Failed to parse resource [jar:file:/var/lib/xwiki/data/extension/repository/io%252Enetty%253Anetty-tcnative%253A%2524%257Bos%252Edetected%252Eclassifier%257D/2%252E0%252E25%252EFinal/io%252Enetty%253Anetty-tcnative%253A%2524%257Bos%252Edetected%252Eclassifier%257D-2%252E0%252E25%252EFinal.jar!/ApplicationResources_en_US.properties] as translation bundle
java.util.zip.ZipException: zip file is empty
	at java.util.zip.ZipFile.open(Native Method)
	at java.util.zip.ZipFile.<init>(ZipFile.java:225)
	at java.util.zip.ZipFile.<init>(ZipFile.java:155)
	at java.util.jar.JarFile.<init>(JarFile.java:166)
	at java.util.jar.JarFile.<init>(JarFile.java:103)
	at sun.net.www.protocol.jar.URLJarFile.<init>(URLJarFile.java:93)
	at sun.net.www.protocol.jar.URLJarFile.getJarFile(URLJarFile.java:69)
	at sun.net.www.protocol.jar.JarFileFactory.get(JarFileFactory.java:99)
	at sun.net.www.protocol.jar.JarURLConnection.connect(JarURLConnection.java:122)
	at sun.net.www.protocol.jar.JarURLConnection.getInputStream(JarURLConnection.java:152)
	at java.net.URL.openStream(URL.java:1045)
	at org.xwiki.localization.internal.AbstractURLResourceTranslationBundle.loadResourceLocaleBundle(AbstractURLResourceTranslationBundle.java:143)
	at org.xwiki.localization.internal.AbstractURLResourceTranslationBundle.createBundle(AbstractURLResourceTranslationBundle.java:183)
	at org.xwiki.localization.internal.AbstractCachedTranslationBundle.getSynchLocalizedBundle(AbstractCachedTranslationBundle.java:91)
	at org.xwiki.localization.internal.AbstractCachedTranslationBundle.getLocalizedBundle(AbstractCachedTranslationBundle.java:75)
	at org.xwiki.localization.internal.AbstractCachedTranslationBundle.getTranslation(AbstractCachedTranslationBundle.java:108)
	at org.xwiki.localization.internal.DefaultLocalizationManager.getTranslation(DefaultLocalizationManager.java:76)
	at org.xwiki.localization.script.LocalizationScriptService.render(LocalizationScriptService.java:364)
	at org.xwiki.localization.script.LocalizationScriptService.render(LocalizationScriptService.java:341)
	at org.xwiki.localization.script.LocalizationScriptService.render(LocalizationScriptService.java:314)
	at org.xwiki.localization.script.LocalizationScriptService.render(LocalizationScriptService.java:215)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.apache.velocity.util.introspection.UberspectImpl$VelMethodImpl.doInvoke(UberspectImpl.java:395)
	at org.apache.velocity.util.introspection.UberspectImpl$VelMethodImpl.invoke(UberspectImpl.java:384)
	at org.apache.velocity.runtime.parser.node.ASTMethod.execute(ASTMethod.java:173)
	at org.apache.velocity.runtime.parser.node.ASTReference.execute(ASTReference.java:280)
	at org.apache.velocity.runtime.parser.node.ASTReference.render(ASTReference.java:369)
	at org.apache.velocity.runtime.parser.node.ASTBlock.render(ASTBlock.java:72)
	at org.apache.velocity.runtime.parser.node.SimpleNode.render(SimpleNode.java:342)
	at org.apache.velocity.runtime.parser.node.ASTIfStatement.render(ASTIfStatement.java:106)
	at org.apache.velocity.runtime.parser.node.ASTBlock.render(ASTBlock.java:72)
	at org.apache.velocity.runtime.parser.node.ASTIfStatement.render(ASTIfStatement.java:87)
	at org.apache.velocity.runtime.parser.node.SimpleNode.render(SimpleNode.java:342)
	at org.xwiki.velocity.internal.DefaultVelocityEngine.evaluateInternal(DefaultVelocityEngine.java:259)
	at org.xwiki.velocity.internal.DefaultVelocityEngine.evaluate(DefaultVelocityEngine.java:222)
	at com.xpn.xwiki.render.DefaultVelocityManager.evaluate(DefaultVelocityManager.java:358)
	at com.xpn.xwiki.internal.template.InternalTemplateManager.evaluateContent(InternalTemplateManager.java:884)
	at com.xpn.xwiki.internal.template.InternalTemplateManager.render(InternalTemplateManager.java:755)
	at com.xpn.xwiki.internal.template.InternalTemplateManager.lambda$renderFromSkin$0(InternalTemplateManager.java:730)
	at com.xpn.xwiki.internal.security.authorization.DefaultAuthorExecutor.call(DefaultAuthorExecutor.java:98)
	at com.xpn.xwiki.internal.template.InternalTemplateManager.renderFromSkin(InternalTemplateManager.java:729)
	at com.xpn.xwiki.internal.template.InternalTemplateManager.render(InternalTemplateManager.java:715)
	at com.xpn.xwiki.internal.template.DefaultTemplateManager.render(DefaultTemplateManager.java:108)
	at com.xpn.xwiki.web.XWikiAction.renderInit(XWikiAction.java:656)
	at com.xpn.xwiki.web.XWikiAction.execute(XWikiAction.java:287)
	at com.xpn.xwiki.web.XWikiAction.execute(XWikiAction.java:215)
	at org.apache.struts.action.RequestProcessor.processActionPerform(RequestProcessor.java:425)
	at org.apache.struts.action.RequestProcessor.process(RequestProcessor.java:228)
	at org.apache.struts.action.ActionServlet.process(ActionServlet.java:1913)
	at org.apache.struts.action.ActionServlet.doGet(ActionServlet.java:449)
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:622)
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:729)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:292)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:207)
	at com.xpn.xwiki.web.ActionFilter.doFilter(ActionFilter.java:112)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:240)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:207)
	at org.xwiki.wysiwyg.filter.ConversionFilter.doFilter(ConversionFilter.java:109)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:240)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:207)
	at org.xwiki.container.servlet.filters.internal.SetHTTPHeaderFilter.doFilter(SetHTTPHeaderFilter.java:63)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:240)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:207)
	at org.xwiki.container.servlet.filters.internal.SavedRequestRestorerFilter.doFilter(SavedRequestRestorerFilter.java:208)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:240)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:207)
	at org.xwiki.container.servlet.filters.internal.SetCharacterEncodingFilter.doFilter(SetCharacterEncodingFilter.java:111)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:240)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:207)
	at org.xwiki.resource.servlet.RoutingFilter.doFilter(RoutingFilter.java:132)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:240)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:207)
	at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:213)
	at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:106)
	at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:494)
	at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:141)
	at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:79)
	at org.apache.catalina.valves.AbstractAccessLogValve.invoke(AbstractAccessLogValve.java:616)
	at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:88)
	at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:522)
	at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1095)
	at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:672)
	at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1520)
	at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.run(NioEndpoint.java:1476)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
	at java.lang.Thread.run(Thread.java:748)

and sure enough, that jar file is 0 bytes.

ok, so somehow I got a bunch of 0 byte jar files

root@wiki:/var/lib/xwiki/data/extension/repository# find . -type f -size 0 -print
./io%2Enetty%3Anetty-transport-native-epoll%3A%24%7Bos%2Edetected%2Ename%7D-%24%7Bos%2Edetected%2Earch%7D/4%2E0%2E52%2EFinal/io%2Enetty%3Anetty-transport-native-epoll%3A%24%7Bos%2Edetected%2Ename%7D-%24%7Bos%2Edetected%2Earch%7D-4%2E0%2E52%2EFinal.jar
./io%2Enetty%3Anetty-transport-native-epoll%3A%24%7Bos%2Edetected%2Ename%7D-%24%7Bos%2Edetected%2Earch%7D/4%2E1%2E39%2EFinal/io%2Enetty%3Anetty-transport-native-epoll%3A%24%7Bos%2Edetected%2Ename%7D-%24%7Bos%2Edetected%2Earch%7D-4%2E1%2E39%2EFinal.jar
./io%2Enetty%3Anetty-tcnative%3A%24%7Bos%2Edetected%2Eclassifier%7D/2%2E0%2E6%2EFinal/io%2Enetty%3Anetty-tcnative%3A%24%7Bos%2Edetected%2Eclassifier%7D-2%2E0%2E6%2EFinal.jar
./io%2Enetty%3Anetty-tcnative%3A%24%7Bos%2Edetected%2Eclassifier%7D/2%2E0%2E25%2EFinal/io%2Enetty%3Anetty-tcnative%3A%24%7Bos%2Edetected%2Eclassifier%7D-2%2E0%2E25%2EFinal.jar
./io%2Enetty%3Anetty-transport-native-unix-common%3A%24%7Bos%2Edetected%2Ename%7D-%24%7Bos%2Edetected%2Earch%7D/4%2E1%2E39%2EFinal/io%2Enetty%3Anetty-transport-native-unix-common%3A%24%7Bos%2Edetected%2Ename%7D-%24%7Bos%2Edetected%2Earch%7D-4%2E1%2E39%2EFinal.jar

So now I have essentially a broken wiki. All the page macros are coming up on my pages as “unknown macro: X” where X is the name of the macro.

Should I uninstall and then reinstall the wiki to attempt resolve this problem? Find replacements for the 0 byte jar files?

Any advice or suggestions would be appreciated.

Have you deleted /var/lib/xwiki/data/solr as indicated on https://www.xwiki.org/xwiki/bin/view/ReleaseNotes/Data/XWiki/11.4/#HIssuesspecifictoXWiki11.4 ? If not then it’s most probably the root cause of everything.

1 Like

I missed that instruction in the 11.4 release notes. That’s resolved the solr errors, but not the 0 byte jar file problem.

I’ve gone to the main wiki and the distribution wizard is ready to run.

No idea why those are broken (maybe some weird network issue). You should probably delete the folder of each of those and reinstall them.

It looks like my initial problem was not deleting the solr directory.

The distribution wizard is now running correctly for the subwikis and the macros are no longer showing errors on the pages. Thank you for that.

Do you have any suggestions where I could find those jar files to download again? From a maven repository somewhere?

(I think I may have seen an error message during the upgrade about a non responsive repository somewhere, but I stupidly didn’t capture the information before I lost it from my terminal scroll buffer)

Hmm actually there is something very weird in the name of that extension (and the others), the :${os.detected.name}-${os.detected.arch} part. No idea how they ended up there. I would just delete the folder of those extensions and see if you have some warning at startup related to invalid extensions. None of those are dependencies of the standard flavor.

sounds like a plan.

I’ve removed those directories, and done a fresh stop/start of tomcat8

I see the following warning:

2019-08-26 20:13:18,503 [localhost-startStop-1] WARN  ltInstalledExtensionRepository - Invalid extension [io.netty:netty-example/4.1.39.Final] on namespace [wiki:xwiki] (InvalidExtensionException: No compatible extension is installed for dependency [io.netty:netty-tcnative:${os.detected.classifier}-2.0.25.Final])

Is there an easy way to find out which extension may be requesting this?

Next, I access the main instance of the wiki. During the initialization, I get the following errors:

2019-08-26 20:15:03,146 [XWiki Scheduler initialization] ERROR c.x.x.p.s.SchedulerPlugin      - Failed to restore job with in document [xwiki:Scheduler.WatchListHourlyNotifier] and wiki [xwiki]
com.xpn.xwiki.plugin.scheduler.SchedulerPluginException: Error number 90006 in 5: Exception in plugin [com.xpn.xwiki.plugin.scheduler.SchedulerPlugin]: Error while loading job class for job : WatchList hourly notifier
	at com.xpn.xwiki.plugin.scheduler.SchedulerPlugin.scheduleJob(SchedulerPlugin.java:426)
	at com.xpn.xwiki.plugin.scheduler.SchedulerPlugin.register(SchedulerPlugin.java:316)
	at com.xpn.xwiki.plugin.scheduler.SchedulerPlugin.register(SchedulerPlugin.java:309)
	at com.xpn.xwiki.plugin.scheduler.SchedulerPlugin.restoreExistingJobs(SchedulerPlugin.java:294)
	at com.xpn.xwiki.plugin.scheduler.SchedulerPlugin.initAsync(SchedulerPlugin.java:168)
	at com.xpn.xwiki.plugin.scheduler.SchedulerPlugin.access$000(SchedulerPlugin.java:80)
	at com.xpn.xwiki.plugin.scheduler.SchedulerPlugin$1.run(SchedulerPlugin.java:127)
	at org.xwiki.context.concurrent.ExecutionContextRunnable.run(ExecutionContextRunnable.java:70)
	at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.ClassNotFoundException: org.xwiki.watchlist.internal.job.WatchListJob
	at org.xwiki.classloader.URIClassLoader$1.run(URIClassLoader.java:193)
	at org.xwiki.classloader.URIClassLoader$1.run(URIClassLoader.java:180)
	at java.security.AccessController.doPrivileged(Native Method)
	at org.xwiki.classloader.URIClassLoader.findClass(URIClassLoader.java:179)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
	at org.xwiki.classloader.xwiki.internal.ContextNamespaceURLClassLoader.loadClass(ContextNamespaceURLClassLoader.java:176)
	at java.lang.Class.forName0(Native Method)
	at java.lang.Class.forName(Class.java:348)
	at com.xpn.xwiki.plugin.scheduler.SchedulerPlugin.scheduleJob(SchedulerPlugin.java:361)
	... 8 common frames omitted
2019-08-26 20:15:05,978 [XWiki Scheduler initialization] ERROR c.x.x.p.s.SchedulerPlugin      - Failed to restore job with in document [xwiki:Scheduler.WatchListWeeklyNotifier] and wiki [xwiki]
com.xpn.xwiki.plugin.scheduler.SchedulerPluginException: Error number 90006 in 5: Exception in plugin [com.xpn.xwiki.plugin.scheduler.SchedulerPlugin]: Error while loading job class for job : WatchList weekly notifier
	at com.xpn.xwiki.plugin.scheduler.SchedulerPlugin.scheduleJob(SchedulerPlugin.java:426)
	at com.xpn.xwiki.plugin.scheduler.SchedulerPlugin.register(SchedulerPlugin.java:316)
	at com.xpn.xwiki.plugin.scheduler.SchedulerPlugin.register(SchedulerPlugin.java:309)
	at com.xpn.xwiki.plugin.scheduler.SchedulerPlugin.restoreExistingJobs(SchedulerPlugin.java:294)
	at com.xpn.xwiki.plugin.scheduler.SchedulerPlugin.initAsync(SchedulerPlugin.java:168)
	at com.xpn.xwiki.plugin.scheduler.SchedulerPlugin.access$000(SchedulerPlugin.java:80)
	at com.xpn.xwiki.plugin.scheduler.SchedulerPlugin$1.run(SchedulerPlugin.java:127)
	at org.xwiki.context.concurrent.ExecutionContextRunnable.run(ExecutionContextRunnable.java:70)
	at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.ClassNotFoundException: org.xwiki.watchlist.internal.job.WatchListJob
	at org.xwiki.classloader.URIClassLoader$1.run(URIClassLoader.java:193)
	at org.xwiki.classloader.URIClassLoader$1.run(URIClassLoader.java:180)
	at java.security.AccessController.doPrivileged(Native Method)
	at org.xwiki.classloader.URIClassLoader.findClass(URIClassLoader.java:179)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
	at org.xwiki.classloader.xwiki.internal.ContextNamespaceURLClassLoader.loadClass(ContextNamespaceURLClassLoader.java:176)
	at java.lang.Class.forName0(Native Method)
	at java.lang.Class.forName(Class.java:348)
	at com.xpn.xwiki.plugin.scheduler.SchedulerPlugin.scheduleJob(SchedulerPlugin.java:361)
	... 8 common frames omitted
2019-08-26 20:15:06,023 [XWiki Scheduler initialization] ERROR c.x.x.p.s.SchedulerPlugin      - Failed to restore job with in document [xwiki:Scheduler.WatchListDailyNotifier] and wiki [xwiki]
com.xpn.xwiki.plugin.scheduler.SchedulerPluginException: Error number 90006 in 5: Exception in plugin [com.xpn.xwiki.plugin.scheduler.SchedulerPlugin]: Error while loading job class for job : WatchList daily notifier
	at com.xpn.xwiki.plugin.scheduler.SchedulerPlugin.scheduleJob(SchedulerPlugin.java:426)
	at com.xpn.xwiki.plugin.scheduler.SchedulerPlugin.register(SchedulerPlugin.java:316)
	at com.xpn.xwiki.plugin.scheduler.SchedulerPlugin.register(SchedulerPlugin.java:309)
	at com.xpn.xwiki.plugin.scheduler.SchedulerPlugin.restoreExistingJobs(SchedulerPlugin.java:294)
	at com.xpn.xwiki.plugin.scheduler.SchedulerPlugin.initAsync(SchedulerPlugin.java:168)
	at com.xpn.xwiki.plugin.scheduler.SchedulerPlugin.access$000(SchedulerPlugin.java:80)
	at com.xpn.xwiki.plugin.scheduler.SchedulerPlugin$1.run(SchedulerPlugin.java:127)
	at org.xwiki.context.concurrent.ExecutionContextRunnable.run(ExecutionContextRunnable.java:70)
	at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.ClassNotFoundException: org.xwiki.watchlist.internal.job.WatchListJob
	at org.xwiki.classloader.URIClassLoader$1.run(URIClassLoader.java:193)
	at org.xwiki.classloader.URIClassLoader$1.run(URIClassLoader.java:180)
	at java.security.AccessController.doPrivileged(Native Method)
	at org.xwiki.classloader.URIClassLoader.findClass(URIClassLoader.java:179)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
	at org.xwiki.classloader.xwiki.internal.ContextNamespaceURLClassLoader.loadClass(ContextNamespaceURLClassLoader.java:176)
	at java.lang.Class.forName0(Native Method)
	at java.lang.Class.forName(Class.java:348)
	at com.xpn.xwiki.plugin.scheduler.SchedulerPlugin.scheduleJob(SchedulerPlugin.java:361)
	... 8 common frames omitted

Key words in the above errors:

Error number 90006 in 5: Exception in plugin [com.xpn.xwiki.plugin.scheduler.SchedulerPlugin]: Error while loading job class for job : WatchList hourly notifier
Error number 90006 in 5: Exception in plugin [com.xpn.xwiki.plugin.scheduler.SchedulerPlugin]: Error while loading job class for job : WatchList weekly notifier
Error number 90006 in 5: Exception in plugin [com.xpn.xwiki.plugin.scheduler.SchedulerPlugin]: Error while loading job class for job : WatchList daily notifier

I don’t recall seeing this error before.

The requesting extension is io.netty:netty-example but no idea why you have io.netty:netty-example in the first place. But this is interesting, creating a jira issue about that weird dependency handling of io.netty:netty-example in Extension Manager. It’s possible netty-example came as a dependency of some extension that you don’t have anymore so you should probably just uninstall it in EM UI.

This suggest you installed the watchlist extension which restored the watchlist schedulers but it should actually be uninstalled (and the watchlist schedulers be deleted).

Weird. The netty packages were installed as a dependancy of “Apache Hadoop HDFS”. How in heck that extension got installed, I have no idea. Removing them now.

I did have watchlist installed before, but I removed it an upgrade or two ago. I’ll see if I can find any residual stuff lying around in the extensions.

root@wiki:/var/lib/xwiki/data# locate *.jar |grep xwiki|grep watch
/usr/lib/xwiki/WEB-INF/lib/xwiki-platform-notifications-filters-watch-11.3.jar
/usr/lib/xwiki/WEB-INF/lib/xwiki-platform-notifications-filters-watchlistbridge-11.3.jar
/var/lib/xwiki/data/extension/repository/org%2Exwiki%2Eplatform%3Axwiki-platform-watchlist-api/11%2E2/org%2Exwiki%2Eplatform%3Axwiki-platform-watchlist-api-11%2E2.jar
/var/lib/xwiki/data/extension/repository/org%2Exwiki%2Eplatform%3Axwiki-platform-watchlist-api/11%2E3/org%2Exwiki%2Eplatform%3Axwiki-platform-watchlist-api-11%2E3.jar

And there we go. The extension seems to be installed, but searching for “watchlist” in the extension manager for installed extensions of all the wikis and subwikis turns up nothing. I think I’ll just remove those two watchlist jar files.

What a trip down the rabbit hole today!

edit: and any other watchlist related files.

How do I delete the schedulers? I’m not sure where to find them.

edit: wait, found it, under Applications, Scheduler

Ok, last problem I hope.

My wiki is behind an nginx ssl proxy. Prior to today’s upgrade, it was working perfectly.

After the upgrade, some of the external links are being written incorrectly, but only for the panels

For example, subwiki 1:
02
you can see that the url for the “more applications” link has a https://server:80/ in the url. However the links on the page are written correctly without the :80 added.

subwiki2:
02%202
not only are the links incorrect, but the image properties for the thumbnails are written as https://server:80/

main wiki: same as subwiki 2, although the links within the pages of the wiki’s themselves appear fine.

Another example from subwiki 2: The dashboard link under application on the right is correct, but the left has the https://server:80/ written for the image and for the link.

Weird!

Actually the issue is not even the wrong port, it should be a local URL in the HTML (it should start at /xwiki which would make the browser complete it with the host you have in the address bar).

I see this is a subwiki, do you have the same issue on the main wiki with this panel ?

Could you go to Panels.Applications page and see if the current version is the standard one (compare version first line in the history and second line).

Yes, the main wiki has the same issue as the subwikis

The history panels for the subwikis show that the panels app is the current version
58

I’ve updated to 11.7. I still have the same issue with the links in the panels pointing to the wrong url.

Any idea where I can look to debug this?