XWiki fails with 500 – HibernateException : The ClassLoaderService can not be reused. This instance was stopped already

snawaz · April 18, 2020, 12:53pm

Background

I’ve written my own authenticator extension and I build my docker image deriving FROM xwiki:11.10.4-postgres-tomcat where I also COPY my jar and edit xwiki.cfg pointing authclass to my class. This is perfect so far, as it worked for me, during my experiments.

However, once I deployed it to GCP Kubernetes, it failed to work as it attempted to connect to old deleted database which I had used during my experiments. I found /usr/local/xwiki/data/hibernate.cfg.xml keeps that old database connections info. I realize /usr/local/xwiki/ is still the old content being used. So I deleted this directory. Restarted my pod. My wiki came up.

I was happy!

But my happiness was short lived, as I realized my authenticator isn’t great, it has some issues. So I added few logs and few more things. I deployed it again.

Now it fails to come up. The wiki page loads up to 50% or so, then fails with exception, as shown below.

Stacktrace (UI)

HTTP Status 500 – Internal Server Error
Type Exception Report

Message com.xpn.xwiki.XWikiException: Error number 11007 in 0: Failed to extract Entity Resource Reference from URL [http://wiki.my-domain.com/bin/view/Main/]

Description The server encountered an unexpected condition that prevented it from fulfilling the request.

Exception

javax.servlet.ServletException: com.xpn.xwiki.XWikiException: Error number 11007 in 0: Failed to extract Entity Resource Reference from URL [http://wiki.my-domain.com/bin/view/Main/]
	org.apache.struts.action.RequestProcessor.processException(RequestProcessor.java:520)
	org.apache.struts.action.RequestProcessor.processActionPerform(RequestProcessor.java:427)
	org.apache.struts.action.RequestProcessor.process(RequestProcessor.java:228)
	org.apache.struts.action.ActionServlet.process(ActionServlet.java:1913)
	org.apache.struts.action.ActionServlet.doGet(ActionServlet.java:449)
	javax.servlet.http.HttpServlet.service(HttpServlet.java:634)
	javax.servlet.http.HttpServlet.service(HttpServlet.java:741)
	com.xpn.xwiki.web.ActionFilter.doFilter(ActionFilter.java:112)
	org.xwiki.wysiwyg.filter.ConversionFilter.doFilter(ConversionFilter.java:109)
	org.xwiki.container.servlet.filters.internal.SetHTTPHeaderFilter.doFilter(SetHTTPHeaderFilter.java:63)
	org.xwiki.container.servlet.filters.internal.SavedRequestRestorerFilter.doFilter(SavedRequestRestorerFilter.java:208)
	org.xwiki.container.servlet.filters.internal.SetCharacterEncodingFilter.doFilter(SetCharacterEncodingFilter.java:111)
	org.xwiki.resource.servlet.RoutingFilter.doFilter(RoutingFilter.java:132)
Root Cause

com.xpn.xwiki.XWikiException: Error number 11007 in 0: Failed to extract Entity Resource Reference from URL [http://wiki.my-domain.com/bin/view/Main/]
	com.xpn.xwiki.XWiki.initializeResourceFromURL(XWiki.java:1031)
	com.xpn.xwiki.XWiki.getXWiki(XWiki.java:891)
	com.xpn.xwiki.web.XWikiAction.execute(XWikiAction.java:285)
	com.xpn.xwiki.web.XWikiAction.execute(XWikiAction.java:218)
	org.apache.struts.action.RequestProcessor.processActionPerform(RequestProcessor.java:425)
	org.apache.struts.action.RequestProcessor.process(RequestProcessor.java:228)
	org.apache.struts.action.ActionServlet.process(ActionServlet.java:1913)
	org.apache.struts.action.ActionServlet.doGet(ActionServlet.java:449)
	javax.servlet.http.HttpServlet.service(HttpServlet.java:634)
	javax.servlet.http.HttpServlet.service(HttpServlet.java:741)
	com.xpn.xwiki.web.ActionFilter.doFilter(ActionFilter.java:112)
	org.xwiki.wysiwyg.filter.ConversionFilter.doFilter(ConversionFilter.java:109)
	org.xwiki.container.servlet.filters.internal.SetHTTPHeaderFilter.doFilter(SetHTTPHeaderFilter.java:63)
	org.xwiki.container.servlet.filters.internal.SavedRequestRestorerFilter.doFilter(SavedRequestRestorerFilter.java:208)
	org.xwiki.container.servlet.filters.internal.SetCharacterEncodingFilter.doFilter(SetCharacterEncodingFilter.java:111)
	org.xwiki.resource.servlet.RoutingFilter.doFilter(RoutingFilter.java:132)
Root Cause

java.lang.RuntimeException: Failed to locate wiki descriptor for alias [wiki.my-domain.com]
	org.xwiki.url.internal.standard.AbstractWikiReferenceExtractor.getWikiDescriptorByAlias(AbstractWikiReferenceExtractor.java:80)
	org.xwiki.url.internal.standard.DomainWikiReferenceExtractor.resolveDomainBasedWikiReference(DomainWikiReferenceExtractor.java:61)
	org.xwiki.url.internal.standard.DomainWikiReferenceExtractor.extract(DomainWikiReferenceExtractor.java:47)
	org.xwiki.url.internal.standard.entity.BinEntityResourceReferenceResolver.extractWikiReference(BinEntityResourceReferenceResolver.java:51)
	org.xwiki.url.internal.standard.entity.AbstractEntityResourceReferenceResolver.resolve(AbstractEntityResourceReferenceResolver.java:87)
	org.xwiki.url.internal.standard.entity.AbstractEntityResourceReferenceResolver.resolve(AbstractEntityResourceReferenceResolver.java:50)
	org.xwiki.url.internal.GenericResourceReferenceResolver.resolve(GenericResourceReferenceResolver.java:61)
	org.xwiki.url.internal.GenericResourceReferenceResolver.resolve(GenericResourceReferenceResolver.java:47)
	org.xwiki.url.internal.standard.StandardExtendedURLResourceReferenceResolver.resolve(StandardExtendedURLResourceReferenceResolver.java:65)
	org.xwiki.url.internal.standard.StandardExtendedURLResourceReferenceResolver.resolve(StandardExtendedURLResourceReferenceResolver.java:43)
	org.xwiki.url.internal.DefaultResourceReferenceResolver.resolve(DefaultResourceReferenceResolver.java:93)
	org.xwiki.url.internal.DefaultResourceReferenceResolver.resolve(DefaultResourceReferenceResolver.java:48)
	com.xpn.xwiki.XWiki.initializeResourceFromURL(XWiki.java:1026)
	com.xpn.xwiki.XWiki.getXWiki(XWiki.java:891)
	com.xpn.xwiki.web.XWikiAction.execute(XWikiAction.java:285)
	com.xpn.xwiki.web.XWikiAction.execute(XWikiAction.java:218)
	org.apache.struts.action.RequestProcessor.processActionPerform(RequestProcessor.java:425)
	org.apache.struts.action.RequestProcessor.process(RequestProcessor.java:228)
	org.apache.struts.action.ActionServlet.process(ActionServlet.java:1913)
	org.apache.struts.action.ActionServlet.doGet(ActionServlet.java:449)
	javax.servlet.http.HttpServlet.service(HttpServlet.java:634)
	javax.servlet.http.HttpServlet.service(HttpServlet.java:741)
	com.xpn.xwiki.web.ActionFilter.doFilter(ActionFilter.java:112)
	org.xwiki.wysiwyg.filter.ConversionFilter.doFilter(ConversionFilter.java:109)
	org.xwiki.container.servlet.filters.internal.SetHTTPHeaderFilter.doFilter(SetHTTPHeaderFilter.java:63)
	org.xwiki.container.servlet.filters.internal.SavedRequestRestorerFilter.doFilter(SavedRequestRestorerFilter.java:208)
	org.xwiki.container.servlet.filters.internal.SetCharacterEncodingFilter.doFilter(SetCharacterEncodingFilter.java:111)
	org.xwiki.resource.servlet.RoutingFilter.doFilter(RoutingFilter.java:132)
Root Cause

org.xwiki.wiki.manager.WikiManagerException: Failed to locate XWiki.XWikiServerClass document for wiki alias [wiki.my-domain.com]
	org.xwiki.wiki.internal.descriptor.document.DefaultWikiDescriptorDocumentHelper.findXWikiServerClassDocumentReference(DefaultWikiDescriptorDocumentHelper.java:128)
	org.xwiki.wiki.internal.descriptor.document.DefaultWikiDescriptorDocumentHelper.findXWikiServerClassDocument(DefaultWikiDescriptorDocumentHelper.java:140)
	org.xwiki.wiki.internal.descriptor.DefaultWikiDescriptorManager.getByAlias(DefaultWikiDescriptorManager.java:147)
	org.xwiki.url.internal.standard.AbstractWikiReferenceExtractor.getWikiDescriptorByAlias(AbstractWikiReferenceExtractor.java:78)
	org.xwiki.url.internal.standard.DomainWikiReferenceExtractor.resolveDomainBasedWikiReference(DomainWikiReferenceExtractor.java:61)
	org.xwiki.url.internal.standard.DomainWikiReferenceExtractor.extract(DomainWikiReferenceExtractor.java:47)
	org.xwiki.url.internal.standard.entity.BinEntityResourceReferenceResolver.extractWikiReference(BinEntityResourceReferenceResolver.java:51)
	org.xwiki.url.internal.standard.entity.AbstractEntityResourceReferenceResolver.resolve(AbstractEntityResourceReferenceResolver.java:87)
	org.xwiki.url.internal.standard.entity.AbstractEntityResourceReferenceResolver.resolve(AbstractEntityResourceReferenceResolver.java:50)
	org.xwiki.url.internal.GenericResourceReferenceResolver.resolve(GenericResourceReferenceResolver.java:61)
	org.xwiki.url.internal.GenericResourceReferenceResolver.resolve(GenericResourceReferenceResolver.java:47)
	org.xwiki.url.internal.standard.StandardExtendedURLResourceReferenceResolver.resolve(StandardExtendedURLResourceReferenceResolver.java:65)
	org.xwiki.url.internal.standard.StandardExtendedURLResourceReferenceResolver.resolve(StandardExtendedURLResourceReferenceResolver.java:43)
	org.xwiki.url.internal.DefaultResourceReferenceResolver.resolve(DefaultResourceReferenceResolver.java:93)
	org.xwiki.url.internal.DefaultResourceReferenceResolver.resolve(DefaultResourceReferenceResolver.java:48)
	com.xpn.xwiki.XWiki.initializeResourceFromURL(XWiki.java:1026)
	com.xpn.xwiki.XWiki.getXWiki(XWiki.java:891)
	com.xpn.xwiki.web.XWikiAction.execute(XWikiAction.java:285)
	com.xpn.xwiki.web.XWikiAction.execute(XWikiAction.java:218)
	org.apache.struts.action.RequestProcessor.processActionPerform(RequestProcessor.java:425)
	org.apache.struts.action.RequestProcessor.process(RequestProcessor.java:228)
	org.apache.struts.action.ActionServlet.process(ActionServlet.java:1913)
	org.apache.struts.action.ActionServlet.doGet(ActionServlet.java:449)
	javax.servlet.http.HttpServlet.service(HttpServlet.java:634)
	javax.servlet.http.HttpServlet.service(HttpServlet.java:741)
	com.xpn.xwiki.web.ActionFilter.doFilter(ActionFilter.java:112)
	org.xwiki.wysiwyg.filter.ConversionFilter.doFilter(ConversionFilter.java:109)
	org.xwiki.container.servlet.filters.internal.SetHTTPHeaderFilter.doFilter(SetHTTPHeaderFilter.java:63)
	org.xwiki.container.servlet.filters.internal.SavedRequestRestorerFilter.doFilter(SavedRequestRestorerFilter.java:208)
	org.xwiki.container.servlet.filters.internal.SetCharacterEncodingFilter.doFilter(SetCharacterEncodingFilter.java:111)
	org.xwiki.resource.servlet.RoutingFilter.doFilter(RoutingFilter.java:132)
Root Cause

org.xwiki.query.QueryException: Exception while translating [where doc.object(XWiki.XWikiServerClass).server = :wikiAlias and doc.name like 'XWikiServer%'] XWQL query to the [hql] language. Query statement = [where doc.object(XWiki.XWikiServerClass).server = :wikiAlias and doc.name like 'XWikiServer%']
	org.xwiki.query.xwql.internal.XWQLQueryExecutor.execute(XWQLQueryExecutor.java:129)
	org.xwiki.query.internal.DefaultQueryExecutorManager.execute(DefaultQueryExecutorManager.java:72)
	org.xwiki.query.internal.DefaultQuery.execute(DefaultQuery.java:306)
	org.xwiki.wiki.internal.descriptor.document.DefaultWikiDescriptorDocumentHelper.findXWikiServerClassDocumentReference(DefaultWikiDescriptorDocumentHelper.java:121)
	org.xwiki.wiki.internal.descriptor.document.DefaultWikiDescriptorDocumentHelper.findXWikiServerClassDocument(DefaultWikiDescriptorDocumentHelper.java:140)
	org.xwiki.wiki.internal.descriptor.DefaultWikiDescriptorManager.getByAlias(DefaultWikiDescriptorManager.java:147)
	org.xwiki.url.internal.standard.AbstractWikiReferenceExtractor.getWikiDescriptorByAlias(AbstractWikiReferenceExtractor.java:78)
	org.xwiki.url.internal.standard.DomainWikiReferenceExtractor.resolveDomainBasedWikiReference(DomainWikiReferenceExtractor.java:61)
	org.xwiki.url.internal.standard.DomainWikiReferenceExtractor.extract(DomainWikiReferenceExtractor.java:47)
	org.xwiki.url.internal.standard.entity.BinEntityResourceReferenceResolver.extractWikiReference(BinEntityResourceReferenceResolver.java:51)
	org.xwiki.url.internal.standard.entity.AbstractEntityResourceReferenceResolver.resolve(AbstractEntityResourceReferenceResolver.java:87)
	org.xwiki.url.internal.standard.entity.AbstractEntityResourceReferenceResolver.resolve(AbstractEntityResourceReferenceResolver.java:50)
	org.xwiki.url.internal.GenericResourceReferenceResolver.resolve(GenericResourceReferenceResolver.java:61)
	org.xwiki.url.internal.GenericResourceReferenceResolver.resolve(GenericResourceReferenceResolver.java:47)
	org.xwiki.url.internal.standard.StandardExtendedURLResourceReferenceResolver.resolve(StandardExtendedURLResourceReferenceResolver.java:65)
	org.xwiki.url.internal.standard.StandardExtendedURLResourceReferenceResolver.resolve(StandardExtendedURLResourceReferenceResolver.java:43)
	org.xwiki.url.internal.DefaultResourceReferenceResolver.resolve(DefaultResourceReferenceResolver.java:93)
	org.xwiki.url.internal.DefaultResourceReferenceResolver.resolve(DefaultResourceReferenceResolver.java:48)
	com.xpn.xwiki.XWiki.initializeResourceFromURL(XWiki.java:1026)
	com.xpn.xwiki.XWiki.getXWiki(XWiki.java:891)
	com.xpn.xwiki.web.XWikiAction.execute(XWikiAction.java:285)
	com.xpn.xwiki.web.XWikiAction.execute(XWikiAction.java:218)
	org.apache.struts.action.RequestProcessor.processActionPerform(RequestProcessor.java:425)
	org.apache.struts.action.RequestProcessor.process(RequestProcessor.java:228)
	org.apache.struts.action.ActionServlet.process(ActionServlet.java:1913)
	org.apache.struts.action.ActionServlet.doGet(ActionServlet.java:449)
	javax.servlet.http.HttpServlet.service(HttpServlet.java:634)
	javax.servlet.http.HttpServlet.service(HttpServlet.java:741)
	com.xpn.xwiki.web.ActionFilter.doFilter(ActionFilter.java:112)
	org.xwiki.wysiwyg.filter.ConversionFilter.doFilter(ConversionFilter.java:109)
	org.xwiki.container.servlet.filters.internal.SetHTTPHeaderFilter.doFilter(SetHTTPHeaderFilter.java:63)
	org.xwiki.container.servlet.filters.internal.SavedRequestRestorerFilter.doFilter(SavedRequestRestorerFilter.java:208)
	org.xwiki.container.servlet.filters.internal.SetCharacterEncodingFilter.doFilter(SetCharacterEncodingFilter.java:111)
	org.xwiki.resource.servlet.RoutingFilter.doFilter(RoutingFilter.java:132)
Root Cause

com.xpn.xwiki.XWikiException: Error number 3202 in 3: Exception while reading document [xwiki:XWiki.XWikiServerClass]
	com.xpn.xwiki.store.XWikiHibernateStore.loadXWikiDoc(XWikiHibernateStore.java:1000)
	com.xpn.xwiki.store.XWikiCacheStore.loadXWikiDoc(XWikiCacheStore.java:363)
	com.xpn.xwiki.XWiki.getDocument(XWiki.java:2116)
	com.xpn.xwiki.XWiki.getDocument(XWiki.java:2178)
	com.xpn.xwiki.XWiki.getXClass(XWiki.java:6746)
	com.xpn.xwiki.XWiki.getClass(XWiki.java:6759)
	com.xpn.xwiki.doc.DefaultDocumentAccessBridge.isPropertyCustomMapped(DefaultDocumentAccessBridge.java:552)
	org.xwiki.query.xwql.internal.QueryContext$PropertyInfo.isCustomMapped(QueryContext.java:153)
	org.xwiki.query.xwql.internal.QueryContext$ObjectInfo.isCustomMapped(QueryContext.java:122)
	org.xwiki.query.xwql.internal.hql.ObjectPrinter.print(ObjectPrinter.java:37)
	org.xwiki.query.xwql.internal.hql.Printer.print(Printer.java:70)
	org.xwiki.query.xwql.internal.hql.XWQLtoHQLTranslator.translate(XWQLtoHQLTranslator.java:61)
	org.xwiki.query.xwql.internal.XWQLQueryExecutor.execute(XWQLQueryExecutor.java:97)
	org.xwiki.query.internal.DefaultQueryExecutorManager.execute(DefaultQueryExecutorManager.java:72)
	org.xwiki.query.internal.DefaultQuery.execute(DefaultQuery.java:306)
	org.xwiki.wiki.internal.descriptor.document.DefaultWikiDescriptorDocumentHelper.findXWikiServerClassDocumentReference(DefaultWikiDescriptorDocumentHelper.java:121)
	org.xwiki.wiki.internal.descriptor.document.DefaultWikiDescriptorDocumentHelper.findXWikiServerClassDocument(DefaultWikiDescriptorDocumentHelper.java:140)
	org.xwiki.wiki.internal.descriptor.DefaultWikiDescriptorManager.getByAlias(DefaultWikiDescriptorManager.java:147)
	org.xwiki.url.internal.standard.AbstractWikiReferenceExtractor.getWikiDescriptorByAlias(AbstractWikiReferenceExtractor.java:78)
	org.xwiki.url.internal.standard.DomainWikiReferenceExtractor.resolveDomainBasedWikiReference(DomainWikiReferenceExtractor.java:61)
	org.xwiki.url.internal.standard.DomainWikiReferenceExtractor.extract(DomainWikiReferenceExtractor.java:47)
	org.xwiki.url.internal.standard.entity.BinEntityResourceReferenceResolver.extractWikiReference(BinEntityResourceReferenceResolver.java:51)
	org.xwiki.url.internal.standard.entity.AbstractEntityResourceReferenceResolver.resolve(AbstractEntityResourceReferenceResolver.java:87)
	org.xwiki.url.internal.standard.entity.AbstractEntityResourceReferenceResolver.resolve(AbstractEntityResourceReferenceResolver.java:50)
	org.xwiki.url.internal.GenericResourceReferenceResolver.resolve(GenericResourceReferenceResolver.java:61)
	org.xwiki.url.internal.GenericResourceReferenceResolver.resolve(GenericResourceReferenceResolver.java:47)
	org.xwiki.url.internal.standard.StandardExtendedURLResourceReferenceResolver.resolve(StandardExtendedURLResourceReferenceResolver.java:65)
	org.xwiki.url.internal.standard.StandardExtendedURLResourceReferenceResolver.resolve(StandardExtendedURLResourceReferenceResolver.java:43)
	org.xwiki.url.internal.DefaultResourceReferenceResolver.resolve(DefaultResourceReferenceResolver.java:93)
	org.xwiki.url.internal.DefaultResourceReferenceResolver.resolve(DefaultResourceReferenceResolver.java:48)
	com.xpn.xwiki.XWiki.initializeResourceFromURL(XWiki.java:1026)
	com.xpn.xwiki.XWiki.getXWiki(XWiki.java:891)
	com.xpn.xwiki.web.XWikiAction.execute(XWikiAction.java:285)
	com.xpn.xwiki.web.XWikiAction.execute(XWikiAction.java:218)
	org.apache.struts.action.RequestProcessor.processActionPerform(RequestProcessor.java:425)
	org.apache.struts.action.RequestProcessor.process(RequestProcessor.java:228)
	org.apache.struts.action.ActionServlet.process(ActionServlet.java:1913)
	org.apache.struts.action.ActionServlet.doGet(ActionServlet.java:449)
	javax.servlet.http.HttpServlet.service(HttpServlet.java:634)
	javax.servlet.http.HttpServlet.service(HttpServlet.java:741)
	com.xpn.xwiki.web.ActionFilter.doFilter(ActionFilter.java:112)
	org.xwiki.wysiwyg.filter.ConversionFilter.doFilter(ConversionFilter.java:109)
	org.xwiki.container.servlet.filters.internal.SetHTTPHeaderFilter.doFilter(SetHTTPHeaderFilter.java:63)
	org.xwiki.container.servlet.filters.internal.SavedRequestRestorerFilter.doFilter(SavedRequestRestorerFilter.java:208)
	org.xwiki.container.servlet.filters.internal.SetCharacterEncodingFilter.doFilter(SetCharacterEncodingFilter.java:111)
	org.xwiki.resource.servlet.RoutingFilter.doFilter(RoutingFilter.java:132)
Root Cause

java.lang.IllegalStateException: EntityManagerFactory is closed
	org.hibernate.internal.SessionFactoryImpl.validateNotClosed(SessionFactoryImpl.java:548)
	org.hibernate.internal.SessionFactoryImpl.getCache(SessionFactoryImpl.java:848)
	org.hibernate.internal.AbstractSharedSessionContract.<init>(AbstractSharedSessionContract.java:153)
	org.hibernate.internal.AbstractSessionImpl.<init>(AbstractSessionImpl.java:29)
	org.hibernate.internal.SessionImpl.<init>(SessionImpl.java:223)
	org.hibernate.internal.SessionFactoryImpl$SessionBuilderImpl.openSession(SessionFactoryImpl.java:1305)
	org.hibernate.internal.SessionFactoryImpl.openSession(SessionFactoryImpl.java:494)
	com.xpn.xwiki.internal.store.hibernate.HibernateStore.beginTransaction(HibernateStore.java:777)
	com.xpn.xwiki.store.XWikiHibernateBaseStore.beginTransaction(XWikiHibernateBaseStore.java:577)
	com.xpn.xwiki.store.XWikiHibernateStore.loadXWikiDoc(XWikiHibernateStore.java:870)
	com.xpn.xwiki.store.XWikiCacheStore.loadXWikiDoc(XWikiCacheStore.java:363)
	com.xpn.xwiki.XWiki.getDocument(XWiki.java:2116)
	com.xpn.xwiki.XWiki.getDocument(XWiki.java:2178)
	com.xpn.xwiki.XWiki.getXClass(XWiki.java:6746)
	com.xpn.xwiki.XWiki.getClass(XWiki.java:6759)
	com.xpn.xwiki.doc.DefaultDocumentAccessBridge.isPropertyCustomMapped(DefaultDocumentAccessBridge.java:552)
	org.xwiki.query.xwql.internal.QueryContext$PropertyInfo.isCustomMapped(QueryContext.java:153)
	org.xwiki.query.xwql.internal.QueryContext$ObjectInfo.isCustomMapped(QueryContext.java:122)
	org.xwiki.query.xwql.internal.hql.ObjectPrinter.print(ObjectPrinter.java:37)
	org.xwiki.query.xwql.internal.hql.Printer.print(Printer.java:70)
	org.xwiki.query.xwql.internal.hql.XWQLtoHQLTranslator.translate(XWQLtoHQLTranslator.java:61)
	org.xwiki.query.xwql.internal.XWQLQueryExecutor.execute(XWQLQueryExecutor.java:97)
	org.xwiki.query.internal.DefaultQueryExecutorManager.execute(DefaultQueryExecutorManager.java:72)
	org.xwiki.query.internal.DefaultQuery.execute(DefaultQuery.java:306)
	org.xwiki.wiki.internal.descriptor.document.DefaultWikiDescriptorDocumentHelper.findXWikiServerClassDocumentReference(DefaultWikiDescriptorDocumentHelper.java:121)
	org.xwiki.wiki.internal.descriptor.document.DefaultWikiDescriptorDocumentHelper.findXWikiServerClassDocument(DefaultWikiDescriptorDocumentHelper.java:140)
	org.xwiki.wiki.internal.descriptor.DefaultWikiDescriptorManager.getByAlias(DefaultWikiDescriptorManager.java:147)
	org.xwiki.url.internal.standard.AbstractWikiReferenceExtractor.getWikiDescriptorByAlias(AbstractWikiReferenceExtractor.java:78)
	org.xwiki.url.internal.standard.DomainWikiReferenceExtractor.resolveDomainBasedWikiReference(DomainWikiReferenceExtractor.java:61)
	org.xwiki.url.internal.standard.DomainWikiReferenceExtractor.extract(DomainWikiReferenceExtractor.java:47)
	org.xwiki.url.internal.standard.entity.BinEntityResourceReferenceResolver.extractWikiReference(BinEntityResourceReferenceResolver.java:51)
	org.xwiki.url.internal.standard.entity.AbstractEntityResourceReferenceResolver.resolve(AbstractEntityResourceReferenceResolver.java:87)
	org.xwiki.url.internal.standard.entity.AbstractEntityResourceReferenceResolver.resolve(AbstractEntityResourceReferenceResolver.java:50)
	org.xwiki.url.internal.GenericResourceReferenceResolver.resolve(GenericResourceReferenceResolver.java:61)
	org.xwiki.url.internal.GenericResourceReferenceResolver.resolve(GenericResourceReferenceResolver.java:47)
	org.xwiki.url.internal.standard.StandardExtendedURLResourceReferenceResolver.resolve(StandardExtendedURLResourceReferenceResolver.java:65)
	org.xwiki.url.internal.standard.StandardExtendedURLResourceReferenceResolver.resolve(StandardExtendedURLResourceReferenceResolver.java:43)
	org.xwiki.url.internal.DefaultResourceReferenceResolver.resolve(DefaultResourceReferenceResolver.java:93)
	org.xwiki.url.internal.DefaultResourceReferenceResolver.resolve(DefaultResourceReferenceResolver.java:48)
	com.xpn.xwiki.XWiki.initializeResourceFromURL(XWiki.java:1026)
	com.xpn.xwiki.XWiki.getXWiki(XWiki.java:891)
	com.xpn.xwiki.web.XWikiAction.execute(XWikiAction.java:285)
	com.xpn.xwiki.web.XWikiAction.execute(XWikiAction.java:218)
	org.apache.struts.action.RequestProcessor.processActionPerform(RequestProcessor.java:425)
	org.apache.struts.action.RequestProcessor.process(RequestProcessor.java:228)
	org.apache.struts.action.ActionServlet.process(ActionServlet.java:1913)
	org.apache.struts.action.ActionServlet.doGet(ActionServlet.java:449)
	javax.servlet.http.HttpServlet.service(HttpServlet.java:634)
	javax.servlet.http.HttpServlet.service(HttpServlet.java:741)
	com.xpn.xwiki.web.ActionFilter.doFilter(ActionFilter.java:112)
	org.xwiki.wysiwyg.filter.ConversionFilter.doFilter(ConversionFilter.java:109)
	org.xwiki.container.servlet.filters.internal.SetHTTPHeaderFilter.doFilter(SetHTTPHeaderFilter.java:63)
	org.xwiki.container.servlet.filters.internal.SavedRequestRestorerFilter.doFilter(SavedRequestRestorerFilter.java:208)
	org.xwiki.container.servlet.filters.internal.SetCharacterEncodingFilter.doFilter(SetCharacterEncodingFilter.java:111)
	org.xwiki.resource.servlet.RoutingFilter.doFilter(RoutingFilter.java:132)

Note The full stack trace of the root cause is available in the server logs.
Apache Tomcat/8.5.54

Stacktrace (logs)

Here is this stack trace which is not on the UI, and it is the first one in the log. It is being repeated many times. I also noticed it is unable to load many kinds of documents: Main.WebPreferences, XWiki.XWikiPreferences and many classes.

2020-04-18 12:19:48,761 ERROR PreferencesConfigurationSource - Failed to access configuration property 
com.xpn.xwiki.XWikiException: Error number 3202 in 3: Exception while reading document [xwiki:Main.WebPreferences]
	at com.xpn.xwiki.store.XWikiHibernateStore.loadXWikiDoc(XWikiHibernateStore.java:1000)
	at com.xpn.xwiki.store.XWikiCacheStore.loadXWikiDoc(XWikiCacheStore.java:363)
	at com.xpn.xwiki.XWiki.getDocument(XWiki.java:2116)
	at com.xpn.xwiki.XWiki.getDocument(XWiki.java:2178)
	at org.xwiki.configuration.internal.AbstractXWikiPreferencesConfigurationSource.getBaseObject(AbstractXWikiPreferencesConfigurationSource.java:86)
	at org.xwiki.configuration.internal.AbstractXWikiPreferencesConfigurationSource.getBaseProperty(AbstractXWikiPreferencesConfigurationSource.java:139)
	at org.xwiki.configuration.internal.AbstractXWikiPreferencesConfigurationSource.getBaseProperty(AbstractXWikiPreferencesConfigurationSource.java:156)
	at org.xwiki.configuration.internal.AbstractDocumentConfigurationSource.getPropertyValue(AbstractDocumentConfigurationSource.java:326)
	at org.xwiki.configuration.internal.AbstractDocumentConfigurationSource.containsKey(AbstractDocumentConfigurationSource.java:199)
	at org.xwiki.configuration.internal.AbstractSpacesConfigurationSource$SpaceConfigurationSource.containsKey(AbstractSpacesConfigurationSource.java:172)
	at org.xwiki.configuration.internal.AbstractCompositeConfigurationSource.containsKey(AbstractCompositeConfigurationSource.java:45)
	at org.xwiki.configuration.internal.AbstractCompositeConfigurationSource.containsKey(AbstractCompositeConfigurationSource.java:45)
	at org.xwiki.configuration.internal.AbstractCompositeConfigurationSource.getProperty(AbstractCompositeConfigurationSource.java:95)
	at org.xwiki.url.internal.DefaultURLConfiguration.getURLFormatId(DefaultURLConfiguration.java:55)
	at org.xwiki.url.internal.URLExecutionContextInitializer.initialize(URLExecutionContextInitializer.java:55)
	at org.xwiki.context.internal.DefaultExecutionContextManager.runInitializers(DefaultExecutionContextManager.java:131)
	at org.xwiki.context.internal.DefaultExecutionContextManager.initialize(DefaultExecutionContextManager.java:103)
	at org.xwiki.container.servlet.internal.DefaultServletContainerInitializer.initializeRequest(DefaultServletContainerInitializer.java:116)
	at org.xwiki.container.servlet.internal.DefaultServletContainerInitializer.initializeRequest(DefaultServletContainerInitializer.java:125)
	at org.xwiki.resource.servlet.ResourceReferenceHandlerServlet.initializeContainerComponent(ResourceReferenceHandlerServlet.java:139)
	at org.xwiki.resource.servlet.ResourceReferenceHandlerServlet.service(ResourceReferenceHandlerServlet.java:85)
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:741)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:231)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
	at org.xwiki.container.servlet.filters.internal.SetHTTPHeaderFilter.doFilter(SetHTTPHeaderFilter.java:63)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
	at org.apache.catalina.core.ApplicationDispatcher.invoke(ApplicationDispatcher.java:728)
	at org.apache.catalina.core.ApplicationDispatcher.processRequest(ApplicationDispatcher.java:470)
	at org.apache.catalina.core.ApplicationDispatcher.doForward(ApplicationDispatcher.java:356)
	at org.apache.catalina.core.ApplicationDispatcher.forward(ApplicationDispatcher.java:316)
	at org.xwiki.resource.servlet.RoutingFilter.doFilter(RoutingFilter.java:145)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
	at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:199)
	at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:96)
	at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:543)
	at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:139)
	at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:81)
	at org.apache.catalina.valves.AbstractAccessLogValve.invoke(AbstractAccessLogValve.java:690)
	at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:87)
	at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:343)
	at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:615)
	at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:65)
	at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:818)
	at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1627)
	at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
	at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
	at java.base/java.lang.Thread.run(Thread.java:834)
Caused by: org.hibernate.HibernateException: HHH000469: The ClassLoaderService can not be reused. This instance was stopped already.
	at org.hibernate.boot.registry.classloading.internal.ClassLoaderServiceImpl.getAggregatedClassLoader(ClassLoaderServiceImpl.java:272)
	at org.hibernate.boot.registry.classloading.internal.ClassLoaderServiceImpl.loadJavaServices(ClassLoaderServiceImpl.java:248)
	at org.hibernate.boot.registry.StandardServiceRegistryBuilder.applyServiceContributors(StandardServiceRegistryBuilder.java:357)
	at org.hibernate.boot.registry.StandardServiceRegistryBuilder.build(StandardServiceRegistryBuilder.java:328)
	at com.xpn.xwiki.internal.store.hibernate.HibernateStore.build(HibernateStore.java:287)
	at com.xpn.xwiki.internal.store.hibernate.HibernateStore.initHibernate(HibernateStore.java:231)
	at com.xpn.xwiki.store.XWikiHibernateBaseStore.initHibernate(XWikiHibernateBaseStore.java:193)
	at com.xpn.xwiki.store.XWikiHibernateBaseStore.checkHibernate(XWikiHibernateBaseStore.java:482)
	at com.xpn.xwiki.store.XWikiHibernateStore.loadXWikiDoc(XWikiHibernateStore.java:867)
	... 50 common frames omitted

From all these, I find this line important:

Caused by: org.hibernate.HibernateException: HHH000469: The ClassLoaderService can not be reused. This instance was stopped already.

But I dont get why do I get that?

vmassol · April 20, 2020, 4:44pm

On this topic, I’m not sure it’s the best. The best IMO is to use the standard image and to deploy your extension as an extension (i.e. install it with the extension manager or using the REST API).

One reason is that your authenticator might have some dependencies.

Anyway this is just a FYI and not the cause of your problem it seems.

vmassol · April 20, 2020, 4:47pm

No idea either. Seems to be related to your authenticator. Apparently XWiki is stopped. Maybe the xwiki webapps fails to be deployed because of problems with your authenticator?

snawaz · April 20, 2020, 6:07pm

No idea either. Seems to be related to your authenticator. Apparently XWiki is stopped. Maybe the xwiki webapps fails to be deployed because of problems with your authenticator?

Interesting. How did you infer that XWiki is stopped?

My custom endpoints /custom/liveness and /custom/readiness are still being invoked by Kubenetes and I can see the logs from the wiki container. That means, xwiki webapps deployed and running?

snawaz · April 20, 2020, 6:27pm

That sounds very useful in general.

However, how would that work in case of authenticator given that I’d also need to modify the authclass key in xwiki.cfg?

vmassol · April 20, 2020, 6:29pm

That’s very easy, you just modify whatever you want in your xwiki.cfg file

vmassol · April 20, 2020, 6:30pm

Maybe you missed the fact that the xwiki config files are present in your local mapped volume? See GitHub - xwiki/xwiki-docker: Dockerfile to build and run XWiki on docker

snawaz · April 23, 2020, 5:26am

I tried to debug it further. So I commented authclass line in xwiki.cfg on my persistent storage (thinking it’d work fine if my extension have problem) and I restarted the k8s pod.

Now I’m getting this:

23-Apr-2020 05:17:55.639 INFO [localhost-startStop-1] org.apache.catalina.startup.HostConfig.deployDirectory Deployment of web application directory [/usr/local/tomcat/webapps/ROOT] has finished in [19,984] ms
23-Apr-2020 05:17:55.642 INFO [main] org.apache.coyote.AbstractProtocol.start Starting ProtocolHandler ["http-nio-8080"]
23-Apr-2020 05:17:55.662 INFO [main] org.apache.catalina.startup.Catalina.start Server startup in 20103 ms
2020-04-23 05:17:56,857 ERROR aultExtensionRepositoryManager - Unexpected error when trying to find extension [org.xwiki.contrib.authentication:xwiki-authenticator-snawaz/1.0.0] in repository [maven-xwiki:maven:https://nexus.xwiki.org/nexus/content/groups/public] 
org.xwiki.extension.ResolveException: Failed to resolve artifact [org.xwiki.contrib.authentication:xwiki-authenticator-snawaz:jar:1.0.0] descriptor
	at org.xwiki.extension.repository.aether.internal.AetherExtensionRepository.resolveMaven(AetherExtensionRepository.java:491)
	at org.xwiki.extension.repository.aether.internal.AetherExtensionRepository.resolveMaven(AetherExtensionRepository.java:474)
	at org.xwiki.extension.repository.aether.internal.AetherExtensionRepository.resolveMaven(AetherExtensionRepository.java:468)
	at org.xwiki.extension.repository.aether.internal.AetherExtensionRepository.resolve(AetherExtensionRepository.java:273)
	at org.xwiki.extension.repository.internal.DefaultExtensionRepositoryManager.resolve(DefaultExtensionRepositoryManager.java:286)
	at org.xwiki.extension.repository.internal.core.DefaultCoreExtensionScanner.updateExtensions(DefaultCoreExtensionScanner.java:116)
	at org.xwiki.extension.repository.internal.core.DefaultCoreExtensionRepository$1.run(DefaultCoreExtensionRepository.java:138)
	at java.base/java.lang.Thread.run(Thread.java:834)
Caused by: org.eclipse.aether.resolution.ArtifactResolutionException: Could not transfer artifact org.xwiki.contrib.authentication:xwiki-authenticator-snawaz:pom:1.0.0 from/to maven-xwiki (https://nexus.xwiki.org/nexus/content/groups/public): status code: 503, reason phrase: Service Temporarily Unavailable (503)
	at org.eclipse.aether.internal.impl.DefaultArtifactResolver.resolve(DefaultArtifactResolver.java:424)
	at org.eclipse.aether.internal.impl.DefaultArtifactResolver.resolveArtifacts(DefaultArtifactResolver.java:229)
	at org.eclipse.aether.internal.impl.DefaultArtifactResolver.resolveArtifact(DefaultArtifactResolver.java:207)
	at org.xwiki.extension.repository.aether.internal.AetherExtensionRepository.downloadPom(AetherExtensionRepository.java:624)
	at org.xwiki.extension.repository.aether.internal.AetherExtensionRepository.resolveMaven(AetherExtensionRepository.java:485)
	... 7 common frames omitted
Caused by: org.eclipse.aether.transfer.ArtifactTransferException: Could not transfer artifact org.xwiki.contrib.authentication:xwiki-authenticator-snawaz:pom:1.0.0 from/to maven-xwiki (https://nexus.xwiki.org/nexus/content/groups/public): status code: 503, reason phrase: Service Temporarily Unavailable (503)
	at org.eclipse.aether.connector.basic.ArtifactTransportListener.transferFailed(ArtifactTransportListener.java:52)
	at org.eclipse.aether.connector.basic.BasicRepositoryConnector$TaskRunner.run(BasicRepositoryConnector.java:369)
	at org.eclipse.aether.util.concurrency.RunnableErrorForwarder$1.run(RunnableErrorForwarder.java:75)
	at org.eclipse.aether.connector.basic.BasicRepositoryConnector$DirectExecutor.execute(BasicRepositoryConnector.java:644)
	at org.eclipse.aether.connector.basic.BasicRepositoryConnector.get(BasicRepositoryConnector.java:262)
	at org.eclipse.aether.internal.impl.DefaultArtifactResolver.performDownloads(DefaultArtifactResolver.java:499)
	at org.eclipse.aether.internal.impl.DefaultArtifactResolver.resolve(DefaultArtifactResolver.java:401)
	... 11 common frames omitted
Caused by: org.apache.http.client.HttpResponseException: status code: 503, reason phrase: Service Temporarily Unavailable (503)
	at org.eclipse.aether.transport.http.HttpTransporter.handleStatus(HttpTransporter.java:472)
	at org.eclipse.aether.transport.http.HttpTransporter.execute(HttpTransporter.java:294)
	at org.eclipse.aether.transport.http.HttpTransporter.implGet(HttpTransporter.java:246)
	at org.eclipse.aether.spi.connector.transport.AbstractTransporter.get(AbstractTransporter.java:67)
	at org.eclipse.aether.connector.basic.BasicRepositoryConnector$GetTaskRunner.runTask(BasicRepositoryConnector.java:457)
	at org.eclipse.aether.connector.basic.BasicRepositoryConnector$TaskRunner.run(BasicRepositoryConnector.java:364)
	... 16 common frames omitted

Why does it still try to get my extension when it is not needed anymore? (I checked xwiki.cfg again. The line is still commented). Also, where does it get the name of extension? I previously assumed that the only point of control is authclass key in xwiki.cfg. If I comment that, it should use the default authenticator by default, as if there was no user-defined extension at all?

snawaz · April 23, 2020, 5:32pm

Wow. You’re right. I’m able to reproduce the same error without GCP Kubernetes. I ran it on my local docker.

My extension also defines custom endpoints, to be used by k8s as probes.

/snawaz/liveness
/snawaz/readiness

Those are causing problems in k8s, though they didn’t cause any issue locally as previously I didn’t hit them as soon as I start the container. So this time, I wrote this HTML/JS which periodically hits the probes endpoints every 10 seconds, just like k8s does (by default) — irrespective of whether wiki is running or not. Once this html+js app is up and running in the browser, I restarted the container. Voila! I got the same error as I did on k8s.

However, if I first let the container starts, then hit the endpoints, they work fine. So the endpoints seem to be fine by themselves?

Looks like a race condition? Or something must be initialized before the probes become active! How do I ensure that?

<html>
  <head>
    <title> probes </title>
    <script>
       function probe() {
         setInterval(do_probe, 10 * 1000); //every 10 seconds
       }
       async function do_probe() {
         const lr = await fetch("http://localhost/snawaz/liveness");
         const rr = await fetch("http://localhost/snawaz/readiness");
         console.log("liveness: ", lr.status); 
         console.log("readiness: ", rr.status); 
       }
       probe()
    </script>
  </head>
</html>