Various issues/errors after deployment (Missing links, pictures, etc...)

Hi,

We are having some odd issues recently and I’m not sure what is causing it, if it’s something configured incorrectly, or perhaps just the way our containers are setup.

Originally client said that they were getting lots of 500 errors trying to use the wiki, which was investigated and after a redeployment, the 500 errors went away, but then when users logged in, lots of things were broken. For example, the profile pictures are broken, the admin links are missing, various language strings appear to be missing. (See screenshots: https://ibb.co/JrVTstX https://ibb.co/MpsDPZv)

I have also noticed other errors navigating around the site, for example if I go to view/edit a user’s profile, I get the following:

org.xwiki.rendering.macro.MacroExecutionException: Failed to evaluate Velocity Macro for content [#########################
## CSS & Javascripts
#########################
#set($discard = $xwiki.ssx.use("XWiki.XWikiUserSheet"))
#set($discard = $xwiki.jsx.use("XWiki.XWikiUserSheet"))
#########################
## Setting categories
#########################
#set($categories = [])

## load the user menu from the 'org.xwiki.plaftorm.user.profile.menu' UIXP.
#foreach ($uix in $services.uix.getExtensions('org.xwiki.plaftorm.user.profile.menu', {'sortByParameter': 'priority'}))
  #if(!$uix.parameters.containsKey('isActive') || $uix.parameters.get('isActive') != 'false')
    #if (!$uix.parameters.containsKey('id'))
      ## when no id is explicitly provided, we use the id of the UIX.
      #set ($discard = $uix.parameters.put('id', $uix.id))
    #end
    #if ($uix.parameters.containsKey('icon'))
      #set ($discard = $uix.parameters.put('glyphicon', $uix.parameters.get('icon')))
    #end
    #set ($discard = $uix.parameters.put('uix', $uix))
    #set($discard = $categories.add($uix.parameters))
  #end
#end
#########################
## Current category
#########################
#set($currentCategory = "$!request.category")
#if($currentCategory == "")
  #set($currentCategory = $categories[0].get('id'))
#end
#########################
## Creating vertical menu
#########################
#set($userMenu = [{
  'id'       : 'settings',
  'cssClass' : 'user-menu-title',
  'children' : $categories
  }
])
## Setting automaticaly url & css fields of each category
#foreach($category in $userMenu)
  #foreach($subcategory in $category.get('children'))
    #set($id = $subcategory.get('id')) 
    #set($discard = $subcategory.put('url', "?category=${id}"))
    #set($discard = $subcategory.put('cssClass', "user-menu-$id category-tab"))
  #end
#end
#########################
## Display the left menu
#########################
(% id="user-menu-col" %)
(((
  ############
  ## Avatar
  ############
  (% id="avatar" %)
  (((
    #if($request.xpage == 'edituser')
      {{html clean="false"}}
        #resizedUserAvatar($doc.fullName 180)
      {{/html}}
    #else
      ## By specifying the image width we enable server side resizing. The width value we use is greater than the
      ## available space because we don't want to loose too much of the image quality (we rely on the browser to fit the
      ## image in the available space).
      #set($isMyProfile = ($services.model.resolveDocument($xcontext.user) == $doc.documentReference))
      {{attachmentSelector classname="XWiki.XWikiUsers" object="$obj.number" property="avatar" #if ($isMyProfile) savemode="direct" #end defaultValue="XWiki.XWikiUserSheet@noavatar.png" width="180" alternateText="$xwiki.getUserName($doc.fullName, false)" buttontext="$services.localization.render('platform.core.profile.changePhoto')" displayImage="true" filter="png,jpg,jpeg,gif"/}}
    #end
  )))
  ##########
  ## Menu
  ##########
  (% id="user-vertical-menu" %)
  (((
    #verticalNavigation($userMenu, {'translationPrefix' : 'platform.core.profile.category.', 'crtItemId' : $currentCategory, 'cssClass' : 'profile-menu'})
  )))
)))
#########################
## Display the page content
#########################
(% id="user-page-content" %)
(((
  #foreach($category in $userMenu)
    #foreach($subcategory in $category.get('children'))
      #set($tabKey = $subcategory.get('id')) 
      (% id="${tabKey}Pane" class="user-page-pane#if($tabKey != $currentCategory) hidden#end" %)
      (((
        {{html}}$services.rendering.render($subcategory.uix.execute(), 'html/5.0'){{/html}}
      )))
    #end
  #end
)))
#########################
## END
#########################
{{html clean="false"}}
  #if($xcontext.action == 'edit' || $xcontext.action == 'inline')
    <input type='hidden' name='category' value="$!{escapetool.xml($currentCategory)}" />
  #end
  <div class="clearfloats">&nbsp;</div>
{{/html}}]
	at org.xwiki.rendering.internal.macro.velocity.VelocityMacro.evaluateString(VelocityMacro.java:139)
	at org.xwiki.rendering.internal.macro.velocity.VelocityMacro.evaluateString(VelocityMacro.java:52)
	at org.xwiki.rendering.macro.script.AbstractScriptMacro.evaluateBlock(AbstractScriptMacro.java:286)
	at org.xwiki.rendering.macro.script.AbstractScriptMacro.execute(AbstractScriptMacro.java:182)
	at org.xwiki.rendering.macro.script.AbstractScriptMacro.execute(AbstractScriptMacro.java:58)
	at org.xwiki.rendering.internal.transformation.macro.MacroTransformation.transform(MacroTransformation.java:297)
	at org.xwiki.rendering.internal.transformation.DefaultRenderingContext.transformInContext(DefaultRenderingContext.java:183)
	at org.xwiki.rendering.internal.transformation.DefaultTransformationManager.performTransformations(DefaultTransformationManager.java:101)
	at org.xwiki.display.internal.DocumentContentAsyncExecutor.executeInCurrentExecutionContext(DocumentContentAsyncExecutor.java:348)
	at org.xwiki.display.internal.DocumentContentAsyncExecutor.execute(DocumentContentAsyncExecutor.java:221)
	at org.xwiki.display.internal.DocumentContentAsyncRenderer.execute(DocumentContentAsyncRenderer.java:107)
	at org.xwiki.rendering.async.internal.block.AbstractBlockAsyncRenderer.render(AbstractBlockAsyncRenderer.java:157)
	at org.xwiki.rendering.async.internal.block.AbstractBlockAsyncRenderer.render(AbstractBlockAsyncRenderer.java:54)
	at org.xwiki.rendering.async.internal.DefaultAsyncRendererExecutor.syncRender(DefaultAsyncRendererExecutor.java:270)
	at org.xwiki.rendering.async.internal.DefaultAsyncRendererExecutor.render(DefaultAsyncRendererExecutor.java:247)
	at org.xwiki.rendering.async.internal.block.DefaultBlockAsyncRendererExecutor.execute(DefaultBlockAsyncRendererExecutor.java:125)
	at org.xwiki.display.internal.DocumentContentDisplayer.display(DocumentContentDisplayer.java:67)
	at org.xwiki.display.internal.DocumentContentDisplayer.display(DocumentContentDisplayer.java:43)
	at org.xwiki.display.internal.DefaultDocumentDisplayer.display(DefaultDocumentDisplayer.java:96)
	at org.xwiki.display.internal.DefaultDocumentDisplayer.display(DefaultDocumentDisplayer.java:39)
	at org.xwiki.sheet.internal.SheetDocumentDisplayer.display(SheetDocumentDisplayer.java:245)
	at org.xwiki.sheet.internal.SheetDocumentDisplayer.applySheet(SheetDocumentDisplayer.java:225)
	at org.xwiki.sheet.internal.SheetDocumentDisplayer.maybeDisplayWithSheet(SheetDocumentDisplayer.java:180)
	at org.xwiki.sheet.internal.SheetDocumentDisplayer.display(SheetDocumentDisplayer.java:111)
	at org.xwiki.sheet.internal.SheetDocumentDisplayer.display(SheetDocumentDisplayer.java:52)
	at org.xwiki.display.internal.ConfiguredDocumentDisplayer.display(ConfiguredDocumentDisplayer.java:68)
	at org.xwiki.display.internal.ConfiguredDocumentDisplayer.display(ConfiguredDocumentDisplayer.java:42)
	at com.xpn.xwiki.doc.XWikiDocument.display(XWikiDocument.java:1224)
	at com.xpn.xwiki.doc.XWikiDocument.getRenderedContent(XWikiDocument.java:1365)
	at com.xpn.xwiki.doc.XWikiDocument.getRenderedContent(XWikiDocument.java:1332)
	at com.xpn.xwiki.doc.XWikiDocument.getRenderedContent(XWikiDocument.java:1242)
	at com.xpn.xwiki.doc.XWikiDocument.getRenderedContent(XWikiDocument.java:1374)
	at com.xpn.xwiki.api.Document.getRenderedContent(Document.java:789)
	at jdk.internal.reflect.GeneratedMethodAccessor398.invoke(Unknown Source)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:566)
	at org.apache.velocity.util.introspection.UberspectImpl$VelMethodImpl.doInvoke(UberspectImpl.java:565)
	at org.apache.velocity.util.introspection.UberspectImpl$VelMethodImpl.invoke(UberspectImpl.java:548)
	at org.apache.velocity.runtime.parser.node.ASTMethod.execute(ASTMethod.java:219)
	at org.apache.velocity.runtime.parser.node.ASTReference.execute(ASTReference.java:369)
	at org.apache.velocity.runtime.parser.node.ASTReference.render(ASTReference.java:490)
	at org.apache.velocity.runtime.parser.node.ASTBlock.render(ASTBlock.java:144)
	at org.apache.velocity.runtime.parser.node.SimpleNode.render(SimpleNode.java:423)
	at org.apache.velocity.runtime.parser.node.ASTIfStatement.render(ASTIfStatement.java:191)
	at org.apache.velocity.runtime.parser.node.SimpleNode.render(SimpleNode.java:423)
	at org.apache.velocity.Template.merge(Template.java:358)
	at org.apache.velocity.Template.merge(Template.java:262)
	at org.xwiki.velocity.internal.DefaultVelocityEngine.evaluate(DefaultVelocityEngine.java:281)
	at com.xpn.xwiki.render.DefaultVelocityManager.evaluate(DefaultVelocityManager.java:316)
	at com.xpn.xwiki.internal.template.VelocityTemplateEvaluator.evaluateContent(VelocityTemplateEvaluator.java:95)
	at com.xpn.xwiki.internal.template.TemplateAsyncRenderer.evaluateContent(TemplateAsyncRenderer.java:217)
	at com.xpn.xwiki.internal.template.TemplateAsyncRenderer.renderVelocity(TemplateAsyncRenderer.java:180)
	at com.xpn.xwiki.internal.template.TemplateAsyncRenderer.render(TemplateAsyncRenderer.java:137)
	at com.xpn.xwiki.internal.template.TemplateAsyncRenderer.render(TemplateAsyncRenderer.java:53)
	at org.xwiki.rendering.async.internal.DefaultAsyncRendererExecutor.lambda$syncRender$0(DefaultAsyncRendererExecutor.java:264)
	at com.xpn.xwiki.internal.security.authorization.DefaultAuthorExecutor.call(DefaultAuthorExecutor.java:98)
	at org.xwiki.rendering.async.internal.DefaultAsyncRendererExecutor.syncRender(DefaultAsyncRendererExecutor.java:264)
	at org.xwiki.rendering.async.internal.DefaultAsyncRendererExecutor.render(DefaultAsyncRendererExecutor.java:247)
	at org.xwiki.rendering.async.internal.block.DefaultBlockAsyncRendererExecutor.render(DefaultBlockAsyncRendererExecutor.java:154)
	at com.xpn.xwiki.internal.template.InternalTemplateManager.render(InternalTemplateManager.java:772)
	at com.xpn.xwiki.internal.template.InternalTemplateManager.renderFromSkin(InternalTemplateManager.java:745)
	at com.xpn.xwiki.internal.template.InternalTemplateManager.renderFromSkin(InternalTemplateManager.java:725)
	at com.xpn.xwiki.internal.template.InternalTemplateManager.render(InternalTemplateManager.java:711)
	at com.xpn.xwiki.internal.template.DefaultTemplateManager.render(DefaultTemplateManager.java:78)
	at com.xpn.xwiki.XWiki.evaluateTemplate(XWiki.java:2502)
	at com.xpn.xwiki.web.Utils.parseTemplate(Utils.java:179)
	at com.xpn.xwiki.web.XWikiAction.execute(XWikiAction.java:576)
	at com.xpn.xwiki.web.XWikiAction.execute(XWikiAction.java:250)
	at org.apache.struts.action.RequestProcessor.processActionPerform(RequestProcessor.java:425)
	at org.apache.struts.action.RequestProcessor.process(RequestProcessor.java:228)
	at org.apache.struts.action.ActionServlet.process(ActionServlet.java:1913)
	at org.apache.struts.action.ActionServlet.doPost(ActionServlet.java:462)
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:660)
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:741)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:231)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
	at com.xpn.xwiki.web.ActionFilter.doFilter(ActionFilter.java:112)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
	at org.xwiki.wysiwyg.filter.ConversionFilter.doFilter(ConversionFilter.java:109)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
	at org.xwiki.container.servlet.filters.internal.SetHTTPHeaderFilter.doFilter(SetHTTPHeaderFilter.java:63)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
	at org.xwiki.container.servlet.filters.internal.SavedRequestRestorerFilter.doFilter(SavedRequestRestorerFilter.java:208)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
	at org.xwiki.container.servlet.filters.internal.SetCharacterEncodingFilter.doFilter(SetCharacterEncodingFilter.java:111)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
	at org.xwiki.resource.servlet.RoutingFilter.doFilter(RoutingFilter.java:132)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
	at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:199)
	at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:96)
	at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:543)
	at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:139)
	at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:81)
	at org.apache.catalina.valves.AbstractAccessLogValve.invoke(AbstractAccessLogValve.java:690)
	at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:87)
	at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:343)
	at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:615)
	at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:65)
	at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:818)
	at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1627)
	at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
	at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
	at java.base/java.lang.Thread.run(Thread.java:834)
Caused by: org.xwiki.velocity.XWikiVelocityException: Failed to evaluate content with namespace [xwiki:XWiki.XWikiUserSheet]
	at org.xwiki.velocity.internal.DefaultVelocityEngine.evaluate(DefaultVelocityEngine.java:289)
	at com.xpn.xwiki.render.DefaultVelocityManager.evaluate(DefaultVelocityManager.java:316)
	at org.xwiki.rendering.internal.macro.velocity.VelocityMacro.evaluateString(VelocityMacro.java:131)
	... 110 more
Caused by: org.apache.velocity.exception.VelocityException: Error invoking method 'get(java.lang.Integer)' in java.util.ArrayList at xwiki:XWiki.XWikiUserSheet[line 30, column 36]
	at org.apache.velocity.runtime.parser.node.ASTIndex.execute(ASTIndex.java:218)
	at org.apache.velocity.runtime.parser.node.ASTReference.execute(ASTReference.java:369)
	at org.apache.velocity.runtime.parser.node.ASTReference.value(ASTReference.java:701)
	at org.apache.velocity.runtime.parser.node.ASTExpression.value(ASTExpression.java:72)
	at org.apache.velocity.runtime.parser.node.ASTSetDirective.render(ASTSetDirective.java:240)
	at org.apache.velocity.runtime.parser.node.ASTBlock.render(ASTBlock.java:144)
	at org.apache.velocity.runtime.parser.node.ASTIfStatement.render(ASTIfStatement.java:172)
	at org.apache.velocity.runtime.parser.node.SimpleNode.render(SimpleNode.java:423)
	at org.apache.velocity.Template.merge(Template.java:358)
	at org.apache.velocity.Template.merge(Template.java:262)
	at org.xwiki.velocity.internal.DefaultVelocityEngine.evaluate(DefaultVelocityEngine.java:281)
	... 112 more
Caused by: java.lang.reflect.InvocationTargetException
	at java.base/jdk.internal.reflect.GeneratedMethodAccessor220.invoke(Unknown Source)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:566)
	at org.apache.velocity.util.introspection.UberspectImpl$VelMethodImpl.doInvoke(UberspectImpl.java:565)
	at org.apache.velocity.util.introspection.UberspectImpl$VelMethodImpl.invoke(UberspectImpl.java:548)
	at org.apache.velocity.runtime.parser.node.ASTIndex.execute(ASTIndex.java:192)
	... 122 more
Caused by: java.lang.IndexOutOfBoundsException: Index 0 out of bounds for length 0
	at java.base/jdk.internal.util.Preconditions.outOfBounds(Preconditions.java:64)
	at java.base/jdk.internal.util.Preconditions.outOfBoundsCheckIndex(Preconditions.java:70)
	at java.base/jdk.internal.util.Preconditions.checkIndex(Preconditions.java:248)
	at java.base/java.util.Objects.checkIndex(Objects.java:372)
	at java.base/java.util.ArrayList.get(ArrayList.java:458)
	... 128 more

Another issue I’ve noticed is things are all unaligned strangely, e.g: https://ibb.co/m0MmRgR

As far as I know, nothing has changed in the configuration from what was working before. I can attach the xwiki.cfg file if required.

We are running xwiki through docker on Amazon’s ECR. As far as I know, the ‘deployment’ process when we update it is: a pull from dockerregistry, tag it and then push it to ECR (according to one of our sys-admins).

When I ssh into the container itself, it appears that the /usr/local/xwiki/ directory where the config data is stored, is mounted as a Network File System:

name-prod-fs1.name.domain.net:/export/sitedata/xwiki-prod.name.domainu.net/xwiki-data nfs4 2.0T 793G 1.1T 43% /usr/local/xwiki

One of the other developers theorised that this might be the problem, as when the redeployment is done it might be failing to mount it, though I don’t know if that is the case or not.

I appreicate it might be hard to help as you don’t know really how this is all setup and I can try and give more info if needed.

Has anyone ever run into similar issues of missing elements of the wiki? Would anyone be able to shed any light on the possible causes?