Many attachments are reported missing after upgrade to 10.4

Help / Discuss
#1

Just completed an upgrade to 10.4 last night and found that several attachments are reported missing. Prior to this upgrade, we were using filesystem storage for attachments for several years.

I have the following settings in my new xwiki.cfg as per documentation:

xwiki.store.recyclebin.content.hint=file
xwiki.store.attachment.hint=file
xwiki.store.attachment.versioning.hint=file
xwiki.store.attachment.recyclebin.content.hint=file
xwiki.store.attachment.recyclebin.hint=file

When I check in the filesystem permanent directory /var/local/scd/storage/... I can locate the attachments in the filesystem; however, those are the same attachments XWiki doesn’t think exists. In my Admin application > DB to Filesystem Porter macro/app I am not even seeing those “missing” attachments appear (namely my user’s profile picture).

My catalina.out is filled with this several times when a page loads where my user profile pic should appear:

[https://MYWIKI.elasticbeanstalk.com/bin/download/XWiki/USERNAME/E-Profile.jpg?width=120&height=120&keepAspectRatio=true] WARN  c.x.x.p.i.ImagePlugin          - Failed to transform image attachment.
com.xpn.xwiki.XWikiException: Error number 3231 in 3: Exception while loading attachment Attachment xwiki:XWiki.USERNAME@E-Profile.jpg
....
Caused by: org.hibernate.ObjectNotFoundException: No row with the given identifier exists: [com.xpn.xwiki.doc.XWikiAttachmentContent#-124564659]

I noticed some JIRA ticket about this issue from way back and then another around 10.4 - 10.5 release. Is there anything I can do in the short term to get these back? I would love to get my users back on XWiki as soon as possible.

#2

Since XWiki now support mixed storage for attachments a migration has been executed when you upgraded to 10.4 (you had xwiki.store.migration=1 in xwiki.cfg during the upgrade, right ?) to set the right storage type for each one and for some reason the migrator did not seen that those specific attachments were on the file system store. You did not get any error or warning in the log during the migration ?

Could you indicate the exact path of those attachments on the file system and where they are located in the wiki. Maybe there is something special about those attachments which is not well supported by the migrator.

Also I did not understood if you have the issue with all filesystem attachments or just some.

Actually the documentation indicate that you should not set xwiki.store.attachment.recyclebin.hint = file since 9.10RC1.

You should NEVER use the old attachment porter anymore as indicated on https://snippets.xwiki.org/xwiki/bin/view/Extension/Filesystem+Attachment+Porter. Better removing it.

The easiest way to fix an attachment is to upload it again if you have it (but seems its the case). Still I would love to understand what is the cause of this in case it’s a bug in the migrator.

#3

you had xwiki.store.migration=1 in xwiki.cfg during the upgrade, right ?

Yes, I have that setting and it’s enabled.

You did not get any error or warning in the log during the migration ?

I did see several of those “could not find attachment” but I figured that since it was an upgrade and I had removed the old SOLR subdirectory of my permanent directory (to not carry over old schema, etc.) that it would resolve itself after I did a full re-index from the XWikiPreferences > Search settings in the UI.

Could you indicate the exact path of those attachments on the file system and where they are located in the wiki. Maybe there is something special about those attachments which is not well supported by the migrator.

One example is my profile picture. It should exist in MYWIKI.com/bin/download/XWiki/USERNAME/E-Profile.jpg in the wiki. On my permanent directory / filesystem it appears under /var/local/scd/storage/xwiki/XWiki/USERNAME/~this/attachments and in fact both my old profile picture (which does appear to be found) and my current profile picture (the one that is missing) are in there with tomcat:tomcat 755 permissions and both have some file size to them so not empty.

Also I did not understood if you have the issue with all filesystem attachments or just some.

The issue is not with all attachments, but only some. For example, we have a landing page for all users that has an activity stream. Only 1 user (out of about 7 or 8 in the last 50 items of activity) has their profile pic loading.

I must have misread that part about the recyclebin.hint setting. There are many attachments missing on old reports we wrote so hunting down the images would be a really time intensive exercise that we would not want to do. It dates back a few years.

#4

Which version of XWiki have you upgraded from ?

Well this really did not had anything to do with SOLR, you should find your missing attachments listed. Would be great if you could find those logs again :slight_smile: Unless you are actually talking about Solr error logs in which case it was not during the migration but after it. The migration is the first log you get printed right after you access XWiki fro the first time with a browser.

This is really not how path looks like in 10.4, are you sure you actually have the image files in that and not just folders ? The right path should look like /var/local/scd/storage/xwiki/%58%57iki/%55%53%45%52%4E%41%4D%45/~this/attachments/%45-%50rofile.jpg/E-Profile.jpg (upper case letters are encoded now to avoid collision in case insensitive filesystem).

It’s easy to indicate in the DB that the attachment content is located on the filesystem but first I would need to be sure that you actually have the files in the right place.

#5

We have upgraded from 6.0.1 to 10.4. We added those standard customizations for our DB connection in hibernate.cfg.xml, a few simple things in xwiki.cfg and xwiki.properties including those that I mentioned above. The logs have been rotated but I did see the giant block of migration INFO lines — I’ll attach them here.

#6

Shockingly, I’m not finding the catalina.out logs with the read out showing the migration happening that I saw yesterday. It’s as if log rotation didn’t kick in or was inadvertently turned off until after. I have a snapshot from before the migration to 10.4 that I’ll spin up into a test environment and try to re-capture the output.

#7

I ran the attachments checker on my main wiki and found that out of 617 attachments, 170 of them were reporting either unvalid (exception Cannot invoke method getContent() on null object) or (exception Error number 3231 in 3: Exception while loading attachment Attachment.... I may just have to re-initialize this wiki farm in a test environment to get more accurate logs.

#8

Looks like you were right. I re-initialized the DB to a snapshot before the migration to 10.4 and re-ran the migration. It looks like I ran into a DATABASECHANGELOG error (couldn’t release it for writing) and this time it didn’t occur so it was able to run completely fine. I can see all of my attachments now. Folder structure now reflects what you were mentioning for 10.4. Thanks for the helpful clues !

#9

Great news !

Do you think the issue you had in previous upgrade is something that can be improved on XWiki side ?

#10

I’m not sure if maybe my large database that we have carried forward for close to 5 years is to blame. I think the amount of info that the log reads out is very verbose and it is hard to see errors buried in there - but it’s also nice to be able to search through and pinpoint / search for specific files.