Icons are gone after upgrade to 11.10.1

moscardo · July 7, 2020, 9:16am

Hi,
After I have upgraded our testing instance, I cannot see most of the icons anymore. I have followed the upgrade path between versions (I went from 10.11 to 11.10.1).

I have tried cleaning up the browser cache, removing solr/* files…

I also get plenty of:

com.xpn.xwiki.XWikiException: Error number 3002 in 3: The attachment [Attachment xwiki:Invitation.InvitationConfig@InvitationSectionIcon.png] (file /usr/local/xwiki/data/store/file/xwiki/e/c/528ed7a2fc0bf34fd62a1125fadbdf/attachments/d/a/b02315e591f84ef8a9f92fd91afec2/f.png) could not be found in the filesystem attachment store.
	at org.xwiki.store.legacy.store.internal.FilesystemAttachmentStore.loadAttachmentContent(FilesystemAttachmentStore.java:214)

vmassol · July 7, 2020, 9:20am

Hi, could you tell us the location of the xwiki permanent directory for your xwiki 10.11 instance and whether you set up the new instance to point to that permanent directory? That would explain the issue if the new instance is not pointing to it.

See https://www.xwiki.org/xwiki/bin/view/Documentation/AdminGuide/Configuration/#HPermanentDirectory

Thanks

moscardo · July 7, 2020, 9:42am

Hi Vincent,
I have the same directory as a permanent:

xwiki.properties:environment.permanentDirectory=/usr/local/xwiki/data
root@itswiki-759c4c4555-hm9gm:/usr/local/xwiki/data#  df -h .
Filesystem                                                     Size  Used Avail Use% Mounted on
10.11.5.186:/trident_pvc_e1c7a3fe_fc5e_43db_b7d3_4d474d76d65e  7.7G  661M  7.0G   9% /usr/local/xwiki
root@itswiki-759c4c4555-hm9gm:/usr/local/xwiki/data# ls -ltr
total 124
drwxr-x---. 3 99 99  4096 Jun 12 13:33 jobs
drwxr-x---. 4 99 99  4096 Jun 12 13:33 extension
drwxr-x---. 3 99 99  4096 Jun 12 13:33 cache
drwxr-x---. 9 99 99  4096 Jun 20 00:00 mails
drwxr-x---. 3 99 99  4096 Jun 26 15:03 store
-rw-r--r--. 1 99 99  5810 Jun 26 15:04 hibernate.cfg.xml
-rw-r--r--. 1 99 99 46462 Jun 26 15:04 xwiki.properties
-rw-r--r--. 1 99 99 29583 Jun 26 15:08 xwiki.cfg
drwxr-x---. 3 99 99  4096 Jul  7 09:06 solr-1594112796091
drwxr-x---. 5 99 99  4096 Jul  7 09:06 solr

vmassol · July 7, 2020, 10:04am

Maybe you could tell us the exact steps you did to perform the upgrade?

moscardo · July 7, 2020, 10:15am

From a new container I pulled the config files and I have merged them.
Mount the permanent data into the new path.
Adjust the new location for the database(url)
Set xwiki.store.migration=1
Delete solr/* data from permanent storage
Launch new deployment

Could this be related to “Known issues” https://www.xwiki.org/xwiki/bin/view/ReleaseNotes/Data/XWiki/11.9/ ? I am doing SSL termination on the ingress (traefik running on k8s) and then plain text to the backend.

vmassol · July 7, 2020, 1:01pm

Note that this is no longer needed, we now manage this automatically when needed

vmassol · July 7, 2020, 1:02pm

Your process seems good. What we need to find out is why you have some attachments defined in the DB that do not have the corresponding files on the filesystem in the perm dir.

vmassol · July 7, 2020, 1:03pm

No, these are all the open bugs that exist but there are none that I know of about your issue.

vmassol · July 7, 2020, 1:03pm

Now going to 11.10.1 is not a great idea. It’s much better to go to 11.10.5 (the LTS), which has a lot of bugs fixed compared to 11.10.1. You could try to upgrade to 11.10.5 now. It’s possible that this is an old bug that we fixed.

moscardo · July 7, 2020, 1:10pm

thanks! I will go ahead and upgrade it. What is confusing during the upgrade path is that if at some version I need to delete solr/* but for the next release I don’t need to do that step, I would still do it, I am not sure if this could cause more troubles.

vmassol · July 7, 2020, 1:20pm

Whenever you need to do something it’s written in the release notes. If it’s not written you must not do it. Also the location changed in the last versions and you should never delete the store/solr/ directory (there’s no more solr directory), as you’ll delete important data (all the events in the wiki). See https://www.xwiki.org/xwiki/bin/view/Documentation/AdminGuide/Configuration/#HPermanentDirectory

vmassol · July 7, 2020, 1:21pm

The reason for deleting the solr/ directory in the past is whenever solr changed its schema and XWiki needed to do a reindex. We now handle this automatically.

moscardo · July 7, 2020, 1:25pm

I am upgrading, this is what I get (also got it during the previous upgrade)

vmassol · July 7, 2020, 1:32pm

Yes, as I said, this means that you have attachments in some pages and the metadata for these attachments are in the DB but they cannot be found on the filesystem in the permanent directory.

What I cannot fathom/guess is why you don’t have them in your permanent directory.

So most likely the upgrade (which is a good thing) is not going to fix this.

moscardo · July 7, 2020, 2:46pm

right , it didn’t fix the issue. I am confused because with version 10.11 I had less data directories than I have now with 11.10.5, I am not usre if the migration moved things around:
This is what I had previously:

And this is what it looks like after the upgrade:

vmassol · July 7, 2020, 2:57pm

No idea, maybe @tmortagne has an idea?

moscardo · September 16, 2020, 10:35am

I manage to hit this issue again.
I wen to test instance, copied data from production and DB dump as well, injected into test, and perform the upgrade, all wen fine (first I moved to 11.0 and then to 11.10). So I thought I may have done a mistake before, so I decided to go ahead and do it in production and the same happened again, a lot of icons are missing. I wen straight from 10.11 to 11.10 merging files, changing storage location and deleting solr.
I have now screwed my prod wiki…

Any help here?

WARN  d.AttachmentUnifiedDiffBuilder - Failed to compare the content of attachment [img.png]. Root cause: XWikiException: Error number 3002 in 3: The attachment [Attachment xwiki:XWiki.XWikiSyntaxImages@img.png] (file /usr/local/xwiki/data/store/file/xwiki/b/a/aae6d74b0f5b2287bb6b9ed962b946/attachments/d/1/0c88f869301b1238f53cfdff8e9d7c/f.png) could not be found in the filesystem attachment store. 
2020-09-16 10:35:55,914 [XWiki initialization] WARN  d.AttachmentUnifiedDiffBuilder - Failed to compare the content of attachment [img.png]. Root cause: XWikiException: Error number 3002 in 3: The attachment [Attachment xwiki:XWiki.XWikiSyntaxLinks@img.png] (file /usr/local/xwiki/data/store/file/xwiki/8/d/649084435d65ddb826004b06a09929/attachments/d/1/0c88f869301b1238f53cfdff8e9d7c/f.png) could not be found in the filesystem attachment store. 
2020-09-16 10:35:56,468 [XWiki initialization] WARN  d.AttachmentUnifiedDiffBuilder - Failed to compare the content of attachment [AnnotationsSectionIcon.png]. Root cause: XWikiException: Error number 3002 in 3: The attachment [Attachment xwiki:AnnotationCode.AnnotationConfig@AnnotationsSectionIcon.png] (file /usr/local/xwiki/data/store/file/xwiki/1/e/eddb53ddf348eb1c96ab28a0337204/attachments/c/2/c61ac346456238efb1fc44408107dc/f.png) could not be found in the filesystem attachment store.

A lot of entries like this are present and those files were existing before the migration.

moscardo · September 16, 2020, 10:47am

I mean, those where existing before, but never in this location, so I am not sure what has failed in the migration that is now moving the files somewhere else.

moscardo · September 16, 2020, 10:50am

this is the failed migration:

[root@ops-k1n09 mount]# ls -l data/store/file/xwiki/
total 132
drwxr-x---.  3 nobody nobody 4096 sep 16 12:17 1
drwxr-x---.  4 nobody nobody 4096 sep 16 12:17 3
drwxr-x---.  3 nobody nobody 4096 sep 16 12:02 4
drwxr-x---.  4 nobody nobody 4096 abr  1  2019 %41nnotation%43ode
drwxr-x---.  4 nobody nobody 4096 abr  1  2019 %41pp%57ithin%4Dinutes
drwxr-x---.  4 nobody nobody 4096 abr  1  2019 %43%4B%45ditor
drwxr-x---.  8 nobody nobody 4096 jun 27  2019 %44esktop
drwxr-x---. 14 nobody nobody 4096 abr  1  2019 %46lamingo%54hemes
drwxr-x---.  3 nobody nobody 4096 may  7  2019 %48%50%43 cluster
drwxr-x---.  5 nobody nobody 4096 abr  1  2019 %48elp
drwxr-x---. 12 nobody nobody 4096 jun 27  2019 %49%54%53 %4Fps
drwxr-x---.  3 nobody nobody 4096 abr  1  2019 %49nvitation
drwxr-x---.  3 nobody nobody 4096 may 23  2019 %4Dacros
drwxr-x---.  3 nobody nobody 4096 abr  1  2019 %4Dail
drwxr-x---.  4 nobody nobody 4096 may 23  2019 %4Dain
drwxr-x---.  4 nobody nobody 4096 jun 27  2019 %4Eetwork
drwxr-x---.  3 nobody nobody 4096 jul  3  2019 %4Eew%48ome
drwxr-x---.  4 nobody nobody 4096 sep 16 12:17 5
drwxr-x---.  3 nobody nobody 4096 abr  1  2019 %50anels
drwxr-x---.  3 nobody nobody 4096 abr  1  2019 %53andbox
drwxr-x---.  8 nobody nobody 4096 jun 27  2019 %53ecurity %54eam
drwxr-x---.  4 nobody nobody 4096 abr  1  2019 %54emplates
drwxr-x---.  5 nobody nobody 4096 jun 27  2019 %57eb
drwxr-x---. 52 nobody nobody 4096 jul 22 10:49 %58%57iki
drwxr-x---.  3 nobody nobody 4096 abr 27 14:03 %58%57iki%2F%49%54 %4Fperations %54eam%2F%53torage%2F%41rchiving +
drwxr-x---.  3 nobody nobody 4096 sep 16 12:17 8
drwxr-x---.  3 nobody nobody 4096 sep 16 12:02 a
drwxr-x---.  3 nobody nobody 4096 jul  1  2019 asdf
drwxr-x---.  4 nobody nobody 4096 sep 16 12:17 b
drwxr-x---.  3 nobody nobody 4096 sep 16 12:02 c
drwxr-x---.  3 nobody nobody 4096 sep 16 12:08 d
drwxr-x---.  3 nobody nobody 4096 sep 16 12:13 e
drwxr-x---.  4 nobody nobody 4096 sep 16 12:17 f

and this is the test instance that succeeded :

root@itswiki-65db8db489-954zz:/usr/local/xwiki/data/store/file/xwiki# ls -ltr
total 64
drwxr-x---.  7 99 99 4096 Sep 15 14:26 d
drwxr-x---.  9 99 99 4096 Sep 15 14:26 1
drwxr-x---. 13 99 99 4096 Sep 15 14:26 8
drwxr-x---.  6 99 99 4096 Sep 15 14:26 7
drwxr-x---. 11 99 99 4096 Sep 15 14:26 6
drwxr-x---. 15 99 99 4096 Sep 15 14:35 3
drwxr-x---. 14 99 99 4096 Sep 15 14:35 2
drwxr-x---. 10 99 99 4096 Sep 15 14:35 e
drwxr-x---. 13 99 99 4096 Sep 15 14:35 9
drwxr-x---. 10 99 99 4096 Sep 15 14:35 4
drwxr-x---. 11 99 99 4096 Sep 15 14:35 0
drwxr-x---. 12 99 99 4096 Sep 15 14:35 5
drwxr-x---. 12 99 99 4096 Sep 15 14:35 f
drwxr-x---. 12 99 99 4096 Sep 15 15:15 b
drwxr-x---.  8 99 99 4096 Sep 15 15:15 c
drwxr-x---. 14 99 99 4096 Sep 15 15:15 a

So it looks like there was involved a renaming during the migration but didn’t complete but I didn’t get any error regarding that.

moscardo · September 16, 2020, 11:27am

I think this is related to : https://forum.xwiki.org/t/attachment-issues-after-upgrade-from-10-11-to-11-0/4298/29
and it wasn’t clear to me whether I should have renamed /storage to /store/file or this was something that the migration process was supposed to do.

I see this in the logs:

2020-09-16 11:29:23,770 [XWiki initialization] INFO  .HibernateDataMigrationManager - Storage schema updates and data migrations are enabled 
2020-09-16 11:29:24,028 [XWiki initialization] INFO  .HibernateDataMigrationManager - No data migration to apply for wiki [xwiki] currently in version [1138000]

should I bump it to some early version so it could re launch the migration?