Migration import method : idea for the wishlist

Help / Discuss
#1

Hello,

I have done a quick search on the forum around this idea and didn’t find one which would be close, so I write it here.

I would like to add a wish to the XWiki project wishlist:
For the time being there is only one way to upload a Confluence export, before migrating, which is uploading the zip archive through the UI when connected to the XWiki instance.

It comes with some inconvenience when using a remote server to power a XWiki instance, because it needs one download step, followed with one upload step, and a local computer in the middle.

Would it be possible to provide other methods additionnally to the one that exists, in order to be able to remove the “local computer in the middle” to save this step? This is the idea:

  • from within the XWiki instance, be able to provide a link to the file which could be hosted on a file server, and from there, keep the bundle as a zip file in a dedicated directory inside the XWiki install ;
  • from within the XWiki instance, be able to provide the path inside the directories of the install itself where the file could have been uploaded, let’s say, using either a hyperlink as described above, or using a SFTP client.

Why for might you ask? Well, sometimes when working remotely from a place where the internet connection lacks speed, it would allow saving time by preparing the work at one moment, and the rest of the process could be done at any time later.

Also if the migration process fails and needs to be reset, it would be nice to not have to perform the upload again (then having the zip file already available right in a local directory of the XWiki server would be very handy).

Thanks for your work, and thanks to the interest you might have for this suggestion.

#2

The Confluence importer already supports this:
image

#3

Then, if I upload the zip file to /home/my_user/Downloads, I’ll also be able to use the migrator that way?

#4

@ kienphamkien After checking, sorry but your answer is wrong.

You are still uploading from within your own computer. (On Windows).

I want to retrieve the file from within the virtual machine where the server is installed.
Also I checked if I could find the package of the export which I have uploaded in the server (from within my local computer, to the remote server) and a updatedb && locate <my_export.zip> didn’t find it.

#5

No, @jmarkoll , @kienphamkien is right :slightly_smiling_face:

In their screenshot, file:/Users/.... is a path on the remote server , the machine where the XWiki server is running.

It’s this:

The information in the warning is rather clear that you don’t need to upload the file but point to a path.

Now, depending on how your server is installed, your XWiki server may or may not see the file that you have uploaded to that virtual machine:

  • if you’re running XWiki in a container (e.g. docker), the access to files may be different than what you expect;
  • make sure that the permissions on the file uploaded (in your home or elsewhere) allow the user that runs the XWiki server access to that file (for example tomcat would run it with the tomcat user).

The 2 above are not really XWiki related, they’re more general infrastructure managing operations…

This also, I don’t see how it’s related to XWiki (they’re generic linux commands). I mean XWiki has the option to point to a file, the fact that you cannot locate that file after uploading it to the server is not something that XWiki can help with…

Hope this helps.

#6

Hello lucaa,

our VM is installed with Debian 11. So I have followed the XWiki documentation meant for a Debian install (with PostgreSQL).

The rights and permissions are the right ones, I believe. Our VM installs are done with a standard procedure.

I have posted this wish, idea, because I had not found what you are showing, a field meant to retrieve the export from within the server.

However now I can see it. There are even 2 files stored.

What had brought confusion is that the list of fields in the Advanced Setup is very large, and while reading the documentation on the blog, Confluence to XWiki migration guide: from export to customization - XWiki I was in fact expecting a dedicated section, with very few choices, such as:

  • import from your local computer
  • import from an existing archive in your server
  • import from a direct URL

Maybe if you could add a screenshot showing it is a very long list, or just saying something such as, "In the Advanced Setup option you will find a long list of options… "

Also the updatedb and locate commands in the Linux installation is supposed to search everywhere (except if some places are pruned in the mlocate configuration file).

#7

I checked what I have seen in the backoffice, and the 2 files presented in the field “The Source” (file:/var/lib/xwiki/data/store/file/xwiki/0//f.zip and another one) aren’t stored in the server : the command find does not find any f.zip file.

The ownership of the files are the following:
The directory var/lib/xwiki belongs to (user:group) root:root, all files and directories under var/lib/xwiki belong to tomcat:tomcat
should the directory xwiki under /var/lib also belong to tomcat:tomcat for example ?

Or is there something else I am missing?

#8

Note that there is a trick with the standard tomcat9 package on recent debians: it needs to be explicitly told what it’s allowed to access (for example, the xwiki Debian package automatically inject a /etc/systemd/system/tomcat9.service.d/xwiki-tomcat9-systemd.conf file in charge of adding /var/lib/xwiki/data to the allowed folders).

Maybe that’s your problem here ?

#9

Hi @tmortagne thanks for your help.
I have indeed found it: /etc/systemd/system/tomcat9.service.d/xwiki-tomcat9-systemd.conf and it contains:

[Service]
# Need to allow Tomcat9 to read and write in XWiki permanent directory
ReadWritePaths=/var/lib/xwiki/data

This is the default as I haven’t been there at any time so far : isn’t the content of this configuration file alright?

#10

I was not suggesting that this configuration was wrong (XWiki would not work if this configuration is wrong), I just indicated it as an example.

What I mean is that maybe you are trying to access a file that Tomcat is not allowed to access, not because of rights but because it’s not in an explicitly allowed folder.

#11

Hello,
Does it help if I provide rights and permissions I find here?

/var/lib/xwiki/data
drwxr-xr-x 8 tomcat tomcat 4096 May 25 09:59 data

under this data directory, all directories are: drwxr-x--- tomcat tomcat
So, user is rwx, group is r-x, read and access, other no rights. And it is the same scheme in all the sub-directories, I checked. Here is a sample in the xwiki data store file… directory:

/var/lib/xwiki/data/store/file/xwiki# ls -l
total 64
drwxr-x--- 14 tomcat tomcat 4096 May 26 08:14 0
drwxr-x--- 15 tomcat tomcat 4096 May 26 08:14 1
drwxr-x--- 11 tomcat tomcat 4096 May 26 08:14 2
drwxr-x--- 13 tomcat tomcat 4096 May 26 08:13 3
drwxr-x---  9 tomcat tomcat 4096 May 26 08:13 4
drwxr-x--- 11 tomcat tomcat 4096 May 26 08:14 5
drwxr-x--- 10 tomcat tomcat 4096 May 26 08:13 6
drwxr-x--- 11 tomcat tomcat 4096 May 26 09:25 7
drwxr-x--- 14 tomcat tomcat 4096 May 26 08:14 8
drwxr-x--- 11 tomcat tomcat 4096 May 26 09:25 9
drwxr-x--- 11 tomcat tomcat 4096 May 26 09:26 a
drwxr-x---  7 tomcat tomcat 4096 May 26 08:14 b
drwxr-x--- 11 tomcat tomcat 4096 May 25 19:09 c
drwxr-x--- 15 tomcat tomcat 4096 May 26 08:13 d
drwxr-x---  9 tomcat tomcat 4096 May 26 09:25 e
drwxr-x---  8 tomcat tomcat 4096 May 26 08:13 f

So why wouldn’t Tomcat be allowed here?

#12

Again, I’m not talking about /var/lib/xwiki/data. I’m only indicating that if you try to access a file somewhere else, you cannot without telling Tomcat this location is OK.

#13

Oh I get it now. So would it be possible to add a * as a joker at the end of the ReadWritePaths=/var/lib/xwiki/data line? something such as:
ReadWritePaths=/var/lib/xwiki/data* ?
Or maybe with a bash or shell script meant to find any and every file and directory and make it read/write for Tomcat?
Has this already been adressed in that manner before?

#14

Everything in that folder is already accessible, including sub folders, XWiki would not work otherwise. What is not accessible is what is elsewhere on that system.

#15

Then why can’t I find the archive I uploaded using the Confluence Migrator extension, once the extension UI shows the upload complete, from within my terminal/console logged with SSH?