Storage questions (Database and permanent dir) and other migration questions

Hello! Some questions for the Xwiki pros!

I just installed XWiki using the Debian method (so much easier than manual or WAR deployment) on a brand new server to move away from an old server.

When I configured the xwiki files (xwiki.cfg, xwiki.properties), and specified the database parameters in “hibernate.cfg.xml” I was surprised that Xwiki came up with ALL of my data. At this point, the datastore (permanent directory) on the new server was empty!

The database I used for the new Xwiki server is a CLONE of the old database (clone done in MySQL CLI).

Then I copied (rsync -av) the permanent directory from the old server to the new one and restarted tomcat.

Because Xwiki on the new server came up with all my data (while the permanent directory was empty), I assumed that it relies only on database for storage needs (at least in my case)…

Strange thing is, on the old server, the permanent directory (as specified in xwiki.properties) contains the following data so clearly the old xwiki stored data in it…

drwxr-xr-x 3 apache apache 3 Jun 29  2020 1593450338908-0
drwxr-xr-x 3 apache apache 3 Jun 30  2020 1593545080442-0
drwxr-xr-x 4 apache apache 4 Jun 30  2020 1593546464788-0
drwxr-xr-x 4 apache apache 4 Jun 30  2020 1593546464799-0
drwxr-xr-x 4 apache apache 4 Jun 30  2020 1593546909871-0
drwxr-xr-x 3 apache apache 3 Jun 30  2020 1593546911418-0
drwxr-xr-x 3 apache apache 3 Jul 13  2020 cache
drwxr-xr-x 4 apache apache 4 Jul 13  2020 extension
drwxr-xr-x 3 apache apache 3 Jul 13  2020 jobs
drwxr-xr-x 5 apache apache 6 Jul 13  2020 solr
drwxr-xr-x 2 apache apache 2 Aug 17  2020 storage

Question 1: Am I correct to say that xwiki stores data in the mariadb DB instead of the permanent dir?

Question 2: Is it generally better to use disk storage or database storage? I have in mind data integrity and performance. I do daily rsnapshot backups of disk storage as well as mysqldump backups to a remote server.

Question 3: If disk storage is preferable (permanent directory), how can I ensure that xwiki is really using the permanent dir to store wiki data? The old server was configured with a permanent dir, but with the above experience, I am no longer certain of anything… Also, I just checked, and the contents of permanent dir on new server have the exact same timestamps as the data on the old server so clearly xwiki is not using the permament dir…

These are the parameters on the new server

/etc/xwiki/xwiki.cfg (truncated commented out lines to make post smaller)

#---------------------------------------
# Storage
#

#-# Role hints that differentiate implementations of the various storage components. To add a new implementation for
#-# one of the storages, implement the appropriate interface and declare it in a components.xml file (using a role-hint
#-# other than 'default') and put its hint here.

[...]

#-# [Since 1.6M1]
#-# Force the database name for the main wiki.
xwiki.db=xwikidb

#-# [Since 1.6M1]
#-# Add a prefix to all databases names of each wiki.
# xwiki.db.prefix=

/etc/xwiki/xwiki.properties

#-------------------------------------------------------------------------------------
# Environment
#-------------------------------------------------------------------------------------

#-# [Since 3.5M1, replaces the container.persistentDirectory property]
#-# The directory used to store persistent data (data that should persist across server restarts). This is an
#-# important directory containing important data and thus it should never be deleted (it should be backed-up along
#-# with the database).
#-# For example this is where the Extension Manager stores downloaded extensions if the extension.localRepository
#-# property isn't configured.
#-#
#-# You can set:
#-# * an absolute path (recommended)
#-# * a relative path (not recommended at all)but in this case the directory will be relative to where the XWiki server
#-#   is started and thus the user under which XWiki is started will need write permissions for the current directory
#-#
#-# Note if the system property xwiki.data.dir is set then this property is not used.
#-# If neither the system property nor this configuration value here are set then the Servlet container's temporary
#-# directory is used; This is absolutely not recommended since that directory could be wiped out at any time and you
#-# should specify a value.
environment.permanentDirectory = /mnt/data/xwiki-data/

Last question: In addition to configuring hibernate.cfg.xml to connect to the remote mariadb server, I had to specify the database name in xwiki.cfg otherwise tomcat was complaining that the defaut database (xwiki) could not be reached on localhost (access denied for user ‘xwiki’@‘localhost’). Of course this happened as the database server is remote.

Why were the settings of hibernate.cfg.xml not sufficient? On the old server I have not specified anything in xwiki.cfg…

xwiki.cfg

#-# [Since 1.6M1]
#-# Force the database name for the main wiki.
xwiki.db=xwikidb

hibernate.cfg.xml

< property name=“hibernate.connection.url”>jdbc:mariadb://database.localdomain/xwikidb?useSSL=false
< property name=“hibernate.connection.username”>xwikidb
< property name=“hibernate.connection.password”>XXXXXXXXXXXXXXXXXXx
< property name=“hibernate.connection.driver_class”>org.mariadb.jdbc.Driver
< property name=“hibernate.dbcp.poolPreparedStatements”>true
< property name=“hibernate.dbcp.maxOpenPreparedStatements”>20

Okay a bit of follow-up because I still need help, and to help others who may have the same questions…

Question 1: Solved (not as I wanted, but kinda). Xwiki uses the SQL database to store wiki contents. If this is accurate, I dont understand why backing up the permanent directory is necessary but I did anyways. In my case, I cannot even accurately confirm which permanent dir xwiki uses and everything works well so far. Moreover, the time stamps of the contents of the perm dir specified in xwiki.properties have not changed in 6 months…

Question 2: No longer relevant since it uses SQL DB.

Question 3: To confirm that xwiki uses the permanent dir specified in xwiki.properties, I followed the instructions of the Administration Wiki. I am using a DEB installation, so according to the Admin wiki, first option is to look at tomcat’s logs that are in “/var/log/tomcat9/”. In this folder I see:

catalina.2021-11-07.log.gz
catalina.2021-11-08.log.gz
catalina.2021-11-29.log
localhost.2021-11-07.log.gz
localhost.2021-11-08.log.gz
localhost.2021-11-11.log.gz
localhost.2021-11-29.log
localhost_access_log.2021-11-07.txt.gz
localhost_access_log.2021-11-08.txt.gz
localhost_access_log.2021-11-11.txt.gz
localhost_access_log.2021-11-12.txt.gz
localhost_access_log.2021-11-27.txt.gz
localhost_access_log.2021-11-29.txt

Examining “catalina.2021-11-29.log” (the latest log file since restarting tomcat) I see no mention of permanent directory

Tomcat shutdown, entire log file emptied then tomcat started:
29-Nov-2021 14:37:55.181 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Server version name: Apache Tomcat/9.0.31 (Ubuntu)
29-Nov-2021 14:37:55.194 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Server built: Oct 20 2020 12:27:39 UTC
29-Nov-2021 14:37:55.195 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Server version number: 9.0.31.0
29-Nov-2021 14:37:55.195 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log OS Name: Linux
29-Nov-2021 14:37:55.195 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log OS Version: 5.4.0-89-generic
29-Nov-2021 14:37:55.196 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Architecture: amd64
29-Nov-2021 14:37:55.196 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Java Home: /usr/lib/jvm/java-11-openjdk-amd64
29-Nov-2021 14:37:55.196 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log JVM Version: 11.0.11+9-Ubuntu-0ubuntu2.20.04
29-Nov-2021 14:37:55.197 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log JVM Vendor: Ubuntu
29-Nov-2021 14:37:55.198 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log CATALINA_BASE: /var/lib/tomcat9
29-Nov-2021 14:37:55.198 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log CATALINA_HOME: /usr/share/tomcat9
29-Nov-2021 14:37:55.227 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: --add-opens=java.base/java.lang=ALL-UNNAMED
29-Nov-2021 14:37:55.228 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: --add-opens=java.base/java.io=ALL-UNNAMED
29-Nov-2021 14:37:55.228 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: --add-opens=java.rmi/sun.rmi.transport=ALL-UNNAMED
29-Nov-2021 14:37:55.229 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Djava.util.logging.config.file=/var/lib/tomcat9/conf/logging.properties
29-Nov-2021 14:37:55.230 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager
29-Nov-2021 14:37:55.230 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Djava.awt.headless=true
29-Nov-2021 14:37:55.231 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Xmx4096m
29-Nov-2021 14:37:55.231 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Djava.security.egd=file:/dev/./urandom
29-Nov-2021 14:37:55.232 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Djdk.tls.ephemeralDHKeySize=2048
29-Nov-2021 14:37:55.232 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Djava.protocol.handler.pkgs=org.apache.catalina.webresources
29-Nov-2021 14:37:55.233 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Dorg.apache.catalina.security.SecurityListener.UMASK=0027
29-Nov-2021 14:37:55.233 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Dignore.endorsed.dirs=
29-Nov-2021 14:37:55.234 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Dcatalina.base=/var/lib/tomcat9
29-Nov-2021 14:37:55.234 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Dcatalina.home=/usr/share/tomcat9
29-Nov-2021 14:37:55.234 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Djava.io.tmpdir=/tmp
29-Nov-2021 14:37:55.239 INFO [main] org.apache.catalina.core.AprLifecycleListener.lifecycleEvent Loaded APR based Apache Tomcat Native library [1.2.23] using APR version [1.6.5].
29-Nov-2021 14:37:55.240 INFO [main] org.apache.catalina.core.AprLifecycleListener.lifecycleEvent APR capabilities: IPv6 [true], sendfile [true], accept filters [false], random [true].
29-Nov-2021 14:37:55.240 INFO [main] org.apache.catalina.core.AprLifecycleListener.lifecycleEvent APR/OpenSSL configuration: useAprConnector [false], useOpenSSL [true]
29-Nov-2021 14:37:55.246 INFO [main] org.apache.catalina.core.AprLifecycleListener.initializeSSL OpenSSL successfully initialized [OpenSSL 1.1.1f 31 Mar 2020]
29-Nov-2021 14:37:56.244 INFO [main] org.apache.coyote.AbstractProtocol.init Initializing ProtocolHandler [“http-nio-8080”]
29-Nov-2021 14:37:56.307 INFO [main] org.apache.catalina.startup.Catalina.load Server initialization in [1,475] milliseconds
29-Nov-2021 14:37:56.407 INFO [main] org.apache.catalina.core.StandardService.startInternal Starting service [Catalina]
29-Nov-2021 14:37:56.407 INFO [main] org.apache.catalina.core.StandardEngine.startInternal Starting Servlet engine: [Apache Tomcat/9.0.31 (Ubuntu)]
29-Nov-2021 14:37:56.418 INFO [main] org.apache.catalina.startup.HostConfig.deployDescriptor Deploying deployment descriptor [/etc/tomcat9/Catalina/localhost/xwiki.xml]
29-Nov-2021 14:37:56.438 WARNING [main] org.apache.catalina.startup.HostConfig.deployDescriptor The path attribute with value [/xwiki] in deployment descriptor [/etc/tomcat9/Catalina/localhost/xwiki.xml] has been ignored
29-Nov-2021 14:38:33.176 INFO [main] org.apache.catalina.startup.HostConfig.deployDescriptor Deployment of deployment descriptor [/etc/tomcat9/Catalina/localhost/xwiki.xml] has finished in [36,758] ms
29-Nov-2021 14:38:33.178 INFO [main] org.apache.catalina.startup.HostConfig.deployDirectory Deploying web application directory [/var/lib/tomcat9/webapps/ROOT]
29-Nov-2021 14:38:34.727 INFO [main] org.apache.jasper.servlet.TldScanner.scanJars At least one JAR was scanned for TLDs yet contained no TLDs. Enable debug logging for this logger for a complete list of JARs that were scanned but no TLDs were found in them. Skipping unneeded JARs during scanning can improve startup time and JSP compilation time.
29-Nov-2021 14:38:34.748 INFO [main] org.apache.catalina.startup.HostConfig.deployDirectory Deployment of web application directory [/var/lib/tomcat9/webapps/ROOT] has finished in [1,569] ms
29-Nov-2021 14:38:34.756 INFO [main] org.apache.coyote.AbstractProtocol.start Starting ProtocolHandler [“http-nio-8080”]
29-Nov-2021 14:38:34.771 INFO [main] org.apache.catalina.startup.Catalina.start Server startup in [38,462] milliseconds

The other log of interest (localhost.2021-11-29.log) also has nothing in it about permanent dir.

Finally, I tried the second option to get the permanent dir, create a wiki page with the following in it:

{{velocity}}
$services.component.getInstance(‘org.xwiki.environment.Environment’).getPermanentDirectory()
{{/velocity}}

Do I need to do anything other than copy-paste this snippet in a new page because it is not “rendering” the permanent directory address but its simply displaying the snippet as text… I am performing this as ADMIN, which has programming rights.

Any info appreciated!

  1. Markup is stored in the DB, attachments depends on the configuration. Older versions defaulted to DB, newer to file system. See here: https://www.xwiki.org/xwiki/bin/view/Documentation/AdminGuide/Configuration/#HPermanentDirectory

  2. Not into debian, but there should be a catalina.out or so without a date. This is the current log. But, the data directory is only logged during startup of xWiki. Or you can try:

grep permanent catalina*

HTH

Hello,

According to the documentation (https://www.xwiki.org/xwiki/bin/view/Documentation/AdminGuide/Attachments), versions < 10.5 attachments were entirely stored in DB, after that, in filesystem (files and attachments) + DB (metadata and other wiki stuff, parameters, etc).

I run “XWiki Debian 13.9” so I would imagine xwiki is storing some stuff in filesystem.

Like I said, the permanent directory is not being used by xwiki, so it is storing somewhere else. This is what I am trying to fix.

I already looked at the link you gave me, this is where I got the instructions how to configure and test for the permanent directory. Unfortunately, neither of the logs method or the programming method (with the velocity code snippet) works.

The admin user is part of the XWikiAdminGroup, and this group has programming privileges.

Moreover, I tried to find a log named “catalina.out” (without a date appended to its filename) and couldnt find any.

For information, this is a STOCK install performed entirely following the instructions of https://www.xwiki.org/xwiki/bin/view/Documentation/AdminGuide/Installation/InstallationViaAPT/

EDIT: I just browsed the entire filesystem trying to find where xwiki is storing data (outside of the DB), and there are tons of folders and files in this location:

/var/cache/tomcat9/Catalina/localhost/xwiki/store

Example:

/var/cache/tomcat9/Catalina/localhost/xwiki/store/file
/var/cache/tomcat9/Catalina/localhost/xwiki/store/file/xwiki
/var/cache/tomcat9/Catalina/localhost/xwiki/store/file/xwiki/8
/var/cache/tomcat9/Catalina/localhost/xwiki/store/file/xwiki/8/3
/var/cache/tomcat9/Catalina/localhost/xwiki/store/file/xwiki/8/3/927acf911df7547cf26345b2c3e89c
/var/cache/tomcat9/Catalina/localhost/xwiki/store/file/xwiki/8/3/927acf911df7547cf26345b2c3e89c/attachments
/var/cache/tomcat9/Catalina/localhost/xwiki/store/file/xwiki/8/3/927acf911df7547cf26345b2c3e89c/attachments/9
/var/cache/tomcat9/Catalina/localhost/xwiki/store/file/xwiki/8/3/927acf911df7547cf26345b2c3e89c/attachments/9/2
/var/cache/tomcat9/Catalina/localhost/xwiki/store/file/xwiki/8/3/927acf911df7547cf26345b2c3e89c/attachments/9/2/0bc685fa0da28168319c0126def81b
/var/cache/tomcat9/Catalina/localhost/xwiki/store/file/xwiki/8/3/927acf911df7547cf26345b2c3e89c/attachments/9/2/0bc685fa0da28168319c0126def81b/f.png
/var/cache/tomcat9/Catalina/localhost/xwiki/store/file/xwiki/8/3/927acf911df7547cf26345b2c3e89c/attachments/9/2/0bc685fa0da28168319c0126def81b/~METADATA.xml
/var/cache/tomcat9/Catalina/localhost/xwiki/store/file/xwiki/8/3/927acf911df7547cf26345b2c3e89c/attachments/9/2/0bc685fa0da28168319c0126def81b/fv1.2.png

Is this the file system storage I am searching for? if so, why is xwiki not storing in the permanent directory specified in xwiki.properties? (FYI, no special argument are passed to the tomcat startup file)

Well, the above example looks like something is really off, as the “store/file/xwiki” is part of the cache structure, which I can only assume should not be the case. Again, I have no experience with Debian setup, so I can not confirm or deny. Maybe someone else can that is using Debian…

So my last attempt:
Stop tomcat if you can (e.g. no one is using xWiki)
Start tomcat again
Access the xWiki URL
Check the catalina.??? you deem the current one (verify the dates in the log) for

iPropertiesConfigurationSource
ServletEnvironment

Again, make sure these are the current logs.
HTH

Hello,

Since I grabbed the contents of “/var/cache/” and you’re saying that the folder structure is part of xwiki cache, I believe I am indeed looking at the cache and therefore at the wrong location. So lets forget about this location…

I emptied the logs folder, restarted tomcat, and grepped the catalina.out log file and NOTHING about permanent dir, “iPropertiesConfigurationSource” or even “ServletEnvironment”… This is real strange.

Someone from the development team who packaged the DEB installation needs to jump in because it seems something is off…

Using inotify-tools, I did a filesystem monitor while uploading attachments to see where Xwiki would store data on the FS (if it is)…

Only the following were changed during this exercise:

Setting up watches. Beware: since -r was given, this may take a while!
Watches established.
/var/log/nginx/ MODIFY xwiki-access.log
/var/log/nginx/ MODIFY xwiki-access.log
/var/log/nginx/ MODIFY xwiki-access.log
/var/log/nginx/ MODIFY xwiki-access.log
/var/log/tomcat9/ MODIFY localhost_access_log.2021-12-04.txt
/var/log/nginx/ MODIFY xwiki-access.log
/var/log/nginx/ MODIFY xwiki-access.log
/var/log/nginx/ MODIFY xwiki-access.log
/var/log/nginx/ MODIFY xwiki-access.log
/var/log/nginx/ MODIFY xwiki-access.log
/var/log/tomcat9/ MODIFY localhost_access_log.2021-12-04.txt
/var/log/nginx/ MODIFY xwiki-access.log
/var/log/nginx/ MODIFY xwiki-access.log
/var/log/tomcat9/ MODIFY localhost_access_log.2021-12-04.txt
/var/log/nginx/ MODIFY xwiki-access.log
/var/log/nginx/ MODIFY xwiki-access.log
/var/log/nginx/ MODIFY xwiki-access.log
/var/log/ MODIFY auth.log
/var/log/ MODIFY syslog
/var/log/ MODIFY auth.log
/var/log/journal/16571139d620424caab5b66bb5aa8812/ MODIFY system.journal
/var/log/tomcat9/ MODIFY localhost_access_log.2021-12-04.txt
/var/log/nginx/ MODIFY xwiki-access.log
/var/log/nginx/ MODIFY xwiki-access.log
/var/log/nginx/ MODIFY xwiki-access.log
/var/log/nginx/ MODIFY xwiki-access.log
/var/log/nginx/ MODIFY xwiki-access.log
/var/log/nginx/ MODIFY xwiki-access.log
/var/log/nginx/ MODIFY xwiki-access.log
/var/log/nginx/ MODIFY xwiki-access.log
/var/log/nginx/ MODIFY xwiki-access.log
/var/log/nginx/ MODIFY xwiki-access.log
/var/log/tomcat9/ MODIFY localhost_access_log.2021-12-04.txt

So that pretty much answers the question if Xwiki is using the DB or FS to store attachments, and the answer is DB.

That explains why I backed only the DB from my previous install and everything came back after migration without backing up the so called “permanent directory”…

It is very much possible that if the old xWiki instance was configured to use DB that all the attachments are still there … since they were part of the DB clone and xWiki knows that they are in DB (although you define file as new storage - the old attachments would need to be ported over to the file system using the Admin Module.)

BUT:
Just make 100% that you still use the DB as attachment storage (not recommended any more AFAIK), or make note where the new attachments are stored. Maybe Debian still defaults to DB, but I doubt. Did you copy the configuration from the old to the new server?

I emphasize on this as I once didn’t notice that the permanent directory was not writable by the user tomcat was running under and it fell back to default. Next time I upgrade (I usually delete everything and start from 0), I dropped all the attachments with it. As I only noticed this a few month later (test system), no backup was there any more. So I lost all the attachments since the last upgrade. Just to understand what is on stake …

HTH