Managing configuration changes when upgrading xwiki (using docker)

configuration
upgrade
docker

#1

Dear all,

this thread has the intent to share my experiences and ideas so far, and maybe discuss a better approach (see at the end), and improve xwiki documentation, to maybe help others. I’ve spent quite some time on this topic lately (in total probably 50 hours already)


How do you upgrade xwiki without losing configuration changes?_ (especially when using docker)

Your Guide in Upgrading XWiki (in Docker) says:

Note that your current XWiki configuration files (xwiki.cfg, xwiki.properties and hibernate.cfg.xml) will be preserved.

This is somewhat in contrast to Upgrading the webapp:

The general strategy is to download the new WAR distribution and to replace your current installed WAR with it. […] In addition you would override some XWiki configuration files located in WEB-INF (xwiki.properties and xwiki.cfg to name just two).

(I’ve think I’ve read somewhere about merging the two main configuration files, but can’t find that text atm.)

So basically, when upgrading, you must to merge the existing xwiki.cfg and xwiki.properties, because there may be additions, changes or deletions, using a tool like e.g. WinMerge.

Issue #1: The docker upgrade path violates this, because the docker-entrypoint.sh will use the existing old config files in the permanent data docker volume and copy them back over to the new configuration files within the container, essentially losing the new files, because you can’t access them beforehand.

Solution: Start a new docker container with the new xwiki image, so it’s config files are copied into a new docker volume (or access this container with docker exec). You can then merge these files with your existing one. You could manually download the WAR file and unzip it’s contents, BUT for docker the hibernate.cfg.xml is here on github so you have to compare with there…


Issue #2: How do you do all that, without loosing track, running on different systems, containers, docker-compose git repositories and whatnot?

My approach: Using git(hub). We have a in-house github repository where I’ve committed all files where we had to do changes. These are on the master branch, and tagged with the corresponding xwiki version.

When doing a version upgrade, I create a new branch e.g. 10.7-test , get all config files of the “vanilla” docker container and paste them in the repository. Then, using e.g. SourceTree, I review all changes in all files, and use Stage hunk or Discard hunks (or line) until everything is merged, and commit and push this to branch 10.7-test

In a xwiki test environment, similar to production, the xwiki data volume contains the git repository. There i checkout branch 10.7-test, and proceed with stopping the xwiki container and recreating it with the new 10.7 docker image. When it starts, it is now using the already merged configurations.
If all function tests with the new version are satisfactory, I merge 10.7-test into master, tag it with version 10.7 and push.

On the xwiki production server, in the xwiki permanent data docker volume, I pull from the master branch, and repeat the other upgrade steps like on the test environment.

THAT is quite an undertaking, considering we’d like to upgrade more frequently in the future.
Currently I’m managing this, but I want to delegate this to our IT departement (god help me)

If you have any suggestions, tips, etc. on this, it would be greatly appreciated.
Maybe I’m overcomplicating things :slight_smile:

I’m currently preparing to switch from xwiki 8.4.3 WAR on tomcat on windows to xwiki 10.8 tomcat-postgres on docker on linux CentOS. It’s a complete new installation where all content will be imported via XAR Export/Import. Wish me luck.

Thank you
BR
Mario


#2

Yes you’re correct. However in practice (i.e. 99% of the time) there’s nothing to merge and we have defaults for all new parameters we add. So the only thing to merge is if we update some old values (the main one that comes to mind is the plugin one).

What I’d propose here is to update script that does the copy and to copy also the original files and/or provide a diff or even better do the merge as we do it in the debian distribution.

Note that there are other files that could also need to be merged: web.xml if you modified it, clustering config (but this is not supported yet in the docker setup OOB), logging configuration, etc.

Would be great if you could raise a jira issue for this at https://jira.xwiki.org/projects/XDOCKER/summary

And

I’ll need to read that slowly and process it. Please note that the docker distributions are quite new and have been done by me and:

  • I’ve learning docker
  • I’ve never put any application in production using docker

Thus you probably know a lot more than I do on the topic.

I’m actually expecting contributions from users on this! Pull Requests on https://github.com/xwiki-contrib/docker-xwiki would be awesome :slight_smile:

Please raise all issues on https://jira.xwiki.org/projects/XDOCKER/summary

Thanks.
It’s great that you’re starting this discussion! I’d really love that we have production-ready docker images for XWiki so anything that improves it is good.


#3

@vmassol: I know you are mostly interested in the docker part, but would you mind broadening up the topic to config handling in general (in case of updates)?

Because I feel that I merge much more than in only 1% of the updates.


#4

I develop xwiki but I don’t put it in production myself so I could be wrong. Could you elaborate on what you have to merge? Thanks


#5

So I looked into the last two actual updates 9.11.4->10.4->10.6.1 and every update had at least one merge, or to be more precise: something showed in the diffs that was not configured by me.

9.11.4->10.4:
Store changed to file from hibernate, affected: xwiki.cfg
Additional unicode settings in hibernate.xml for mysql connection: hibernate.xml

10.4->10.6.1:
xar protection and autowatch settings: xwiki.properties

Currently preparing the update 10.6.1->10.7
extension.oldflavors: xwiki.properties

So currently it’s a 3/3 merges/upgrade in my case.
Although I don’t have any “proof” any more but I’m pretty sure that I had to merge quite some stuff while updating from 8.x->9.x


#6

You never have anything to merge in xwiki.properties normally. There are always default values.


#7

Except you changed the rendering transformations, like I did to use numbered headings.


#8

hmm didn’t know about this one. This violates the “There are always default values.” rule we have which is not good. @tmortagne any idea? Couldn’t we have those asa default?


#9

I don’t recall any change from our side. What did we change? :slight_smile:

AFAIK it’s still:

# rendering.transformations = macro, icon


#10

Sorry bad phrasing. I changed the rendering transformation settings to include numbered headings transformation.


#11

Sure it would need to be merged if/when we change the default value for transformations and only if you wanted to add any new transformation that we would add (since you have your own changes).


#12

OK I see we have different approaches to this: My approach is to merge xwiki.properties because I have local changes AND I want to have the current list of default values.
But I agree with you that this is technically not necessary because nothing changes if I don’t merge, because these values are not active and just document default values.
Except when there is the exception like in the 10.6->10.7 upgrade. Maybe I’m too long in this industry to just not trust these claims unverified. No offense!


#13

hehe we all do that (triple check everything). Now this is not a conflict. It’s a simple merge and can be automated, which is what the debian distribution does and what you can do manually too (we have some instructions on myxwiki.org but it’s dead ATM so can’t fetch it).


#14

Example:

sudo mv xwiki.new/WEB-INF/web.xml xwiki.new/WEB-INF/web.xml.ori; sudo sdiff xwiki/WEB-INF/web.xml xwiki.new/WEB-INF/web.xml.ori -B -W -E -o xwiki.new/WEB-INF/web.xml
sudo mv xwiki.new/WEB-INF/xwiki.cfg xwiki.new/WEB-INF/xwiki.cfg.ori; sudo sdiff xwiki/WEB-INF/xwiki.cfg xwiki.new/WEB-INF/xwiki.cfg.ori -B -W -E -o xwiki.new/WEB-INF/xwiki.cfg
sudo mv xwiki.new/WEB-INF/xwiki.properties xwiki.new/WEB-INF/xwiki.properties.ori; sudo sdiff xwiki/WEB-INF/xwiki.properties xwiki.new/WEB-INF/xwiki.properties.ori -B -W -E -o xwiki.new/WEB-INF/xwiki.properties
sudo mv xwiki.new/WEB-INF/struts-config.xml xwiki.new/WEB-INF/struts-config.xml.ori; sudo sdiff xwiki/WEB-INF/struts-config.xml xwiki.new/WEB-INF/struts-config.xml.ori -B -W -E -o xwiki.new/WEB-INF/struts-config.xml
sudo mv xwiki.new/WEB-INF/classes/logback.xml xwiki.new/WEB-INF/classes/logback.xml.ori; sudo sdiff xwiki/WEB-INF/classes/logback.xml xwiki.new/WEB-INF/classes/logback.xml.ori -B -W -E -o xwiki.new/WEB-INF/classes/logback.xml

#15

Thank you so much for this example. I knew it would backfire that I postponed to learn about diff again and again. :sweat_smile:
But why not go a step further and allow for a $conrfigfile_local.$ext that only holds the really changed stuff and is applied on top of the defaults?
That should be really simple for the .cfg files and somehow doable for the xml files like hibernate.xml?
If these _local files are not provided by xwiki distributions we could more or less blindly update xwiki and adjust changed defaults (if they have been overriden) if we approve the changes.
And even more it would be easy to mount these files into docker containers.
What do you think?


#16

Yes that would be a good idea (actually this is something I’ve been wanting for a long time, see for example https://markmail.org/message/6cvm5hocvtbqtgp6 - It’s been discussed again last year I think too, would need to spend some time finding the reference to the discussion).

What would be important though is that it wouldn’t support the ability to unset a property (unless we introduce a special syntax)/ We would also need to handle the strategy for overwriting a map (for example the extension.oldflavors you mentioned - will it add a new item or remove it).

Have a good weekend.


#17

Why not start simple (don’t care about unset and maps) and adjust on the go? These two topics are likely to not have right answers. Whatever you decide to implement there will always be the case where you wished you had done it differently. And while this feature develops there is always the fallback to the “old way”. Maybe starting simple would help to not postpone it another eight years :wink:.


#18

Sure. It has not been postponed because of this limitation :wink: It has not been done because nobody worked on it and no PR was sent either…


#19

Actually I think the current docker image is already quite production ready, so you did a good job :+1:
Currently, my time is more limited again, but I’ve planned to make further pull-requests if possible.

I need to process your discussion with @rbr , I’m quite new to linux all together and acutally started to learn about docker a few months ago.


Another issue regarding upgrading xwiki (in docker):

Issue #3: Solr data ist not deleted when upgrading xwiki docker image. Upgrading the webapp says that:

Best is to not copy the solr/ subdirectory since the SOLR schema could have changed and it’s best to let XWiki reindex using the new schema

But it resides in /usr/local/xwiki/data/solr within the xwiki data volume.
Suggestion: Add an “upgrade” condition in docker-entrypoint.sh besides first_start and other_starts that deletes the SOLR data? Detecting an upgrade could be tricky though…

Will add this at the first post in the thread.


Things I’m currently missing in the docker image/compose template for making it “enterprise-production-ready”:

  1. LDAP Authentication: Very easy, just set xwiki.authentication.authclass=org.xwiki.contrib.ldap.XWikiLDAPAuthServiceImpl
    and then install the LDAP application extension and set it up via Administration GUI. This can be done via ENV variable and the existing docker-entrypoint.sh script I guess?
    Optionally, also add xwiki.authentication.ldap.trylocal=1 (very important for us)

  2. TLS/SSL configuration for tomcat: This is actually my last big TODO to check for docker, which I need our IT for. I hope it can be done easily.

  3. Tomcat java opts variable: -Xmx1024m ist not enough for our installation, so i have to edit setenv.sh

I will add these three at the top in the first post of the thread.


Other minor things we did to the default config (looking at my git diff at the moment):

  • Setting attachment store to file -> Fixed in current xwiki (database storage was not usable for us)
  • xwiki.plugin.watchlist.automaticwatch=none because our normal users don’t want that
  • xwiki.plugin.image.defaultQuality = 0.8 because our faces were ugly in our avatars :smiley:
  • openoffice.homePath and autostart -> openoffice is already included in the docker image, LibreOffice did get better results when importing M$ office though
  • rendering.imageDimensionsIncludedInImageURL = false because the default behaviour is not useful
  • Adding markdown to xwiki.rendering.syntaxes

Of course, you could but everything into environment variables.

Additionally, maybe Performance parameters for tuning? We currently have sometimes performance hickups on our running 8.4.3 tomcat postgres on windows server. I’ve tuned some values but still.

But we never installed the optional database indexes, maybe thats the reason.

I plan to to this also in den docker environment. Namely: postgresql performance parameters, the missing DB indexes, xwiki performance parameters. e.g. the Admin Tools Application tells me that

Your cache configuration is too low for a production usage. Your cache is set to 100 documents, it should be 1000 or more

Shouldn’t that be 500 per default?

   149  #-# Maximum number of documents to keep in the cache.
   150  #-# The default is 500.
   151  # xwiki.store.cache.capacity=500

Maybe there is more to be done. I’m playing around with a standalone SOLR container, but not sure if that helps anything, but it sounds reasonable to keep it seperate from the xwiki server.


If we can’t get everything we need into the official image, we can still fork the git repository and roll our own. Not everybody needs all that stuff. Thats the sex appeal of automated build processes.

BR


#20

I found in general:
If you can get away with it don’t terminate SSL/TLS on tomcat and use a reverse proxy ( outside of the web docker) instead. Creating java keystores is normally harder than just throwing the plain text files in Apache/nginx, especially with enterprise signed certs. If you want SSL on the default port (443) you need to run tomcat as root or take other additional measures like jsvc or firewall rules. Even if this is not a problem with docker as you can connect 443:8443 (or whatever the SSL port is) you still have the issue to get the jks in the container and change the config (don’t know if this is prepared).

LDAP Authentication and a lot of the other custom settings would be really easy if you could mount _local versions of the config files into the container as discussed above.