Managing configuration changes when upgrading xwiki (using docker)

configuration
upgrade
docker
#3

@vmassol: I know you are mostly interested in the docker part, but would you mind broadening up the topic to config handling in general (in case of updates)?

Because I feel that I merge much more than in only 1% of the updates.

#4

I develop xwiki but I don’t put it in production myself so I could be wrong. Could you elaborate on what you have to merge? Thanks

#5

So I looked into the last two actual updates 9.11.4->10.4->10.6.1 and every update had at least one merge, or to be more precise: something showed in the diffs that was not configured by me.

9.11.4->10.4:
Store changed to file from hibernate, affected: xwiki.cfg
Additional unicode settings in hibernate.xml for mysql connection: hibernate.xml

10.4->10.6.1:
xar protection and autowatch settings: xwiki.properties

Currently preparing the update 10.6.1->10.7
extension.oldflavors: xwiki.properties

So currently it’s a 3/3 merges/upgrade in my case.
Although I don’t have any “proof” any more but I’m pretty sure that I had to merge quite some stuff while updating from 8.x->9.x

#6

You never have anything to merge in xwiki.properties normally. There are always default values.

#7

Except you changed the rendering transformations, like I did to use numbered headings.

#8

hmm didn’t know about this one. This violates the “There are always default values.” rule we have which is not good. @tmortagne any idea? Couldn’t we have those asa default?

#9

I don’t recall any change from our side. What did we change? :slight_smile:

AFAIK it’s still:

# rendering.transformations = macro, icon

#10

Sorry bad phrasing. I changed the rendering transformation settings to include numbered headings transformation.

#11

Sure it would need to be merged if/when we change the default value for transformations and only if you wanted to add any new transformation that we would add (since you have your own changes).

#12

OK I see we have different approaches to this: My approach is to merge xwiki.properties because I have local changes AND I want to have the current list of default values.
But I agree with you that this is technically not necessary because nothing changes if I don’t merge, because these values are not active and just document default values.
Except when there is the exception like in the 10.6->10.7 upgrade. Maybe I’m too long in this industry to just not trust these claims unverified. No offense!

#13

hehe we all do that (triple check everything). Now this is not a conflict. It’s a simple merge and can be automated, which is what the debian distribution does and what you can do manually too (we have some instructions on myxwiki.org but it’s dead ATM so can’t fetch it).

#14

Example:

sudo mv xwiki.new/WEB-INF/web.xml xwiki.new/WEB-INF/web.xml.ori; sudo sdiff xwiki/WEB-INF/web.xml xwiki.new/WEB-INF/web.xml.ori -B -W -E -o xwiki.new/WEB-INF/web.xml
sudo mv xwiki.new/WEB-INF/xwiki.cfg xwiki.new/WEB-INF/xwiki.cfg.ori; sudo sdiff xwiki/WEB-INF/xwiki.cfg xwiki.new/WEB-INF/xwiki.cfg.ori -B -W -E -o xwiki.new/WEB-INF/xwiki.cfg
sudo mv xwiki.new/WEB-INF/xwiki.properties xwiki.new/WEB-INF/xwiki.properties.ori; sudo sdiff xwiki/WEB-INF/xwiki.properties xwiki.new/WEB-INF/xwiki.properties.ori -B -W -E -o xwiki.new/WEB-INF/xwiki.properties
sudo mv xwiki.new/WEB-INF/struts-config.xml xwiki.new/WEB-INF/struts-config.xml.ori; sudo sdiff xwiki/WEB-INF/struts-config.xml xwiki.new/WEB-INF/struts-config.xml.ori -B -W -E -o xwiki.new/WEB-INF/struts-config.xml
sudo mv xwiki.new/WEB-INF/classes/logback.xml xwiki.new/WEB-INF/classes/logback.xml.ori; sudo sdiff xwiki/WEB-INF/classes/logback.xml xwiki.new/WEB-INF/classes/logback.xml.ori -B -W -E -o xwiki.new/WEB-INF/classes/logback.xml
#15

Thank you so much for this example. I knew it would backfire that I postponed to learn about diff again and again. :sweat_smile:
But why not go a step further and allow for a $conrfigfile_local.$ext that only holds the really changed stuff and is applied on top of the defaults?
That should be really simple for the .cfg files and somehow doable for the xml files like hibernate.xml?
If these _local files are not provided by xwiki distributions we could more or less blindly update xwiki and adjust changed defaults (if they have been overriden) if we approve the changes.
And even more it would be easy to mount these files into docker containers.
What do you think?

#16

Yes that would be a good idea (actually this is something I’ve been wanting for a long time, see for example https://markmail.org/message/6cvm5hocvtbqtgp6 - It’s been discussed again last year I think too, would need to spend some time finding the reference to the discussion).

What would be important though is that it wouldn’t support the ability to unset a property (unless we introduce a special syntax)/ We would also need to handle the strategy for overwriting a map (for example the extension.oldflavors you mentioned - will it add a new item or remove it).

Have a good weekend.

#17

Why not start simple (don’t care about unset and maps) and adjust on the go? These two topics are likely to not have right answers. Whatever you decide to implement there will always be the case where you wished you had done it differently. And while this feature develops there is always the fallback to the “old way”. Maybe starting simple would help to not postpone it another eight years :wink:.

#18

Sure. It has not been postponed because of this limitation :wink: It has not been done because nobody worked on it and no PR was sent either…

#19

Actually I think the current docker image is already quite production ready, so you did a good job :+1:
Currently, my time is more limited again, but I’ve planned to make further pull-requests if possible.

I need to process your discussion with @rbr , I’m quite new to linux all together and acutally started to learn about docker a few months ago.

Another issue regarding upgrading xwiki (in docker):

Issue #3: Solr data ist not deleted when upgrading xwiki docker image. Upgrading the webapp says that:

Best is to not copy the solr/ subdirectory since the SOLR schema could have changed and it’s best to let XWiki reindex using the new schema

But it resides in /usr/local/xwiki/data/solr within the xwiki data volume.
Suggestion: Add an “upgrade” condition in docker-entrypoint.sh besides first_start and other_starts that deletes the SOLR data? Detecting an upgrade could be tricky though…

Will add this at the first post in the thread.

Things I’m currently missing in the docker image/compose template for making it “enterprise-production-ready”:

  1. LDAP Authentication: Very easy, just set xwiki.authentication.authclass=org.xwiki.contrib.ldap.XWikiLDAPAuthServiceImpl
    and then install the LDAP application extension and set it up via Administration GUI. This can be done via ENV variable and the existing docker-entrypoint.sh script I guess?
    Optionally, also add xwiki.authentication.ldap.trylocal=1 (very important for us)

  2. TLS/SSL configuration for tomcat: This is actually my last big TODO to check for docker, which I need our IT for. I hope it can be done easily.

  3. Tomcat java opts variable: -Xmx1024m ist not enough for our installation, so i have to edit setenv.sh

I will add these three at the top in the first post of the thread.

Other minor things we did to the default config (looking at my git diff at the moment):

  • Setting attachment store to file -> Fixed in current xwiki (database storage was not usable for us)
  • xwiki.plugin.watchlist.automaticwatch=none because our normal users don’t want that
  • xwiki.plugin.image.defaultQuality = 0.8 because our faces were ugly in our avatars :smiley:
  • openoffice.homePath and autostart -> openoffice is already included in the docker image, LibreOffice did get better results when importing M$ office though
  • rendering.imageDimensionsIncludedInImageURL = false because the default behaviour is not useful
  • Adding markdown to xwiki.rendering.syntaxes

Of course, you could but everything into environment variables.

Additionally, maybe Performance parameters for tuning? We currently have sometimes performance hickups on our running 8.4.3 tomcat postgres on windows server. I’ve tuned some values but still.

But we never installed the optional database indexes, maybe thats the reason.

I plan to to this also in den docker environment. Namely: postgresql performance parameters, the missing DB indexes, xwiki performance parameters. e.g. the Admin Tools Application tells me that

Your cache configuration is too low for a production usage. Your cache is set to 100 documents, it should be 1000 or more

Shouldn’t that be 500 per default?

   149  #-# Maximum number of documents to keep in the cache.
   150  #-# The default is 500.
   151  # xwiki.store.cache.capacity=500

Maybe there is more to be done. I’m playing around with a standalone SOLR container, but not sure if that helps anything, but it sounds reasonable to keep it seperate from the xwiki server.

If we can’t get everything we need into the official image, we can still fork the git repository and roll our own. Not everybody needs all that stuff. Thats the sex appeal of automated build processes.

BR

#20

I found in general:
If you can get away with it don’t terminate SSL/TLS on tomcat and use a reverse proxy ( outside of the web docker) instead. Creating java keystores is normally harder than just throwing the plain text files in Apache/nginx, especially with enterprise signed certs. If you want SSL on the default port (443) you need to run tomcat as root or take other additional measures like jsvc or firewall rules. Even if this is not a problem with docker as you can connect 443:8443 (or whatever the SSL port is) you still have the issue to get the jks in the container and change the config (don’t know if this is prepared).

LDAP Authentication and a lot of the other custom settings would be really easy if you could mount _local versions of the config files into the container as discussed above.

#21

I may be singing the same old song again, but I still think the documentation to set up a dev setup is just not complete enough to intrigue developers that are not full-time Java and maven cracks.

#22

Have you checked https://dev.xwiki.org/xwiki/bin/view/Community/Contributing and specifically https://dev.xwiki.org/xwiki/bin/view/Community/Contributing#HContributeCode

This leads you to the Onboarding area:
https://dev.xwiki.org/xwiki/bin/view/Onboarding/