High traffic on XWiki instance led to database inconsistency regarding commenting date

Help / Discuss
#1

Recently we had a phase of high wiki usage - >100 users were accessing the wiki in a short time frame. What the users did was mainly commenting on existing pages.

At some point comments were no more saved into the database - the users sent the comment and got the green “Comment saved” but after refreshing the page the comment was not displayed. This happened to all documents from that time on, no commenting was possible (except for a few exceptions which I still don’t understand).

Analyzing this I found a related error which was that editing objects was not possible for pages with comments - a 4001 was thrown in editobject.vm. (Comments are technically also objects as I understand it.) For pages without comments, editing objects still worked. For the above error, the log gave a IndexOutOfBounds error description.

Having direct database access, I could nail down the cause to a missing timestamp in table xwikidates for an xws_id related to a comment property - the entry in xws_value was NULL. I directly updated this value with a meaningful timestamp and all the above errors were resolved.

That is, I do not have an urgent requirement for help. However, this issue obviously caused some stability concerns in me - this must not happen again for us, otherwise thy buy-in for the wiki will be seriously at stake. My questions:

  • How can such a thing happen?
  • Has anyone experienced something similar?
  • Might it be a thing specifically related to comments?
  • Do we need to prevent highly concurrent editing usage of the wiki at all?
  • Giving the green “comment saved” when it is actually NOT saved is an issue in itself. Can this false positive be prevented?

Thanks for any pointers. Happy to provide more details if needed. We are using 13.10 with PostgreSQL and Tomcat behind a Nginx proxy and a Kubernetes deployment. Over all, XWiki is a great application and we’re grateful having it installed.

#2

If you have a way to reproduce this, it could be interesting to test with 14.10.x (or more).

What I can see in the code is that in 13.10 the comment action (the comments UI is indeed using a dedicated commentadd endpoint and is not going through the more generic xobject API) the concurrently handling of comments is not so great (all the requests which are commenting on the same document are manipulating the same instance of document and those documents are not really thread safe, so you could indeed end up with some strange results when several comments are added at the exact same time). This changed in 15.2-rc-1 and 14.10.7 and they each work on their side and manipulate a different document instance.

That being said even in a more recent version of XWiki, while I doubt you are going to end up with a broken document, the behavior might not be perfect either: if several comments are saved exactly at the same time, It’s possible you end up with only one of those comments being visible in the end (the last actual save overwriting the previous ones).

#3

Upgrading is something we don’t desire yet because of the effort. But if there are very clear reasons to do so we might consider it.

However, I don’t see how we can reproduce this behaviour. As said, at some point of high usage a user comment was saved with no timestamp for whatever reason. I have no details of that very activity and the circumstances.

What I could easily reproduce is the behaviour IF there is a NULL date in xwsdates for a xws_id which belongs to a comment object – I could just reverse my update and set the timestamp to NULL again. That is something you probably could also do if you had a 13.10 somewhere – just access the database of the XWiki instance and set such a timestamp to NULL. In 13.10 you should probably see the same behaviour (editing objects not possible + no comments can be saved any more) – in 14.10 maybe not. But I think you’re more after reproducing the root cause (NULL timestamp).

And, after looking into the code, can you also imagine how a comment can be saved without timestamp? That seems to be a bit different from an incomplete transaction where in some table some entry was not written which had belonged to the transaction. Instead, the database row was written into the xwikidates table, only with a NULL timestamp.

Is there a more stable way of collecting many user inputs concurrently? Would it maybe be an AWM solution, a live table or something else?

And also, any ideas on the issue with the misleading green “Comment saved” message?

#4

To avoid any concurrent problem, we generally recommend having a separated document for each entry, which is indeed how AWM (and most XWiki applications) works.

#5

Yes, this is not really supposed to happen.