Xwiki 10.5 entended cpu spike while adding new pages

[vmassol]

I actually don’t know how to script that, so yes, I’m doing it manually. If I’m not insane by now, I soon will be!

Something like this:

{{velocity}}
#if ($request.confirm == '1')
  #set ($targetDoc = $xwiki.getDocument('XWiki.SomeUser'))
  #set ($result = $targetDoc.removeObjects('XWiki.Notifications.Code.NotificationFilterPreferenceClass'))
  #if ($result)
    #set ($discard = $targetDoc.save('Removed all notification filter xobjects'))
    XObjects removed!
  #else
    Error: XObjects not removed!
  #end
#else
  [[Remove XObjects>>||queryString='confirm=1']]
#end
{{velocity}}

[vmassol]

FWIW I’ve put the snippet here: http://snippets.xwiki.org/xwiki/bin/view/Extension/Remove%20Objects/

[pdwalker]

Thank you!

I’m going to let this run overnight and see if it’s finished by the morning.

[edit] and done! it only took a few minutes to run!

I’ll test to see if I get any more spikes when editing/creating the pages and let you know.

[pdwalker]

@pdwalker once you’ve finished deleting the notification filters, it’ll be interesting to see if that’s the problem of your CPU spikes. If it is, I still don’t understand what would be the relationship between Creating Pages and Notification code executing though. We’ll see. Let us know when you’re done. Also we need to fix the Notifications performance issue we’ve discovered recently when there are lots of filters (one thing we need to do is check why this is taking long from a DB point of view since a WHERE clause with right indexes shouldn’t take that long).

Oh, the difference is night and day. Editing/saving is fast and responsive. No cpu spikes, no delays, no errors, no timeout messages.

Those notification filters were definitely having some kind of adverse effect.

[vmassol]

Oh, the difference is night and day. Editing/saving is fast and responsive. No cpu spikes, no delays, no errors, no timeout messages.

Those notification filters were definitely having some kind of adverse effect.

Ok, very interesting, thanks.

@gdelhumeau We really have a big issue here that we need to fix somehow. This could actually make XWiki 9.11.x and 10.x not recommended/usable.

it looks like for every page I created, a filter was automatically created to notify me of any changes made to the pages

This looks like autowatch is turned on. @gdelhumeau is it on by default now? (it used to be off by default for the watchlist).

[gdelhumeau]

@pdwalker: Could you send me a XAR of the page in the state it was when you had the problem? I need to digg more to understand why you have this problem. I remember a bug at some point, when the same filter was saved a thousand of times because of some concurrency issues. Having ~700 filters is not supposed to happen in normal conditions, so I suspect a bug there.

[pdwalker]

@gdelhumeau,

All I had to do was make a new page to start seeing the performance issues. It affected everything i did. The problem completely disappeared once I deleted all the notification filters.

It feels like there may be some kind of O(n^2) algorithm running in there somewhere that’s not noticeable under normal circumstances.

@vmassol,

The ~700 plus filters is likely my fault as I may have turned it on by default when i was first setting up the wiki. I’d need to do a brand new install to see if that was really the case.

There is one potential problem that I think actually does need fixing; when I was first setting things up, I created a subwiki called “test” for testing the confluence page importing. The pages I manually created there resulted in thr creation of some notification filters.

Later, I deleted the subwiki, but yesterday I noticed that all the notification filters remained behind (but without showing a hyperlink to the now deleted page). That definitely shouldn’t be. Those notification filters should be removed when when the subwiki is removed.

I have previously seen errors in Catalina.out referencing the deleted subwiki and I think that this might have been the source of those errors. I’ll know soon enough.

Lastly, just an observation (warning: I know absolutely nothing about xwiki internals so could be completely wrong here) - it seems weird to me that notifications for page changes are associated with user objects rather than the page objects. Is there a particular reason why it was designed this way? If the “who to notify” data were associated with the page data it seems that it’d be easier to clean up old notifications when a page was deleted.

[rbr]

Hi,

It’s actually true for my installation (10.4) that autosubscribe is on for newly created users (synced from LDAP).
To be honest I’m not a big fan of this because I had to answer a lot of these “why do I get these (notification) mails?”. And people actively creating content and pages collected hundreds of this notification filters too.
In combination with the filters not being cleaned up on page delete (as @pdwalker) mentioned this is a “time bomb”. People who are actively editing will collect enough filters over time to bring down the wiki.
It is the whole point of my wiki and the migration to xwiki to encourage more active contributions.

Please make this whole notification feature opt-in not opt-out! Also please add some kind of bulk delete for the filters!

Will check later how many people already have tons of notification filters, thanks for the script @vmassol.

[Pbas]

Will check later how many people already have tons of notification filters, thanks for the script @vmassol.

Hello, I was worry about this issue myself then I writed this snippet to display a nice filtered table of users with one or more NotificationFilterPreferenceClass (or another class) objects
If that can can help

{{velocity}}
#set($className='XWiki.Notifications.Code.NotificationFilterPreferenceClass')
**Users with 1 or more $className object:**

$xwiki.ssfx.use("js/xwiki/table/table.css")
$xwiki.jsfx.use("js/xwiki/table/tablefilterNsort.js", true)

{{html wiki='true'}}
<table id="tableid" class="grid sortable filterable doOddEven">
  <tr class="sortHeader">
    <th>User</th> <th>Object number</th>
  </tr>
#set($hql = "select obj.name from BaseObject obj where obj.className='XWiki.XWikiUsers'")
## display 10 users
## set($results = $xwiki.search($hql, 10, 0))
#set($results = $xwiki.search($hql))
#foreach ($User in $results)
  ## * [[$User]]
  ## if you have more than one object on a page, you will have to loop over them and use "$doc.use"
  #set($MyDoc = $xwiki.getDocument("$User"))
  #set($class = $xwiki.getClass("$className"))
  #set($count = 0)
  #set ($displayUser="")
  ## loop over all objects
  #foreach($obj in $MyDoc.getObjects("$className"))
    #set($count = $velocityCount)
    ##set ($displayUser="* [[$User]]: $count")
    #set ($displayUser='<tr><td>' + [[$User]] + '</td><td>' + $count + '</td></tr>')
  #end
$!displayUser
#end
</table> 
{{/html}}

{{/velocity}}

Pascal B

[pdwalker]

Thanks to @Pbas I had a nice way to check the other users on my system and sure enough it seems the other major editor has over 200 notification filters.

Our system also syncs the users with Windows AD, so it appears that enabling notifications is the default.

[pdwalker]

Another benefit I am seeing since I deleted all those notifications - the jvm memory usage is way down; roughly speaking from 2.5GB to 750 MB.

(full sized screenshot)

[gdelhumeau]

Hi.

On the XWiki’s side, we are going to:

• store the filters in a way that it won’t generate 700 objects (even if there is 700 filters)
• create a migration mechanism to convert existing objects into a few ones
• improve the UI so it would be easy to remove batch of filters

It’s going to require some work so I hope we will have this for 10.7.

In the meantime, you could:

• disable autowatch by default, adding into xwiki.properties:

   xwiki.plugin.watchlist.automaticwatch=none

(even if the watchlist is not used anymore, there is a fallback to this property)

• remove exiting filters using the script given by @vmassol

That should help until we provide an official fix. Sorry for that inconvenience. We are working on it!

Guillaume

[gdelhumeau]

Edit: it’s

xwiki.plugin.watchlist.automaticwatch=none

[gdelhumeau]

I have created a new JIRA issue for the filter preferences problem:
https://jira.xwiki.org/browse/XWIKI-15445

For the SQL problem, the following fixed issue should already be an improvement:
https://jira.xwiki.org/browse/XWIKI-15440

Until this problem is properly fixed, you could disable the autowatch mode in XWiki UI’s in the next XWiki version:
http://www.xwiki.org/xwiki/bin/view/ReleaseNotes/Data/XWiki/10.6RC1/#HAbilitytosetthedefaultautowatchmodeintheadministration

Thanks,
Guillaume

[gdelhumeau]

Later, I deleted the subwiki, but yesterday I noticed that all the notification filters remained behind (but without showing a hyperlink to the now deleted page). That definitely shouldn’t be. Those notification filters should be removed when when the subwiki is removed.

I have created the corresponding issue:
https://jira.xwiki.org/browse/XWIKI-15460

Thanks

[tmortagne]

There is one potential problem that I think actually does need fixing; when I was first setting things up, I created a subwiki called “test” for testing the confluence page importing. The pages I manually created there resulted in thr creation of some notification filters.

You should definitely not end up with all those documents taken into account since the filter stream job produce BeginFoldEvent which tells the notification to not take into account any document event during this process.

You used standard Filter Stream app, right ? Not some custom script ?

I think the priority before trying to optimize the notification for such volume is to find how you ended up with such nonsense volume in the first place and fix that. In general we need to make sure the notification module properly take into account BeginFoldEvent marker and that this marker is used as much as possible when it make sense (for example you should not end with a preference for each document in a space you moved, either nothing or a single preference for the space itself).

[pdwalker]

Let me clarify.

The imported pages didn’t create notifications, only those pages I created manually.

The 700+ notification filters included some manually created pages from the test instance, and all the other pages from manually create pages in the main and one other sub wiki.

Does that clear things up?

[tmortagne]

Does that clear things up?

Yep but I still find 700 very big for hand written pages.

[pdwalker]

It’s not when you are converting an old confluence wiki to xwiki by hand.

I wished I could have used the converter, but with the amount of cleanup required, it was actually easier to to recreate the pages and then cut and paste the pages from old to the new.

What “problems” did I have with the confluence converter?

• the xwiki code inherited a lot of markup garbage from the confluence pages, requiring extensive cleanup. (faster to cut and paste funnily enough)
• the document tree structure wasn’t preserved - which would require reorganizing the hierarchy after importing, so I took this opportunity to go through each of the pages to see if they were still needed.

So, 700+ pages? Not a problem in this case.

[tmortagne]

the document tree structure wasn’t preserved - which would require reorganizing the hierarchy after importing, so I took this opportunity to go through each of the pages to see if they were still needed.

Actually it is preserved but using the old parent/child relationship (which is actually what you have on Confluence) but AFAIK you could have used https://extensions.xwiki.org/xwiki/bin/view/Extension/Nested%20Pages%20Migrator%20Application.