Daily Email Notification issues (out of memory, attachments, Slow)

I have a 10.8.1 instance running unaltered code, on a system with 8GB of RAM and JAVA/Tomcat are setup to use 6GB of RAM now.

I’ve been noticing some inconsistencies where not all of my ~80 users are receiving their daily notification emails.
At first this was caused by out of memory issues when the server was configured to use only 2GB of RAM. It would make it to about 20 or so and then bomb. Now that it’s at 6GB it stops at around 50 emails, but with no error this time.

I started looking into this a bit, wondering why something as simple as small emails (maybe 5 changes for the day), would be using so much RAM and I noticed that with every email sent in the batch it’s duplicating the attachments for the HTML formatted message (today the first email had 5 attachments for the email images, the second 10, and 5 are added for each in the batch)

At this point the notification emails are as good as useless because the later emails are so bloated or not going out that I’m just going to have to turn them off.

Looking at JIRA I see 2 potentially related issues: https://jira.xwiki.org/browse/XWIKI-15355 and https://jira.xwiki.org/browse/XWIKI-15329

now, I can look into adjusting the properties in XWiki/Notifications/MailTemplate but eventually I’d like to have HTML emails again. any pointers? I can see where I can remove the image for the logo, but after that it goes down into the methods in macro.vm, and I’m wary of customizing things too much, especially on the file system

So after much experimentation, the out of memory situation is resolved, but the leaky attachments are still causing major problems.

I edited the XWiki/Notifications/MailTemplate HTML template to remove any references to images, and commented out much of the code that is handled by macros.vm and it is still attaching the images, even though the email no longer references them. I even tried removing the template reference to macros.vm and replicating the code therein within the property itself, only removing all image references and the attachments persist. Restarting the instance did not help either.

it appears that the internal JAVA classes are attaching the files regardless of whether they are actually used.

At the core of this is that when the job kicks off, it starts duplicating the attachments every time a new email is generated in the job (or rather, each email contains all attachments from the previous emails in the job). This is fine for a wiki with a few users, but once you hit around 40-50 users this actually chokes some mail servers and/or clients as an email with one change and no details will have at minimum the Logo, the gear cog, a profile picture and a wiki icon, which is 4 attachments. when you extrapolate this, the second email in the job will contain 8 attachments, 3rd will have 12 and so on. after 40 emails it will have 160 attachments, 50 will have 200, and so on…

Ideally, every email would have an an object that contains a list of only the attachments for that email and only those are attached. I haven’t dug deeply into the code but it appears that either an array is not being cleared, is being reused or the temp file issue outlined above is the cause.

so is this bug report time or am I missing a setting somewhere to turn off the attachments?

@ryanp Thanks for digging into this. Looks like an important bug. Could you raise a jira issue about it? Thanks

Will do. thanks for the reply.

Well, I went to submit the bug and found this: https://jira.xwiki.org/browse/XWIKI-15788 Needless to say, I’m not going to post a duplicate issue as that would just clutter up jira. Instead I’ll track that issue and post any additional information I find unless you want me to post an issue relating to attachments being added that aren’t used, which is less of a concern to me than the duplication.

perfect thx!

The development team has resolved the issue of duplicate attachments in the above issue and a fix will be in future releases. Thanks guys, you’re awesome.