Test results of XWiki mailing features on mailtrap.io

ilie.andriuta · May 28, 2021, 12:14pm

Hi!

Following some tests of XWiki mailing features I have run on mailtrap.io online tool (using the Free Plan version), here are the results:

My environment: Windows 10 Pro 64 bit, Chrome 91, using a local instance of XWiki 13.4 on PostgreSQL 13/ Tomcat

I’ve used the following scenario:

Login as Admin
Go to Administer Wiki > Mail > Mail Sending and configured ‘Email address to send from:’ to John Doe<iliexwiki@gmail.com>
Configure server, server port, server username and password with ones provided by mailtrap.io
Insert on ‘Additional properties’ mail.smtp.starttls.enable=true
Create an user with a valid email set on its profile (I used the same email address I’ve created the account on mailtrap.io with)
Login as user (U1)
Go to Notifications > Settings and set all Pages Notifications to ON (both menu and email)
Watch the wiki
Login as Admin
Go to Administer Wiki > Look & Feel > Themes and upload a large logo for the wiki
Create some pages (each with Home page as their parent)
Create/update/comment/like a page
Trigger the ‘Notifications daily email’ job from Scheduler.WebHome
Share a page by email to the user U1
Go to Administer Wiki > Users & Rights > Registration and set ‘Use email verification’ to ‘Yes’
Logout
Register an user
Logout
Click Log-in > ‘Forgot your username?’ and insert the email address registered with the user, then clicked ‘Retrieve username’
Click Log-in > ‘Forgot your password?’ and insert the username U1, then clicked ‘Reset password’

Notifications Email feature

Here is how the email message looks on small mobile devices:

Here is how the email message looks on tablet devices:

Here is how the email message looks on desktop devices:

Spam Analysis and Blacklist

The Spam Analysis tab provides information about Spam and Blacklist, but in this case is not that relevant, since I’ve configured ‘Email address to send from:’ to a Gmail address.

The Spam report score of 2.2 out of 5 (the lower the score, the better) is due to 2 mailtrap.io Rules being affected:

FORGED_GMAIL_RCVD (‘From’ gmail.com does not match ‘Received’ headers)
NML_ADSP_CUSTOM_MED (ADSP custom_med hit, and not from a mailing list)

If on Administer Wiki > Mail > Mail Sending the ‘Email address to send from:’ is not configured, the Spam Report score is 1 with Blacklist Report not available, as expected.

HTML Check

The HTML check tab analyzes the HTML elements and compares them against known compatibilities with 5 popular email clients (Apple Mail, Gmail, Outlook, Yahoo! Mail, Samsung Email) and Others (like Thunderbird, AOL, etc) on 3 categories (Desktop, Mobile and Web), see more details on mailtrap.io - HTML Check.

Overall, mailtrap.io gave a score of 87% Market Support of HTML elements from Email Notifications on the tested email clients:

On Desktop only, the supported score is higher - 91.4%:

The “Not Supported” elements here (2.6%) are related entirely to Outlook due to the fact that some elements are not supported in some clients as following:

Anchor links (not supported on Outlook macOS)
background (not supported in Outlook Windows 10 Mail)
border-radius (not supported in Outlook Windows and Outlook Windows 10 Mail)
overflow (found on Outlook Windows and Outlook Windows 10 Mail)

while some other elements are only partially supported (6% - most of them by Apple Mail client).

On Mobile side, the score is 86.8%:

The “Not Supported” elements (8.8%) are represented by the following:

Anchor links (not supported by most mobile email clients)
text-decoration (not supported on Orange iOS, ProtonMail iOS, SFR iOS)
height property (not supported on Yahoo! Mail Android and Yahoo! Mail iOS)
overflow (found on Outlook Android and Yahoo! Mail Android)

while the most of partially supported elements (from the total of 4.4%) are related to Yahoo! Mail.

The highest supported score is on Web clients side - 95.5%:

Here, the “Not Supported” elements score is very small - 0.6% and represents some elements (like height property) not supported by Yahoo! Mail Desktop Webmail and other less popular email clients.
The partially supported elements have a small score of 3.9%, being related mostly to Yahoo! Mail Desktop Webmail.

Tech Info

The Tech Info tab provides some information about email headers:

Share page by Email feature

For this feature, the HTML Check gave a very high score of 99.4%.

There is, though, a padding element which is partially supported on Outlook Windows and Outlook Windows 10 Mail.

Registration feature

Here it’s quite strange that mailtrap.io stated that the email message had no HTML version and the activation link was displayed as text.

I’ve tested on Gmail Web client and the activation URL was displayed properly as link.

Forgot username feature

For this feature, HTML Check gave a perfect score of 100%:

Reset password feature

For this feature, HTML Check also gave a perfect score of 100%:

Conclusion

According to the results of tests run on mailtrap.io, I would tend to say that the results are good, but there are still some work to do on elements partially supported or not supported by some popular email clients, especially by Outlook and Yahoo! Mail.

lucaa · May 28, 2021, 1:35pm

It’s interesting.

One thing I notice is that these results are only an entry point, they don’t seem to provide the functional consequences - manual verification, I guess, would be needed for the functional consequences (and the evaluation of their criticity).
For example, for the Outlook Desktop client results, there doesn’t seem to be any indication of what is the actual consequence to the user of the 65% support. Also, as a technical person, I can interpret the result (the 4 elements that are not supported), and they don’t seem to cover the causes of Loading... or Loading... ). This is also one observation to note here, that I can only say this because I am a technical person, so the result itself cannot be used as-is, since the objective is a functional one.

Also, there doesn’t seem to be a distinction of client’s versions… shouldn’t there be one?
And also I don’t see Thunderbird email client in the list of desktop apps… it’s not identified at all (it’s in “other” category) or it’s just hidden somewhere?
Also, what is the Gmail in the list of desktop clients? Is there such a thing?

As a conclusion, it appears to me that mailtrap may be interesting for identifying the clients that would need a manual check, but that there may to be a risk of false validtion (to be checked). On a closer look (resulting in the questions above) I am rather skeptical about the verification capabilities of mailtrap, I would need more convincing…

Thanks,
Anca

vmassol · May 28, 2021, 1:43pm

Thanks for your feedback.

My main questions to you would be:

In the list of identified issues by mailtrap, are there false positives?
Isn’t there anything true there that you have not noticed yourself manually for example?

I’ve always been convinced that mailtrap would not capture everything (since it’s based on validation rules and not using the real mail client, which btw, they also support by supporting forwarding the emails to read users using the target mail clients, but that requires a manual verification ofc), but my hope was that it would capture enough to make it useful.

Thanks

ilie.andriuta · May 28, 2021, 1:51pm

Also, there doesn’t seem to be a distinction of client’s versions… shouldn’t there be one?

The client versions are displayed on each element’s advanced section, for example:

!

And also I don’t see Thunderbird email client in the list of desktop apps… it’s not identified at all (it’s in “other” category) or it’s just hidden somewhere?

According to https://help.mailtrap.io/article/60-html-check#filtering, Thunderbird is listed as “less popular client” and listed on “Other” category.

Also, what is the Gmail in the list of desktop clients? Is there such a thing?

Actually I’m not really sure what they mean by that (I’ve also searched a little bit in their documentation), as there isn’t an official Gmail Desktop app, but apparently there is an workaround for it: How to get Gmail as a desktop app.

lucaa · May 28, 2021, 2:43pm

but apparently there is an workaround for it: How to get Gmail as a desktop app.

Technically that’s just creating a shortcut to a webpage which would open in a browser. So it’s still the web client…

lucaa · May 28, 2021, 3:08pm

false positive = a thing that is not an issue but it’s raised by mailtrap?

I would need to check that.

I also need to check that.

“useful” depends on the purpose, as I said in a different thread. If the purpose is to be able to tell: “the latest commit has lowered the compatibility level for this browser” then I guess yes, it can be somewhat useful. However, there are 2 situations which, too frequent, may endup lowering this usefulness (I would say):

the issues that are not detected by mailtrap (false negatives) - compat percentage stays the same so it somewhat validates that the latest commit is “fine”, when it’s actually not really
the issues that are detected by mailtrap in the markup but have no real efect on the mail for a human (the false positives that I need to check above) - in which case one would need to work to “fix the test”.

Too much of these 2 mixed situations may basically make it not better than random, but we still need to check.

Thanks,
Anca