Test results of XWiki mailing features on mailtrap.io

Development
#1

Hi!

Following some tests of XWiki mailing features I have run on mailtrap.io online tool (using the Free Plan version), here are the results:

My environment: Windows 10 Pro 64 bit, Chrome 91, using a local instance of XWiki 13.4 on PostgreSQL 13/ Tomcat

I’ve used the following scenario:

  1. Login as Admin
  2. Go to Administer Wiki > Mail > Mail Sending and configured ‘Email address to send from:’ to John Doe<iliexwiki@gmail.com>
  3. Configure server, server port, server username and password with ones provided by mailtrap.io
  4. Insert on ‘Additional properties’ mail.smtp.starttls.enable=true
  5. Create an user with a valid email set on its profile (I used the same email address I’ve created the account on mailtrap.io with)
  6. Login as user (U1)
  7. Go to Notifications > Settings and set all Pages Notifications to ON (both menu and email)
  8. Watch the wiki
  9. Login as Admin
  10. Go to Administer Wiki > Look & Feel > Themes and upload a large logo for the wiki
  11. Create some pages (each with Home page as their parent)
  12. Create/update/comment/like a page
  13. Trigger the ‘Notifications daily email’ job from Scheduler.WebHome
  14. Share a page by email to the user U1
  15. Go to Administer Wiki > Users & Rights > Registration and set ‘Use email verification’ to ‘Yes’
  16. Logout
  17. Register an user
  18. Logout
  19. Click Log-in > ‘Forgot your username?’ and insert the email address registered with the user, then clicked ‘Retrieve username’
  20. Click Log-in > ‘Forgot your password?’ and insert the username U1, then clicked ‘Reset password’

Notifications Email feature

Here is how the email message looks on small mobile devices:

1_Look_on_Mobile

Here is how the email message looks on tablet devices:

2_Look_on_tablets

Here is how the email message looks on desktop devices:

3_Look_on_Desktop

Spam Analysis and Blacklist

The Spam Analysis tab provides information about Spam and Blacklist, but in this case is not that relevant, since I’ve configured ‘Email address to send from:’ to a Gmail address.

4_Spam_Analysis

The Spam report score of 2.2 out of 5 (the lower the score, the better) is due to 2 mailtrap.io Rules being affected:

  • FORGED_GMAIL_RCVD (‘From’ gmail.com does not match ‘Received’ headers)
  • NML_ADSP_CUSTOM_MED (ADSP custom_med hit, and not from a mailing list)

If on Administer Wiki > Mail > Mail Sending the ‘Email address to send from:’ is not configured, the Spam Report score is 1 with Blacklist Report not available, as expected.

4a_Spam_Analysis_without_From

HTML Check

The HTML check tab analyzes the HTML elements and compares them against known compatibilities with 5 popular email clients (Apple Mail, Gmail, Outlook, Yahoo! Mail, Samsung Email) and Others (like Thunderbird, AOL, etc) on 3 categories (Desktop, Mobile and Web), see more details on mailtrap.io - HTML Check.

Overall, mailtrap.io gave a score of 87% Market Support of HTML elements from Email Notifications on the tested email clients:

5_HTML_check_all

On Desktop only, the supported score is higher - 91.4%:

6_HTML_check_Desktop

The “Not Supported” elements here (2.6%) are related entirely to Outlook due to the fact that some elements are not supported in some clients as following:

  • Anchor links (not supported on Outlook macOS)
  • background (not supported in Outlook Windows 10 Mail)
  • border-radius (not supported in Outlook Windows and Outlook Windows 10 Mail)
  • overflow (found on Outlook Windows and Outlook Windows 10 Mail)

while some other elements are only partially supported (6% - most of them by Apple Mail client).

On Mobile side, the score is 86.8%:

7_HTML_check_Mobile

The “Not Supported” elements (8.8%) are represented by the following:

  • Anchor links (not supported by most mobile email clients)
  • text-decoration (not supported on Orange iOS, ProtonMail iOS, SFR iOS)
  • height property (not supported on Yahoo! Mail Android and Yahoo! Mail iOS)
  • overflow (found on Outlook Android and Yahoo! Mail Android)

while the most of partially supported elements (from the total of 4.4%) are related to Yahoo! Mail.

The highest supported score is on Web clients side - 95.5%:

8_HTML_check_Web

Here, the “Not Supported” elements score is very small - 0.6% and represents some elements (like height property) not supported by Yahoo! Mail Desktop Webmail and other less popular email clients.
The partially supported elements have a small score of 3.9%, being related mostly to Yahoo! Mail Desktop Webmail.

Tech Info

The Tech Info tab provides some information about email headers:

9_Tech_Info

Share page by Email feature

10a_HTML_check_Share_page

For this feature, the HTML Check gave a very high score of 99.4%.

10_HTML_check_Share_page

There is, though, a padding element which is partially supported on Outlook Windows and Outlook Windows 10 Mail.

Registration feature

11_Registration_user

Here it’s quite strange that mailtrap.io stated that the email message had no HTML version and the activation link was displayed as text.

I’ve tested on Gmail Web client and the activation URL was displayed properly as link.

Forgot username feature

12_Forgot_username

For this feature, HTML Check gave a perfect score of 100%:

12a_Forgot_username

Reset password feature

13_Pass_reset

For this feature, HTML Check also gave a perfect score of 100%:

13a_Pass_reset

Conclusion

According to the results of tests run on mailtrap.io, I would tend to say that the results are good, but there are still some work to do on elements partially supported or not supported by some popular email clients, especially by Outlook and Yahoo! Mail.

#2

It’s interesting.

One thing I notice is that these results are only an entry point, they don’t seem to provide the functional consequences - manual verification, I guess, would be needed for the functional consequences (and the evaluation of their criticity).
For example, for the Outlook Desktop client results, there doesn’t seem to be any indication of what is the actual consequence to the user of the 65% support. Also, as a technical person, I can interpret the result (the 4 elements that are not supported), and they don’t seem to cover the causes of https://jira.xwiki.org/browse/XWIKI-18697 or https://jira.xwiki.org/browse/XWIKI-18696 ). This is also one observation to note here, that I can only say this because I am a technical person, so the result itself cannot be used as-is, since the objective is a functional one.

Also, there doesn’t seem to be a distinction of client’s versions… shouldn’t there be one?
And also I don’t see Thunderbird email client in the list of desktop apps… it’s not identified at all (it’s in “other” category) or it’s just hidden somewhere?
Also, what is the Gmail in the list of desktop clients? Is there such a thing?

As a conclusion, it appears to me that mailtrap may be interesting for identifying the clients that would need a manual check, but that there may to be a risk of false validtion (to be checked). On a closer look (resulting in the questions above) I am rather skeptical about the verification capabilities of mailtrap, I would need more convincing…

Thanks,
Anca

#3

Thanks for your feedback.

My main questions to you would be:

  • In the list of identified issues by mailtrap, are there false positives?
  • Isn’t there anything true there that you have not noticed yourself manually for example?

I’ve always been convinced that mailtrap would not capture everything (since it’s based on validation rules and not using the real mail client, which btw, they also support by supporting forwarding the emails to read users using the target mail clients, but that requires a manual verification ofc), but my hope was that it would capture enough to make it useful.

Thanks

#4

Also, there doesn’t seem to be a distinction of client’s versions… shouldn’t there be one?

The client versions are displayed on each element’s advanced section, for example:

Mailtrap_versions !

Mailtrap_versions2 !

And also I don’t see Thunderbird email client in the list of desktop apps… it’s not identified at all (it’s in “other” category) or it’s just hidden somewhere?

According to https://help.mailtrap.io/article/60-html-check#filtering, Thunderbird is listed as “less popular client” and listed on “Other” category.

Also, what is the Gmail in the list of desktop clients? Is there such a thing?

Actually I’m not really sure what they mean by that (I’ve also searched a little bit in their documentation), as there isn’t an official Gmail Desktop app, but apparently there is an workaround for it: How to get Gmail as a desktop app.

#5

but apparently there is an workaround for it: How to get Gmail as a desktop app.

Technically that’s just creating a shortcut to a webpage which would open in a browser. So it’s still the web client…

#6

false positive = a thing that is not an issue but it’s raised by mailtrap?

I would need to check that.

I also need to check that.

“useful” depends on the purpose, as I said in a different thread. If the purpose is to be able to tell: “the latest commit has lowered the compatibility level for this browser” then I guess yes, it can be somewhat useful. However, there are 2 situations which, too frequent, may endup lowering this usefulness (I would say):

  • the issues that are not detected by mailtrap (false negatives) - compat percentage stays the same so it somewhat validates that the latest commit is “fine”, when it’s actually not really
  • the issues that are detected by mailtrap in the markup but have no real efect on the mail for a human (the false positives that I need to check above) - in which case one would need to work to “fix the test”.

Too much of these 2 mixed situations may basically make it not better than random, but we still need to check.

Thanks,
Anca