Supported email clients

Development
#1

Hi!

Currently, the XWiki Email Notifications and other mailing features (like registration, account validation, password reset, username retrieval, etc) are regularly manually tested on Gmail web client.

Due to the fact that the emails could be differently displayed on various email clients, I wanted to ask your opinion on whether we should start defining a list of email clients that should be supported (and tested on).

Any opinions or suggestions on this would be appreciated.

Thanks!

#2

Current thoughts:

  • Testing mail clients is very time consuming since it’s a manual process
  • We need to be able to have something automated
  • What would be awesome would be to find some open source library that can perform HTML/CSS validation for several mail clients and that we would run as part of our validation checks in our build.

Unfortunately I’ve quickly searched for such a library and couldn’t find any. The best option I’ve found so far is https://mailtrap.io/ which is not open source and works as a service. They have a REST API so technically it’s possible to integrate it in our CI but it becomes a point of failure. It’s also not free except for the base plan which might or might not be enough (IMO if we wanted to go this route, we would need some sponsor to pay something on behalf of the xwiki open source project).

At the very least we should manually run our various generated emails through https://mailtrap.io/ to see what information it gives us. Note: @ilie.andriuta has agreed to do this and to publish the results.

Thoughts?

#3

Hello all,

I wanted to ask your opinion on whether we should start defining a list of email clients that should be supported (and tested on).

That’s a definite “yes”. We need to define this, just as we do for browsers and their versions.
This is needed for both testing but also for correctly giving visibility to a developer about the (minimal) support they need to ensure for a mail feature that they may be developing, otherwise the results would depend on the opinion or inspiration of the developer, I guess.

  • Testing mail clients is very time consuming since it’s a manual process
  • We need to be able to have something automated

I think these are related to the “implementation” of the way to get to an objective, but not to the objective itself. Ilie’s question was about whether we should define an objective and, I guess, about what this objective should be (although Ilie did not ask that explicitly). This is to say that:

  • the amount of work needed will depend on the number (and diversity) of clients we decide to support. We cannot evaluate the amount of work unless we know what we want to support.
  • we cannot evaluate a verification tool if we don’t have the objectives to validate against. Passing all tests of mailtrap (or other library) is not an objective itself
  • the fact that we decide to support an email client or not (the objective) is also needed for knowing how to react in front of bugs that are being raised, not only for organising the validation. So we can declare objectives of support without talking about the testing tools or even without testing (:stuck_out_tongue: ) only to know what to do with the bugs raised by users.

While I understand that it’s important to always calibrate the objectives in relation with the processes / tools available to achieve those objectives, I think it’s really interesting, in this case, to see exactly where we all stand in terms of vision of this objective (what would we “want” to support).

Thanks,
Anca

#4

From Email client market share in 2021: Trends from January to March - Litmus (if we trust it), we can see that we would get 77.6% of coverage with:

  • Apple iPhone 38.9%
  • Gmail 27.2%
  • apple Mail 11.5%

Now if we restrict to just desktop, it’s 98.9% with:

  • Apple mail 58.9%
  • Outlook 40%