Support strategy for email client?

Development
#1

Hi everyone,

as part of my BFD yesterday I found a few tickets related to email notifications and bug with them on Outlock. It appeared that those bugs couldn’t be reproduced anymore with more recent versions when @ilie.andriuta tested them so they’ve been closed.

Now we still have from time to time tickets related to emails, and to my knowledge we don’t have a clear support strategy for email clients. So maybe we should clarify this situation: for example, personally I don’t have any Microsoft account and I cannot test with Outlook: I always test mail features with thunderbird on my computer.

I honestly which direction we want to take there, but it might be interesting to document a bit more what we’re actually testing.

wdyt?

#2

I don’t think we’re testing much at all actually :wink:

The mail tests we have are using Greenmail and the tests will just assert the content of the mail returned by the mail client. It would be hard to test that the mail client renders it correctly. We would need some kind of image regression testing for this.

We might have some manual tests done on some mail clients but that’s not even specified. For example https://test.xwiki.org/xwiki/bin/view/Notification%20Tests/Live%20email%20notifications just says An email is sent to the user that has the notifications set to ON, Live..

So for me, we don’t have mail client support (where we check the rendered content in the mail client) at all ATM.

Thanks

#3

Definitely, but I think what @surli is after is more what do we do with bugs related to how a mail renders in a specific mail clients. I don’t think it’s realistic to have the kind of testing that we have for supported browsers, as you mentioned, but there are still things we can say:

  1. we just commit to fix reported bugs for a given list of clients (to be decided if we have some regular manual tests on those, I think @ilie.andriuta use Gmail in some tests)
  2. we don’t commit to fix reported bugs for any client

As for what about the clients for which we don’t commit to fix bugs:

  • a) close as won’t fix issue related to any other mail client
  • b) keep open issues related to any other mail client and people can provide pull requests for those

Either ways, it’s IMO good to have a page stating it along with the other support pages.

I feel like we should at least commit to fix bugs on a few clients with a very high marked share but which are easy for any of the dev to test on (works with Gmail but maybe not so much with Outlook).

I’m really not a fan of a) (which, from what I understand, is more or less what @surli was hoping for when he started this thread. To me, there is no reason to close them, they are just clearly not a priority.

1 Like
#4

So +1 to add an entry at https://dev.xwiki.org/xwiki/bin/view/Community/SupportStrategy/ for mail client support, in which we say:

  • That we currently don’t have tests for checking how mails are displayed in various clients and that manual tests are done from time to time in an ad-hoc manner
  • That we will fix issues reported for Apple Mail, Gmail & Outlook (ie the top 3 from Most Used Email Clients Worldwide [Jan 2024 Update] | Oberlo ).
    • Not sure what “Apple” means, whether it’s iOS mail client or mac’s mail client, or both maybe.
    • As you say the hard part is testing on the various platforms, but amongst the xwiki dev team members we have access to mac, windows and linux so it should be doable.
    • I’ve found one tool which is “free for personal use”, see Free HTML & CSS Email Checker by Mailtrap (found from How to Test Emails in Different Clients | Mailtrap Blog). There might be open source tools, idk. Too bad that Browserstack that doesn’t allow testing email clients.
    • I don’t think we can say we won’t commit to fix issues for apple mail when it’s the #1 mail client used. Same for outlook.
  • For other mail clients, if we don’t close the issues, then we need at least a way to know which are issues that the xwiki core dev team won’t work on so that we can filter them out when we need (it could also be explained that a sponsoring could be done to work on those). And it would be interesting to review our issue list and clean it up to mark issues we don’t plan to work on.

Thanks

#5

Not sure how accurate this source is.

#6

Do you have a better one? I’ve checked 2 or 3 and they had the same numbers.