Updating our support strategy for Safari

Development
#1

At the moment, we advertise Safari as a fully-supported browser. However, we have neither automated nor manual testing on Safari and most developers currently working on XWiki don’t have access to Safari for testing, which makes it also hard to reproduce and fix issues. I propose two changes here:

  1. Degrade the support status for Safari. As Safari has a significant market share, I’m not suggesting to drop it but we should make it clear that Safari isn’t as well-supported as the other browsers and in particular not regularly tested (but still considered when we evaluate, e.g., if a certain web technology can be used).
  2. Add automated testing for Safari using BrowserStack. BrowserStack offers free tests for open source and supports integration with Selenium and Jenkins which should allow automated tests using our existing UI tests in Safari on our existing CI system. We would need to verify that this is indeed possible, and that XWiki indeed qualifies for this open source program (I couldn’t find the specific conditions). We also need to check if this could give XWiki committers access to live testing on Safari (and other browsers) to be able to reproduce and fix any identified issues.

I also cannot say how much work the second change is, which might influence if or when we can implement it. Until we implement it or if we don’t implement it, I would propose to clearly state on the browser support page that there is neither manual nor automatic testing performed on Safari.

Any opinions?

#2

It’s not clear to me why we need BrowserStack for Safari, isn’t it a matter of finding the right docker image with Safari in it like we do for other browsers ?

#3

Safari only supports macOS, and from my understanding it is not legally allowed to run macOS on non-Apple hardware.

#4

Right, it’s not easy to run anything else than Linux in a docker image.

#5

Okay, there might be another option which is to use something similar to Safari, WebDriverManager apparently supports a Safari-like docker image:

Real Safari browser can only run under MacOS \ iOS. These Linux images are using WebKit engine used in Safari browser. From functional point of view real Safari and WebKit browser being used inside these images are equal. However fonts and pixel per pixel rendering could differ.

So this could be another, potentially easier option but it would still mean we’re not actually testing on Safari. We could also decide to go this route and indicate this in the browser support strategy.

#6

Yes, sounds good enough.

#7

+1

thanks

#8

Hello. We had discussed this in the past and one idea was to add support for IE/Edge and Safari on windows by adding a windows agent machine and modifying our jenkins pipeline to support it, when TC would have support for it and when there would be some docker images for it (original message: Re: [xwiki-devs] [Brainstorming] Next gen functional testing for XWiki - Vincent Massol - org.xwiki.devs - MarkMail ).

The assumption was that Safari on Windows shouldn’t be too different from Safari on Mac.

I had also opened an related issue: https://github.com/testcontainers/testcontainers-java/issues/766

Now, one reason we didn’t push on this is also because we realized that Safari has moved to webkit and thus testing on Chrome was testing a lot for Safari too. And it seem it’s been true since we haven’t had a lot problem in lots of years.

So right now, my preference would indeed to do the following:

  • Indicate that we don’t support Safari to the same level as the other browsers
  • Discuss with @ilie.andriuta to see if he’d be ok to do some tests on Safari every N months. Note: Safari can be installed on Windows
  • I’d prefer to not use another testing tool like browserstack. Or we could use it as test if we wanted to move all our tests there but I fear that could degrade stuff in our build (would we be able to fail the build when it fails on browserstack, can we control setup of the VMS there like being able to have LO installed, etc). We would also be quite dependent on the day they stop supporting us.
#9

This is not true anymore, see here:

Chrome doesn’t use Webkit and there are not many similarities. Web standards support is unfortunately very different and there are frequently features that aren’t supported by Safari. Therefore, our current testing strategy doesn’t cover Safari or its engine.

Therefore, if we want to improve the situation without a lot of efforts, I think our best option is really to use the “Safari” docker image that are using a WebKit browser.

Regarding BrowserStack, we would still host the VMs that run XWiki from my understanding, they would just tunnel the HTTP requests from their browser to our test container. But anyways, as indicated in the discussion above, the favorite option is the above-mentioned Safari container, not BrowserStack.

#10

That’s not so easy as it’s not supported by TestContainers AFAIK.

ATM it looks a bit too complex to me for the benefits.

#11

Okay, didn’t know this is a different browser container ecosystem that’s incompatible with what we’re using right now.

Then I guess the only option we have is to degrade the Safari support status and clearly state that while we consider Safari when selecting web technologies or dependencies, we aren’t performing any automated or manual tests on Safari.