Test results depends on Mac device

[navartis]

After updating MacOS up to Catalina 10.15 and Xcode up to 11.1 (11A1027)

I found that tests on other Mac device are failed with reference images saved an my Mac device.

Latest stable version 6.2.0 of library are used.

I spend 2 days trying to find the reason and fix it, implement "Highlight Different Pixels" feature and create request #108 which can help investigate such situations.

But now a time I have no ideas how to fix this issue.

[MaikCL]

Same happen to me, I record all images using Catalina, but running the same test in a Travis environment with macOS 10.14 It's failing for the diffs, as if they were shadows with different contrast

[mime29]

Same issue here. Any idea about the reason?

[navartis]

It seems to me the reason is that Apple in Xcode 11x migrate Simulators processing from CPU to GPU.

GPU calculations faster but not accurate and it result may depends on hardware GPU venders and drivers.

I spend a lot of time, but the only working work around for me is using perPixelTolerance: 6/256
in func FBSnapshotVerifyView.

And this request #108

[luiz-brasil]

It seems to me the reason is that Apple in Xcode 11x migrate Simulators processing from CPU to GPU.

GPU calculations faster but not accurate and it result may depends on hardware GPU venders and drivers.

I spend a lot of time, but the only working work around for me is using perPixelTolerance: 6/256
in func FBSnapshotVerifyView.

And this request #108

@navartis Why are you using the 6/256 number for perPixeltolerance?

[mime29]

[Q] We didn't find a good solution on my side yet but we noticed one thing: The random diff result seems to come from views containing transparent assets (images with alpha channel not set to 1)
Do you observe the same thing on your sides?

[navartis]

It seems to me the reason is that Apple in Xcode 11x migrate Simulators processing from CPU to GPU.
GPU calculations faster but not accurate and it result may depends on hardware GPU venders and drivers.
I spend a lot of time, but the only working work around for me is using perPixelTolerance: 6/256
in func FBSnapshotVerifyView.
And this request #108

@navartis Why are you using the 6/256 number fo perPixeltolerance?

@luiz-brasil Unfortunately '6/256' is a magic number. I hate magic numbers in code, but in this case I have no ideas

[samskiter]

We're also seeing this issue. Interestingly it seems to only occur when using either usesDrawViewHierarchyInRect OR snapshotting during a UI test (not unit test).

Seems that it could be related to being rendered in a UIWindow.

I also got the same results between two Mac Pros (i.e. identical machines produce the same result).

Could this be graphics card related and is it time to raise a radar?

[pigeon56]

I'm so glad that I stumbled across this thread, because I think I'm seeing the same thing.

About a week ago, I started noticing failures in some of my iOS Snapshot tests. These tests passed locally - both when run via Xcode and Fastlane - but failed when run on CI. I'm still trying to figure out which OS our build machine is using (though I suspect it might be 10.14.x), but it just so happens that I updated my machine to Catalina last week, and now I'm getting these failures.

I was able to capture the diff screenshots after running the tests on CI, and they're gray. I asked a designer to take one and bump up the contract and only with a massive bump in contrast were we able to see the slightest traces of some scree elements. (And I really had to squint to see them, even after the adjustment.)

Seems like the workaround at the moment is to bump up the tolerance. Has anyone found a different solution?

[samskiter]

Just to confirm - is everyone else seeing that this not only happens between 10.14 and 10.15 (which is normal for any upgrade) but also between machines that are running the same version of macOS Catalina (10.15)?

Finally, has anyone got a simple test case that could be sent to Apple?

[luiz-brasil]

@samskiter unfortunately I don't have a specific test case, primarily because these errors occur randomly.

I can affirm that the error occurs between different machines with the same version of Mac OS Catalina.

[gspiers]

@samskiter Results seem to be GPU dependent. In the simulator you can get different results by using the integrate or discrete GPUs File -> GPU Selection. I've done a lot of investigation and the only fix was to increase the tolerance of snapshots.

One of the Apple engineers that works on the simulator said the way forward is to increase pixel matching tolerance. Conversation on Twitter. Hope that helps.

[navartis]

😢

[samskiter]

Wow, apple just broke an entire tool. Thanks Apple.

…

On Tue, 5 May 2020 at 17:31, navartis ***@***.***> wrote: 😢 — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#109 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAJWOW56ZLTVHHMZVZYTWC3RQA5MRANCNFSM4JGMB5YA> .

[JustinDSN]

@samskiter Results seem to be GPU dependent. In the simulator you can get different results by using the integrate or discrete GPUs File -> GPU Selection. I've done a lot of investigation and the only fix was to increase the tolerance of snapshots.

One of the Apple engineers that works on the simulator said the way forward is to increase pixel matching tolerance. Conversation on Twitter. Hope that helps.

Because of this issue, what values are teams using for perPixelTolerance and overallTolerance?