Test Apple Major OS releases 2024

[kahest]

Description

Description

iOS 18, tvOS 18, watchOS 11, macOS 15 are about to GA in September 2024. We should test whether our SDK works appropriately on these new OS versions.

Other tasks

Must-have - pre release

Beta Give feedback

1. Test plan manual iOS+macOS/MacCatalyst
2. Test plan manual tvOS
3. Test plan manual watchOS

Must-have - post release

Beta Give feedback

1. Test plan check with customer data

Optional / TBD

Beta Give feedback

1. Add SwiftData operations to iOS-Swift Sample #3105
   Platform: Cocoa +1
2. Enable "mergeable library" feature #3719
   Platform: Cocoa +1
   
3. Add sample app using mergeable libraries #3103
   Platform: Cocoa +1
   
4. Add viewIsAppearing to UIViewController transactions #3100
   Platform: Cocoa Type: Enhancement +2
5. Swift Macro for Instrumenting SwiftUI #3104
   Platform: Cocoa +1

Test plan manual test

Feature	iOS	macOS	tvOS	MacCatalyst
Crashes with symbolication	✅	✅
Error events with symbolication	✅	✅
OOMs	✅	🚫		🚫
App hangs	✅	🚫
Release health	✅
View hierarchy	✅	🚫
Screenshots	✅	🚫
Auto-generated breadcrumbs	✅
Attachments	✅	✅
App start	✅	🚫	☑️
Slow & frozen frames	✅	🚫
UIViewController tracking	✅	🚫
TTID/TTFD	✅	🚫
Network tracking	✅	🚫
File I/O tracking	✅	🚫
Core data tracking	✅	🚫	☑️
User interaction tracking	✅	🚫
Swift UI tracking	✅	✅
MetricKit	✅		☑️
Profiling		🚫	☑️

🚫 = not available
☑️ = works on another platform. Not needed to put in the effort to test on this platform, because we are going to validate the feature with customer data.
✅ = tested successfully

Test plan check with customer data

Check features with customer data once iOS 18 is released. We only check iOS with customer data as things breaking there would have the most significant negative impact because we have, by far, the most projects on iOS. For the other platforms, the manual test above is sufficient.

Feature	iOS
Crashes with symbolication
Error events with symbolication
OOMs
App hangs
Release health
View hierarchy
Screenshots
Auto-generated breadcrumbs
Attachments
App start
Slow & frozen frames
UIViewController tracking
TTID/TTFD
Network tracking
File I/O tracking
Core data tracking
User interaction tracking
Swift UI tracking
MetricKit
Profiling

---

Similar to #3129

[denrase]

iOS Setup

iPhone 13 Pro - iOS 18.0 (22A5350a) Public Beta
Sentry 8.36.0 - TestFlight & Sentry 8.35.1 Debug build with Xcode 16.0 Beta 6

Manual Tests

Crashes with symbolication ✅

• Works as expected
  • https://sentry-sdks.sentry.io/issues/3607568922/?project=5428557&query=is%3Aunresolved+issue.priority%3A%5Bhigh%2C+medium%5D&referrer=issue-stream&statsPeriod=1h&stream_index=4

Error events with symbolication ✅

• Works with "Capture error"
  • https://sentry-sdks.sentry.io/issues/5485500008/events/9eeeff026c74498bb966c919d6925094/?project=5428557&query=is%3Aunresolved+issue.priority%3A%5Bhigh%2C+medium%5D&referrer=next-event&statsPeriod=1h&stream_index=0
• Works with "Capture NSException"
  • https://sentry-sdks.sentry.io/issues/2862440042/?project=5428557&query=is%3Aunresolved+issue.priority%3A%5Bhigh%2C+medium%5D&referrer=issue-stream&statsPeriod=1h&stream_index=0

OOMs ✅

• Works as expected
  • https://sentry-sdks.sentry.io/issues/3808632822/?project=5428557&query=is%3Aunresolved+issue.priority%3A%5Bhigh%2C+medium%5D&referrer=issue-stream&statsPeriod=1h&stream_index=1

App hangs ✅

• Works as expected
  • https://sentry-sdks.sentry.io/issues/5795592786/?project=5428557&query=is%3Aunresolved+issue.priority%3A%5Bhigh%2C+medium%5D&referrer=issue-stream&statsPeriod=1h&stream_index=0

Release Health ✅

• Works as expected
  • https://sentry-sdks.sentry.io/releases/io.sentry.sample.iOS-Swift%408.36.0%2B1716/?project=5428557

View hierarchy ✅ / ❓

• Works as expected in error
  • https://sentry-sdks.sentry.io/issues/2862440042/?project=5428557&query=is%3Aunresolved+issue.priority%3A%5Bhigh%2C+medium%5D&referrer=issue-stream&statsPeriod=1h&stream_index=0
• No view hierarchy in crash, is this expected behaviour? ❓
  • https://sentry-sdks.sentry.io/issues/3607568922/?project=5428557&query=is%3Aunresolved+issue.priority%3A%5Bhigh%2C+medium%5D&referrer=issue-stream&statsPeriod=1h&stream_index=4

Auto-generated breadcrumbs ✅

• Works as expected
  • https://sentry-sdks.sentry.io/issues/3607568922/?project=5428557&query=is%3Aunresolved+issue.priority%3A%5Bhigh%2C+medium%5D&referrer=issue-stream&statsPeriod=1h&stream_index=4

Attachments ✅

• All issues have attachments added though their initial scope setup
• tongario.jpg https://sentry-sdks.sentry.io/api/0/projects/sentry-sdks/sentry-cocoa/events/edead652a4b442cdae42ff2db727be5e/attachments/11464150437/?download=1
• log.txt https://sentry-sdks.sentry.io/api/0/projects/sentry-sdks/sentry-cocoa/events/edead652a4b442cdae42ff2db727be5e/attachments/11464150436/?download=1

App start ✅

• App start metrics are available on sentry.io
  • https://sentry-sdks.sentry.io/insights/mobile/app-startup/?project=5428557&statsPeriod=1h
  • https://sentry-sdks.sentry.io/performance/trace/f931ab2830e54ee6b3e241110d5349b8/?app_start_type=cold&eventId=72667ba90788435b92639c41d0b52c27&groupId=7b81f5a5b2c6aa09&node=span-a4ab8f10f7f04876&node=txn-72667ba90788435b92639c41d0b52c27&primaryRelease=io.sentry.sample.iOS-Swift%408.35.1%2B1&project=5428557&secondaryRelease=io.sentry.sample.iOS-Swift%408.36.0%2B1716&source=app_starts_module&spanDescription=loadView&spanGroup=7b81f5a5b2c6aa09&spanOp=ui.load&statsPeriod=1h&timestamp=1725442353&transaction=ErrorsViewController

Slow & frozen frames ✅

• Pressed "Cause frozen frames"
• Can see "Most Frozen Frames" in dashboard in the Extra screen

• Can see frozen frames in trace view

UIViewController tracking ✅

• Can see view different opened controllers in breadcrumbs of error message (Ui Lifecycle)

TTID/TTFD ✅

• Works as expected

Network tracking ✅

• Can see network breadcrumbs (HTTP)

File I/O tracking

• Transactions have file spans

Core data tracking ✅ / ❓

• Can see some file spans in 'traces' view
• Are there supposed to be spans for record creation/deletion? ❓

User interaction tracking ✅

• Issues have breadcrumbs of button presses and session replay
  • https://sentry-sdks.sentry.io/issues/2862440042/?project=5428557&query=is%3Aunresolved+issue.priority%3A%5Bhigh%2C+medium%5D&referrer=issue-stream&statsPeriod=1h&stream_index=0

Swift UI tracking ✅

• Could not check, as captureMessage crashes in iOS13-Swift target fixed
• Works as expeted

MetricKit ✅

• MetricKit is added as an integration.
• Worked with disk write issue
  • https://sentry-sdks.sentry.io/issues/5796197385/?project=5428557&query=is%3Aunresolved+issue.priority%3A%5Bhigh%2C+medium%5D&referrer=issue-stream&statsPeriod=1h&stream_index=0

Profiling ❌

• Could not check, as the sample app crashes when trying to view profile data.
  • https://sentry-sdks.sentry.io/issues/5795748619/?project=5428557&query=is%3Aunresolved+issue.priority%3A%5Bhigh%2C+medium%5D&referrer=issue-stream&statsPeriod=1h&stream_index=0

• ProfilesViewController shown with count 3, but there is no Flamegraph/Call Tree
  • https://sentry-sdks.sentry.io/performance/summary/profiles/?project=5428557&statsPeriod=1h&transaction=ProfilingViewController

Other

• Device orientation seems to be incorrect
• https://sentry-sdks.sentry.io/issues/2862440042/events/d8050b41d5864f05af59d974be6ec7b4/?project=5428557&query=is%3Aunresolved+issue.priority%3A%5Bhigh%2C+medium%5D&referrer=next-event&statsPeriod=1h&stream_index=0

[denrase]

MacOS (Catalyst) ❌

Cannot run Sample application because Sentry is not found.

Setup

macOS Sequoia 15.0 Beta (24A5331b)
MacBook Pro 15-inch 2019 (Intel)

Xcode Version 16.0 beta 6 (16A5230g)
Runing in debug mode.

Results

Feature	Status	Notes
Crashes with symbolication		--
Error events with symbolication		--
App hangs		--
Release health		--
View hierarchy		--
Screenshots		--
Auto-generated breadcrumbs		--
Attachments		--
App start		--
Slow & frozen frames		--
UIViewController tracking		--
TTID/TTFD		--
Network tracking		--
File I/O tracking		--
Core data tracking		--
User interaction tracking		--
Swift UI tracking		--
MetricKit		--
Profiling		--

[denrase]

watchOS

Did basic tests on watchOS, as the available features are part of the list above.

Setup

watchOS 11 (22R5348a)
Apple Watch Ultra 2
Version 16.0 beta 6 (16A5230g)
Runing in debug mode.

Results

Test	Result	Notes
captureMessage	✅	--
captureError	✅	--
captureException	✅	--
crash	❌	Not showing up on sentry.io. According to logs, envelope sent successfully.
auto breadcrumbs	✅	--
attachments	✅	--

@kahest Could not get SentrySDK.crash() to show up on sentry.io .

[denrase]

macOS

Setup

macOS Sequoia 15.0 Beta (24A5331b)
MacBook Pro 15-inch 2019 (Intel)

Xcode Version 16.0 beta 6 (16A5230g)
Runing in debug mode.

Results

Feature	Status	Notes
Crashes with symbolication	✅	--
Error events with symbolication	✅	--
Release health		--
Auto-generated breadcrumbs	✅	--
Attachments	✅	--
Swift UI tracking	✅	--
MetricKit		Could not provoke MetricKit issue.
Profiling		--