slow macOS - "##[error]The job running on agent Azure Pipelines 9 ran longer than the maximum time of 60 minutes."

[jeffschwMSFT]

Build

https://dnceng.visualstudio.com/internal/_build/results?buildId=2360768&view=results

Error

##[error]The job running on agent Azure Pipelines 9 ran longer than the maximum time of 60 minutes. For more information, see https://go.microsoft.com/fwlink/?linkid=2077134

Build leg reported

vsos

Pull Request

No response

Known issue core information

Fill out the known issue JSON section by following the step by step documentation on how to create a known issue

 {
    "ErrorMessage" : "",
    "BuildRetry": false,
    "ErrorPattern": "The job running on agent Azure Pipelines .+ ran longer than the maximum time of .+ minutes.",
    "ExcludeConsoleLog": false
 }

@dotnet/dnceng

Release Note Category

• Feature changes/additions
• Bug fixes
• Internal Infrastructure Improvements

Release Note Description

Additional information about the issue reported

No response

Known issue validation

Build: 🔎 https://dev.azure.com/dnceng/internal/_build/results?buildId=2360768
Error message validated: [The job running on agent Azure Pipelines .+ ran longer than the maximum time of .+ minutes.]
Result validation: ✅ Known issue matched with the provided build.
Validation performed at: 2/7/2024 1:03:13 AM UTC

Report

Build	Repository	Step Name	Console log	Pull Request
3005157	dotnet-maui	Build macOS (Debug)	Log
1474440	dotnet/maui	Controls ViewBaseTests,VisualStateManager,Window	Log	dotnet/maui#31755
1474425	dotnet/maui	Controls Layout	Log	dotnet/maui#29800
1474418	dotnet/maui	Controls CollectionView	Log	dotnet/maui#29922
1474392	dotnet/runtime	maccatalyst-x64 Release AllSubsets_CoreCLR	Log	dotnet/runtime#129299
1474170	dotnet/runtime	maccatalyst-arm64 Release AllSubsets_CoreCLR_RuntimeTests	Log
1474114	dotnet/runtime	osx-arm64 Debug Libraries_CheckedCoreCLR	Log	dotnet/runtime#129661
3004754	dotnet-runtime	Upload Build Artifact LinuxMonoAOTarm64	Log
3004753	dotnet-runtime	Upload Build Artifact BrowserWasm	Log
1473998	dotnet/runtime	osx-arm64 Debug Libraries_CheckedCoreCLR	Log	dotnet/runtime#129637
1473844	dotnet/maui	Controls CollectionView	Log	dotnet/maui#35960
3004466	dotnet-aspnetcore	Build: macOS x64	Log
1473612	dotnet/runtime	maccatalyst-arm64 Release AllSubsets_CoreCLR	Log
1473432	dotnet/aspnetcore	Build: macOS x64	Log	dotnet/aspnetcore#67327
1473409	dotnet/aspnetcore	Test: macOS	Log	dotnet/aspnetcore#67111
3004194	dotnet-maui	Controls CollectionView	Log
1473256	dotnet/aspnetcore	Build: macOS arm64	Log	dotnet/aspnetcore#63817
1473225	dotnet/maui	Controls CollectionView	Log
1473203	dotnet/maui	Controls Shell	Log	dotnet/maui#35626
3004261	dotnet-efcore	macOS	Log
3004213	dotnet-maui	RunOniOS_MauiRelease_CoreCLR ARM64	Log
1473194	dotnet/aspnetcore	Build: macOS arm64	Log	dotnet/aspnetcore#67325
3004193	dotnet-maui	RunOniOS_BlazorRelease x64	Log
1473180	dotnet/runtime	osx-arm64 Debug Libraries_CheckedCoreCLR	Log	dotnet/runtime#129475
1473163	dotnet/maui	Controls CollectionView	Log	dotnet/maui#34408
1473172	dotnet/aspnetcore	Build: macOS arm64	Log	dotnet/aspnetcore#67323
1473098	dotnet/aspnetcore	Build: macOS arm64	Log
1473116	dotnet/runtime	osx-arm64 Debug Libraries_CheckedCoreCLR	Log	dotnet/runtime#129555
1472827	dotnet/aspnetcore	Build: macOS arm64	Log	dotnet/aspnetcore#67265
1473029	dotnet/runtime	osx-arm64 Debug Libraries_CheckedCoreCLR	Log	dotnet/runtime#129553
3004091	dotnet-maui	RunOniOS_MauiDebug_CoreCLR ARM64	Log
1473003	dotnet/aspnetcore	Build: macOS x64	Log	dotnet/aspnetcore#67249
1472970	dotnet/runtime	osx-arm64 Debug Libraries_CheckedCoreCLR	Log	dotnet/runtime#129605
3004115	dotnet-efcore	macOS	Log
1471217	dotnet/aspnetcore	Build: macOS x64	Log	dotnet/aspnetcore#67258
1472846	dotnet/maui	Controls Shell	Log
1472885	dotnet/runtime	osx-arm64 Debug CoreCLR_Libraries	Log	dotnet/runtime#129627
1472850	dotnet/aspnetcore	Test: macOS	Log	dotnet/aspnetcore#66487
1472824	dotnet/aspnetcore	Build: macOS arm64	Log	dotnet/aspnetcore#67093
1472767	dotnet/runtime	osx-arm64 Debug CoreCLR_Libraries	Log	dotnet/runtime#129578
1472667	dotnet/runtime	maccatalyst-arm64 Release AllSubsets_Mono_RuntimeTests	Log
1472659	dotnet/aspnetcore	Build: macOS arm64	Log	dotnet/aspnetcore#66331
1472646	dotnet/aspnetcore	Build: macOS arm64	Log	dotnet/aspnetcore#66442
1472592	dotnet/runtime	osx-arm64 Debug Libraries_CheckedCoreCLR	Log	dotnet/runtime#129403
1472519	dotnet/maui	Controls Shell	Log	dotnet/maui#36020
1472490	dotnet/maui	Controls CollectionView	Log	dotnet/maui#34600
1472514	dotnet/aspnetcore	Build: macOS arm64	Log	dotnet/aspnetcore#67249
3003905	dotnet-maui	RunOniOS_MauiReleaseTrimFull_CoreCLR ARM64
1472497	dotnet/aspnetcore	Build: macOS x64	Log	dotnet/aspnetcore#66487
1472437	dotnet/runtime	coreclr Pri0 Runtime Tests Run osx x64 checked	Log	dotnet/runtime#129614
1472472	dotnet/maui	Controls CollectionView	Log	dotnet/maui#34758
1472487	dotnet/aspnetcore	Test: macOS	Log
1472479	dotnet/aspnetcore	Test: macOS	Log	dotnet/aspnetcore#67316
1472478	dotnet/aspnetcore	Test: macOS	Log	dotnet/aspnetcore#67315
3003929	dotnet-performance	Performance scenarios windows 22H2 arm64 Ampere 8.0	Log
1472463	dotnet/maui	Controls CollectionView	Log
1472468	dotnet/runtime	osx-arm64 Release NativeAOT_Libraries	Log	dotnet/runtime#128252
1472456	dotnet/aspnetcore	Build: macOS arm64	Log	dotnet/aspnetcore#67296
1472427	dotnet/aspnetcore	Test: macOS	Log	dotnet/aspnetcore#66868
1472423	dotnet/aspnetcore	Test: macOS	Log	dotnet/aspnetcore#67019
1472417	dotnet/aspnetcore	Build: macOS arm64	Log	dotnet/aspnetcore#67091
1472411	dotnet/aspnetcore	Build: macOS x64	Log	dotnet/aspnetcore#67017
1472406	dotnet/aspnetcore	Build: macOS x64	Log	dotnet/aspnetcore#67059
1472398	dotnet/aspnetcore	Build: macOS x64	Log	dotnet/aspnetcore#67178
1472374	dotnet/aspnetcore	Test: macOS	Log	dotnet/aspnetcore#67015
1472370	dotnet/aspnetcore	Build: macOS x64	Log	dotnet/aspnetcore#67146
1472357	dotnet/aspnetcore	Build: macOS arm64	Log	dotnet/aspnetcore#67313
1472311	dotnet/maui	Controls Layout	Log	dotnet/maui#35463
1472292	dotnet/maui	Controls CollectionView	Log	dotnet/maui#35964
1472323	dotnet/aspnetcore	Build: macOS arm64	Log
1472071	dotnet/aspnetcore	Tests: macOS	Log	dotnet/aspnetcore#67267
1472272	dotnet/aspnetcore	Build: macOS x64	Log	dotnet/aspnetcore#67314
1472216	dotnet/aspnetcore	Test: macOS	Log	dotnet/aspnetcore#67313
1472207	dotnet/aspnetcore	Build: macOS x64	Log	dotnet/aspnetcore#67312
1472179	dotnet/runtime	osx-x64 release Runtime_Release	Log	dotnet/runtime#129299
1471131	dotnet/maui	Controls CollectionView	Log
1472101	dotnet/maui	Controls ViewBaseTests,VisualStateManager,Window	Log	dotnet/maui#31755
1472121	dotnet/aspnetcore	Build: macOS x64	Log	dotnet/aspnetcore#67265
1472077	dotnet/aspnetcore	Test: macOS	Log	dotnet/aspnetcore#67015
1472070	dotnet/aspnetcore	Build: macOS arm64	Log	dotnet/aspnetcore#67267
1472032	dotnet/runtime	maccatalyst-arm64 Release AllSubsets_CoreCLR	Log	dotnet/runtime#129593
1472019	dotnet/maui	Controls CollectionView	Log	dotnet/maui#35924
1471968	dotnet/maui	Controls Layout	Log
3003704	dotnet-runtime	Upload Build Artifact LinuxMonoAOTx64	Log
1471911	dotnet/runtime	maccatalyst-x64 Release AllSubsets_CoreCLR	Log
1471848	dotnet/maui	Controls Shell	Log
1471737	dotnet/runtime	osx-arm64 Debug CoreCLR_Libraries	Log	dotnet/runtime#129599
1471288	dotnet/maui	Controls Shell	Log	dotnet/maui#35878
1471699	dotnet/aspnetcore	Test: macOS	Log
1471671	dotnet/maui	Controls Shell	Log	dotnet/maui#35462
1471605	dotnet/aspnetcore	Build: macOS arm64	Log	dotnet/aspnetcore#66487
3003538	dotnet-diagnostics	MacOS_arm64_Release	Log
1471433	dotnet/runtime	maccatalyst-arm64 Release AllSubsets_CoreCLR_RuntimeTests	Log	dotnet/runtime#129299
1470558	dotnet/aspnetcore	Build: macOS arm64	Log	dotnet/aspnetcore#67267
1471442	dotnet/maui	Controls Shell	Log
3003503	dotnet-performance	Performance maui_scenarios_ios NativeAOT CoreCLR osx 26 x64 iPhone17 main	Log
1471435	dotnet/aspnetcore	Test: macOS	Log	dotnet/aspnetcore#67249
1471399	dotnet/runtime	osx-arm64 Debug Libraries_CheckedCoreCLR	Log	dotnet/runtime#129580
3003492	dotnet-efcore	macOS	Log
1471419	dotnet/aspnetcore	Test: macOS	Log
Displaying 100 of 2030 results

Summary

24-Hour Hit Count	7-Day Hit Count	1-Month Count
6	365	2030

[lewing]

🤔

[nagilson]

@dotnet/dnceng @dougbu This has impacted a lot of PRs recently, may you PTAL into expanding this Mac resource?

[ivanpovazan]

@dotnet/dnceng we are hitting this again and it seems there is some issue with communication with Helix machines.

More context on timeouts happening in : https://dev.azure.com/dnceng-public/public/_build/results?buildId=930014&view=results

• On success: Run tests in Helix step reports:

Waiting for completion of job fbec1164-55c5-4fa8-b492-e1ba1b413119 on osx.1200.amd64.open (Details: https://helix.dot.net/api/jobs/fbec1164-55c5-4fa8-b492-e1ba1b413119/details?api-version=2019-06-17 )
Job 7fdc2720-3d98-4bf3-8113-274dacd69c91 on osx.1200.arm64.open is completed with 6 finished work items.
  Job fbec1164-55c5-4fa8-b492-e1ba1b413119 on osx.1200.amd64.open is completed with 6 finished work items.
  Stopping Azure Pipelines Test Run Helix Tests Build_Debugosx.1200.amd64.open (Results: https://dev.azure.com/dnceng-public/public/_build/results?buildId=923987&view=ms.vss-test-web.build-test-results-tab )
  Stopping Azure Pipelines Test Run Helix Tests Build_Debugosx.1200.arm64.open (Results: https://dev.azure.com/dnceng-public/public/_build/results?buildId=923987&view=ms.vss-test-web.build-test-results-tab )

Build succeeded.

SENDHELIXJOB : warning : Helix queue osx.1200.amd64.open was set for estimated removal date of 2025-01-01. In most cases the queue will be removed permanently due to end-of-life; please contact dnceng for any questions or concerns, and we can help you decide how to proceed and discuss other options. [/home/vsts/work/1/s/tests/integration-tests/Apple/Simulator.Tests.proj]
SENDHELIXJOB : warning : Helix queue osx.1200.arm64.open was set for estimated removal date of 2025-01-01. In most cases the queue will be removed permanently due to end-of-life; please contact dnceng for any questions or concerns, and we can help you decide how to proceed and discuss other options. [/home/vsts/work/1/s/tests/integration-tests/Apple/Simulator.Tests.proj]
    2 Warning(s)
    0 Error(s)

Time Elapsed 00:03:14.97
Killing running build processes...

Finishing: Run tests in Helix

ref: https://dev.azure.com/dnceng-public/public/_build/results?buildId=923987&view=logs&j=ccc97bb6-1a23-5e71-fdfa-3cdca4a74749&t=27fc7eb2-ead9-59e1-6679-a637855d40c5

• While on failure - timeout the same step gets stuck with:

Waiting for completion of job 967d92a2-ec10-4332-927f-d28a6563f367 on osx.1200.arm64.open (Details: https://helix.dot.net/api/jobs/967d92a2-ec10-4332-927f-d28a6563f367/details?api-version=2019-06-17 )
  Job 5427159b-500d-49f1-aac0-ec148a492bbe on osx.1200.amd64.open is completed with 6 finished work items.

ref: https://dev.azure.com/dnceng-public/public/_build/results?buildId=930014&view=logs&s=c58bc33c-b825-5bca-90ca-50f6e9293dd8&j=e6966639-fe40-5068-d9ae-681cccecafdf

NOTE: All the tests successfully passed on Helix, but it seems that the communication is lost.

[garath]

Looks like the timeout happened because the osx.1200.arm64.open queue was very busy while the job was running. Right now, I do not think there are any problems with the infrastructure. I will investigate a bit more to see what caused such a back-up.

[garath]

Ah, the queue was consumed with updates and patching. The patching jobs did run longer than necessary and we've communicated with our partner team about the issue. Future jobs will be much shorter and should not overly impact jobs.

[ivanpovazan]

Looks like the timeout happened because the osx.1200.arm64.open queue was very busy while the job was running. Right now, I do not think there are any problems with the infrastructure. I will investigate a bit more to see what caused such a back-up.

We are still experiencing the problem.

Should we try to change the queue to osx.13.arm64?

@dotnet/dnceng

[ilyas1974]

Looks at the failing builds, I'm noticing the jobs that are timing out in the hosted pool (Azure Pipelines) are still using the older hardware. I would recommend moving the workloads to the mac-latest-internal or mac-14-arm64 agent specifications. These have the latest mac hardware associated with them.