slow macOS - "##[error]The job running on agent Azure Pipelines 9 ran longer than the maximum time of 60 minutes."

[jeffschwMSFT]

Build

https://dnceng.visualstudio.com/internal/_build/results?buildId=2360768&view=results

Error

##[error]The job running on agent Azure Pipelines 9 ran longer than the maximum time of 60 minutes. For more information, see https://go.microsoft.com/fwlink/?linkid=2077134

Build leg reported

vsos

Pull Request

No response

Known issue core information

Fill out the known issue JSON section by following the step by step documentation on how to create a known issue

 {
    "ErrorMessage" : "",
    "BuildRetry": false,
    "ErrorPattern": "The job running on agent Azure Pipelines .+ ran longer than the maximum time of .+ minutes.",
    "ExcludeConsoleLog": false
 }

@dotnet/dnceng

Release Note Category

• Feature changes/additions
• Bug fixes
• Internal Infrastructure Improvements

Release Note Description

Additional information about the issue reported

No response

Known issue validation

Build: 🔎 https://dev.azure.com/dnceng/internal/_build/results?buildId=2360768
Error message validated: [The job running on agent Azure Pipelines .+ ran longer than the maximum time of .+ minutes.]
Result validation: ✅ Known issue matched with the provided build.
Validation performed at: 2/7/2024 1:03:13 AM UTC

Report

Build	Definition	Step Name	Console log	Pull Request
2783063	dotnet-performance	Performance maui_scenarios_ios FullAOT Mono osx 15 x64 iPhoneMini12 main	Log
2782949	dotnet-runtime	Performance ios_scenarios iOSMono JIT iOSStripSymbols osx x64 perfiphone12mini net10.0	Log
2782886	dotnet-runtime	Performance ios_scenarios iOSMono JIT iOSLlvmBuild osx x64 perfiphone12mini net10.0	Log
2782841	dotnet-performance	Performance maui_scenarios_ios NativeAOT CoreCLR osx 15 x64 iPhoneMini12 main	Log
2782825	dotnet-dotnet	OSX_arm64	Log
2782776	dotnet-runtime	Performance ios_scenarios iOSNativeAOT JIT iOSStripSymbols osx x64 perfiphone12mini net10.0	Log
2782708	dotnet-runtime	Performance ios_scenarios iOSMono JIT iOSStripSymbols osx x64 perfiphone12mini net10.0	Log
2782632	dotnet-runtime	Performance ios_scenarios iOSMono JIT iOSLlvmBuild osx x64 perfiphone12mini net10.0	Log
2782616	dotnet-runtime	Performance ios_scenarios iOSMono JIT iOSStripSymbols osx x64 perfiphone12mini net10.0	Log
2782525	dotnet-performance	Performance maui_scenarios_ios NativeAOT CoreCLR osx 15 x64 iPhoneMini12 main	Log
2782496	dotnet-runtime	Performance ios_scenarios iOSMono JIT iOSLlvmBuild osx x64 perfiphone12mini net10.0	Log
2782392	dotnet-runtime	Performance ios_scenarios iOSMono JIT iOSLlvmBuild osx x64 perfiphone12mini net10.0	Log
2782321	dotnet-runtime	Performance ios_scenarios iOSMono JIT osx x64 perfiphone12mini net10.0	Log
2782227	dotnet-runtime	Performance ios_scenarios iOSMono JIT iOSStripSymbols osx x64 perfiphone12mini net10.0	Log
2782195	dotnet-performance	Performance maui_scenarios_ios FullAOT Mono osx 15 x64 iPhoneMini12 main	Log
2782163	dotnet-performance	Performance maui_scenarios_ios FullAOT Mono osx 15 x64 iPhoneMini12 main	Log
2782152	dotnet-runtime	Performance ios_scenarios iOSMono JIT iOSStripSymbols osx x64 perfiphone12mini net10.0	Log
2782089	dotnet-performance	Performance maui_scenarios_ios FullAOT Mono osx 15 x64 iPhoneMini12 main	Log
2782052	dotnet-runtime	Performance ios_scenarios iOSMono JIT iOSLlvmBuild osx x64 perfiphone12mini net10.0	Log
2781971	dotnet-runtime	Performance ios_scenarios iOSMono JIT iOSLlvmBuild osx x64 perfiphone12mini net10.0	Log
2781906	dotnet-performance	Performance maui_scenarios_ios FullAOT Mono osx 15 x64 iPhoneMini12 main	Log
2781877	dotnet-runtime	Performance ios_scenarios iOSMono JIT osx x64 perfiphone12mini net10.0	Log
2781848	dotnet-runtime	Performance ios_scenarios iOSMono JIT iOSLlvmBuild osx x64 perfiphone12mini net10.0	Log
2781747	dotnet-runtime	Performance ios_scenarios iOSMono JIT osx x64 perfiphone12mini net10.0	Log
2781666	dotnet-runtime	Performance ios_scenarios iOSMono JIT iOSStripSymbols osx x64 perfiphone12mini net10.0	Log
1133842	dotnet/runtime	osx-x64 Release AllSubsets_Mono_Minijit_RuntimeTests minijit	Log	dotnet/runtime#119004
1133832	dotnet/runtime	osx-x64 Release AllSubsets_Mono_Minijit_RuntimeTests minijit	Log	dotnet/runtime#119170
2781551	dotnet-runtime	Performance ios_scenarios iOSMono JIT iOSStripSymbols osx x64 perfiphone12mini net10.0	Log
1133715	dotnet/runtime	maccatalyst-arm64 Release AllSubsets_Mono_AppSandbox	Log
1133439	dotnet/arcade	Windows_NT Build_Release	Log
1133431	dotnet/sdk	TestBuild: macOS (x64)	Log	dotnet/sdk#50452
2781179	dotnet-runtime	Performance ios_scenarios iOSMono JIT iOSLlvmBuild osx x64 perfiphone12mini net10.0	Log
2781156	dotnet-runtime	Performance ios_scenarios iOSMono JIT iOSStripSymbols osx x64 perfiphone12mini net10.0	Log
2781083	dotnet-runtime	Performance ios_scenarios iOSMono JIT iOSLlvmBuild osx x64 perfiphone12mini net10.0	Log
2781063	dotnet-performance	Performance maui_scenarios_ios NativeAOT CoreCLR osx 15 x64 iPhoneMini12 main	Log
2780996	dotnet-runtime	Performance ios_scenarios iOSMono JIT osx x64 perfiphone12mini net10.0	Log
2780918	dotnet-runtime	Performance ios_scenarios iOSMono JIT iOSStripSymbols osx x64 perfiphone12mini net10.0	Log
2780912	dotnet-sdk	AoT: macOS (x64)	Log
2780846	dotnet-runtime	Performance ios_scenarios iOSMono JIT iOSStripSymbols osx x64 perfiphone12mini net10.0	Log
2780785	dotnet-performance	Performance maui_scenarios_ios NativeAOT CoreCLR osx 15 x64 iPhoneMini12 main	Log
2780763	dotnet-performance	Performance maui_scenarios_ios FullAOT Mono osx 15 x64 iPhoneMini12 main	Log
1132491	dotnet/sdk	TestBuild: macOS (arm64)	Log	dotnet/sdk#50479
2780667	dotnet-runtime	Performance ios_scenarios iOSMono JIT osx x64 perfiphone12mini net10.0	Log
2780642	dotnet-sdk	TestBuild: macOS (x64)	Log
1132428	dotnet/runtime	osx-arm64 Release NativeAOT_Libs	Log
2780600	dotnet-runtime	Performance ios_scenarios iOSMono JIT osx x64 perfiphone12mini net10.0	Log
2780508	dotnet-runtime	Performance ios_scenarios iOSMono JIT iOSStripSymbols osx x64 perfiphone12mini net10.0	Log
2780426	dotnet-runtime	Performance ios_scenarios iOSMono JIT iOSLlvmBuild iOSStripSymbols osx x64 perfiphone12mini net10.0	Log
2780383	dotnet-runtime	Performance ios_scenarios iOSMono JIT iOSLlvmBuild iOSStripSymbols osx x64 perfiphone12mini net10.0	Log
2780350	dotnet-performance	Performance maui_scenarios_ios FullAOT Mono osx 15 x64 iPhoneMini12 main	Log
2780230	dotnet-runtime	Performance ios_scenarios iOSMono JIT iOSLlvmBuild osx x64 perfiphone12mini net10.0	Log
2780201	dotnet-sdk	AoT: macOS (x64)	Log
2780146	dotnet-runtime	Performance ios_scenarios iOSMono JIT osx x64 perfiphone12mini net10.0	Log
1131649	dotnet/sdk	AoT: macOS (x64)	Log	dotnet/sdk#50472
2780070	dotnet-performance	Performance maui_scenarios_ios NativeAOT CoreCLR osx 15 x64 iPhoneMini12 main	Log
1131646	dotnet/sdk	TestBuild: macOS (x64)	Log	dotnet/sdk#50470
1131644	dotnet/performance	Performance nativeaot_scenarios ubuntu 2204 x64 Open 8.0	Log
2780055	dotnet-performance	Performance nativeaot_scenarios ubuntu 2204 x64 Viper 8.0	Log
2780042	dotnet-performance	Performance imagesharp ubuntu 2204 x64 Viper 8.0	Log
2780004	dotnet-runtime	Performance ios_scenarios iOSMono JIT iOSStripSymbols osx x64 perfiphone12mini net10.0	Log
2779838	dotnet-performance	Performance maui_scenarios_ios FullAOT Mono osx 15 x64 iPhoneMini12 main	Log
2779953	dotnet-runtime	Performance ios_scenarios iOSMono JIT osx x64 perfiphone12mini net10.0	Log
2779899	dotnet-runtime	Performance ios_scenarios iOSMono JIT iOSStripSymbols osx x64 perfiphone12mini net10.0	Log
2779850	dotnet-sdk	AoT: macOS (x64)	Log
2779839	dotnet-sdk	TestBuild: macOS (x64)	Log
2779789	dotnet-runtime	Performance ios_scenarios iOSMono JIT osx x64 perfiphone12mini net10.0	Log
1131284	dotnet/arcade	Windows_NT Build_Release	Log	dotnet/arcade#16078
1131246	dotnet/sdk	TestBuild: macOS (x64)	Log
2779688	dotnet-runtime	Performance ios_scenarios iOSMono JIT iOSLlvmBuild osx x64 perfiphone12mini net10.0	Log
2779615	dotnet-performance	Performance mlnet ubuntu 2204 arm64 Ampere 9.0	Log
2779535	dotnet-runtime	Performance ios_scenarios iOSMono JIT iOSLlvmBuild iOSStripSymbols osx x64 perfiphone12mini net10.0	Log
2779304	dotnet-runtime	Performance ios_scenarios iOSMono JIT iOSStripSymbols osx x64 perfiphone12mini net10.0	Log
2779267	dotnet-performance	Performance maui_scenarios_ios FullAOT Mono osx 15 x64 iPhoneMini12 main	Log
1130704	dotnet/runtime	osx-x64 Debug PALTests	Log
2779155	dotnet-runtime	Performance ios_scenarios iOSMono JIT iOSLlvmBuild osx x64 perfiphone12mini net10.0	Log
1130680	dotnet/runtime	Libraries Test Run release coreclr osx x64 Release	Log
2779097	dotnet-runtime	Performance ios_scenarios iOSMono JIT osx x64 perfiphone12mini net10.0	Log
2779058	dotnet-performance	Performance maui_scenarios_ios FullAOT Mono osx 15 x64 iPhoneMini12 main	Log
2778901	dotnet-runtime	Performance ios_scenarios iOSMono JIT iOSLlvmBuild iOSStripSymbols osx x64 perfiphone12mini net10.0	Log
2778778	dotnet-performance	Performance maui_scenarios_ios NativeAOT CoreCLR osx 15 x64 iPhoneMini12 main	Log
2778749	dotnet-runtime	Performance ios_scenarios iOSMono JIT iOSLlvmBuild iOSStripSymbols osx x64 perfiphone12mini net10.0	Log
2778614	dotnet-runtime	Performance ios_scenarios iOSMono JIT iOSLlvmBuild iOSStripSymbols osx x64 perfiphone12mini net10.0	Log
2778544	dotnet-runtime	Performance ios_scenarios iOSMono JIT iOSLlvmBuild osx x64 perfiphone12mini net10.0	Log
2778561	dotnet-performance	Performance maui_scenarios_ios NativeAOT CoreCLR osx 15 x64 iPhoneMini12 main	Log
2778480	dotnet-runtime	Performance ios_scenarios iOSMono JIT iOSStripSymbols osx x64 perfiphone12mini net10.0	Log
2778287	dotnet-performance	Performance maui_scenarios_ios NativeAOT CoreCLR osx 15 x64 iPhoneMini12 main	Log
2778254	dotnet-runtime	Performance ios_scenarios iOSMono JIT iOSStripSymbols osx x64 perfiphone12mini net10.0	Log
2777400	dotnet-runtime	Performance ios_scenarios iOSMono JIT iOSLlvmBuild iOSStripSymbols osx x64 perfiphone12mini net10.0	Log
2778140	dotnet-runtime	Performance ios_scenarios iOSMono JIT iOSLlvmBuild iOSStripSymbols osx x64 perfiphone12mini net10.0	Log
2778051	dotnet-runtime	Performance ios_scenarios iOSMono JIT iOSLlvmBuild iOSStripSymbols osx x64 perfiphone12mini net10.0	Log
2777909	dotnet-performance	Performance akadeindexedset windows Win11 x64 Tiger 9.0	Log
2777891	dotnet-runtime	Performance ios_scenarios iOSMono JIT iOSStripSymbols osx x64 perfiphone12mini net10.0	Log
1129513	dotnet/runtime	osx-arm64 Release NativeAOT_Libs	Log
2777783	dotnet-runtime	Performance ios_scenarios iOSMono JIT iOSLlvmBuild osx x64 perfiphone12mini net10.0	Log
2777698	dotnet-runtime	Performance ios_scenarios iOSMono JIT iOSLlvmBuild iOSStripSymbols osx x64 perfiphone12mini net10.0	Log
2777424	dotnet-performance	Performance maui_scenarios_ios FullAOT Mono osx 15 x64 iPhoneMini12 main	Log
2777585	dotnet-performance	Performance maui_scenarios_ios FullAOT Mono osx 15 x64 iPhoneMini12 main	Log
2777505	dotnet-runtime	Performance ios_scenarios iOSMono JIT iOSLlvmBuild osx x64 perfiphone12mini net10.0	Log
2777457	dotnet-runtime	Performance ios_scenarios iOSMono JIT iOSLlvmBuild iOSStripSymbols osx x64 perfiphone12mini net10.0	Log
1129022	dotnet/sdk	TestBuild: macOS (arm64)	Log	dotnet/sdk#50418
Displaying 100 of 243 results

Summary

24-Hour Hit Count	7-Day Hit Count	1-Month Count
7	81	243

[lewing]

🤔

[nagilson]

@dotnet/dnceng @dougbu This has impacted a lot of PRs recently, may you PTAL into expanding this Mac resource?

[ivanpovazan]

@dotnet/dnceng we are hitting this again and it seems there is some issue with communication with Helix machines.

More context on timeouts happening in : https://dev.azure.com/dnceng-public/public/_build/results?buildId=930014&view=results

• On success: Run tests in Helix step reports:

Waiting for completion of job fbec1164-55c5-4fa8-b492-e1ba1b413119 on osx.1200.amd64.open (Details: https://helix.dot.net/api/jobs/fbec1164-55c5-4fa8-b492-e1ba1b413119/details?api-version=2019-06-17 )
Job 7fdc2720-3d98-4bf3-8113-274dacd69c91 on osx.1200.arm64.open is completed with 6 finished work items.
  Job fbec1164-55c5-4fa8-b492-e1ba1b413119 on osx.1200.amd64.open is completed with 6 finished work items.
  Stopping Azure Pipelines Test Run Helix Tests Build_Debugosx.1200.amd64.open (Results: https://dev.azure.com/dnceng-public/public/_build/results?buildId=923987&view=ms.vss-test-web.build-test-results-tab )
  Stopping Azure Pipelines Test Run Helix Tests Build_Debugosx.1200.arm64.open (Results: https://dev.azure.com/dnceng-public/public/_build/results?buildId=923987&view=ms.vss-test-web.build-test-results-tab )

Build succeeded.

SENDHELIXJOB : warning : Helix queue osx.1200.amd64.open was set for estimated removal date of 2025-01-01. In most cases the queue will be removed permanently due to end-of-life; please contact dnceng for any questions or concerns, and we can help you decide how to proceed and discuss other options. [/home/vsts/work/1/s/tests/integration-tests/Apple/Simulator.Tests.proj]
SENDHELIXJOB : warning : Helix queue osx.1200.arm64.open was set for estimated removal date of 2025-01-01. In most cases the queue will be removed permanently due to end-of-life; please contact dnceng for any questions or concerns, and we can help you decide how to proceed and discuss other options. [/home/vsts/work/1/s/tests/integration-tests/Apple/Simulator.Tests.proj]
    2 Warning(s)
    0 Error(s)

Time Elapsed 00:03:14.97
Killing running build processes...

Finishing: Run tests in Helix

ref: https://dev.azure.com/dnceng-public/public/_build/results?buildId=923987&view=logs&j=ccc97bb6-1a23-5e71-fdfa-3cdca4a74749&t=27fc7eb2-ead9-59e1-6679-a637855d40c5

• While on failure - timeout the same step gets stuck with:

Waiting for completion of job 967d92a2-ec10-4332-927f-d28a6563f367 on osx.1200.arm64.open (Details: https://helix.dot.net/api/jobs/967d92a2-ec10-4332-927f-d28a6563f367/details?api-version=2019-06-17 )
  Job 5427159b-500d-49f1-aac0-ec148a492bbe on osx.1200.amd64.open is completed with 6 finished work items.

ref: https://dev.azure.com/dnceng-public/public/_build/results?buildId=930014&view=logs&s=c58bc33c-b825-5bca-90ca-50f6e9293dd8&j=e6966639-fe40-5068-d9ae-681cccecafdf

NOTE: All the tests successfully passed on Helix, but it seems that the communication is lost.

[garath]

Looks like the timeout happened because the osx.1200.arm64.open queue was very busy while the job was running. Right now, I do not think there are any problems with the infrastructure. I will investigate a bit more to see what caused such a back-up.

[garath]

Ah, the queue was consumed with updates and patching. The patching jobs did run longer than necessary and we've communicated with our partner team about the issue. Future jobs will be much shorter and should not overly impact jobs.

[ivanpovazan]

Looks like the timeout happened because the osx.1200.arm64.open queue was very busy while the job was running. Right now, I do not think there are any problems with the infrastructure. I will investigate a bit more to see what caused such a back-up.

We are still experiencing the problem.

Should we try to change the queue to osx.13.arm64?

@dotnet/dnceng

[ilyas1974]

Looks at the failing builds, I'm noticing the jobs that are timing out in the hosted pool (Azure Pipelines) are still using the older hardware. I would recommend moving the workloads to the mac-latest-internal or mac-14-arm64 agent specifications. These have the latest mac hardware associated with them.