Checkout failure: "Git fetch failed with exit code 128"

[carlossanlop]

Build

https://dev.azure.com/dnceng/public/_build/results?buildId=1706886&view=results

Build leg reported

Libraries Test Run release coreclr Linux_musl x64 Release

Pull Request

Rolling build for runtime

Action required for the engineering services team

To triage this issue (First Responder / @dotnet/dnceng):

• Open the failing build above and investigate
• Add a comment explaining your findings

If this is an issue that is causing build breaks across multiple builds and would get benefit from being listed on the build analysis check, follow the next steps:

1. Add the label "Known Build Error"
2. Edit this issue and add an error string in the Json below that can help us match this issue with future build breaks. You should use the known issues documentation

{
   "ErrorMessage" : "Git fetch failed with exit code: 128",
   "BuildRetry": true
}

Additional information about the issue reported

The Checkout dotnet/runtime@main to s step is the one that failed. Here is the end of the log output:

remote: Compressing objects:  29% (30070/103687)        
remote: Compressing objects:  30% (31107/103687)        
remote: Compressing objects:  31% (32143/103687)        
remote: Compressing objects:  31% (32322/103687)        
remote: Compressing objects:  32% (33180/103687)        
remote: Compressing objects:  33% (34217/103687)        
remote: Compressing objects:  33% (35247/103687)        
remote: Compressing objects:  34% (35254/103687)        
remote: Compressing objects:  35% (36291/103687)        
remote: Compressing objects:  36% (37328/103687)        
remote: Compressing objects:  37% (38365/103687)        
remote: Compressing objects:  37% (38926/103687)        
remote: Compressing objects:  38% (39402/103687)        
error: RPC failed; curl 18 transfer closed with outstanding read data remaining
fatal: protocol error: bad pack header
##[error]Git fetch failed with exit code: 128
Finishing: Checkout dotnet/runtime@main to s

Report

Build	Definition	Step Name	Console log	Pull Request
1222271	dotnet/sdk	Checkout dotnet/sdk@refs/pull/52047/merge to s	Log	dotnet/sdk#52047
1221834	dotnet/runtime	Checkout dotnet/runtime@refs/pull/122225/merge to s	Log	dotnet/runtime#122225
1221128	dotnet/sdk	Checkout dotnet/sdk@refs/pull/51847/merge to s	Log	dotnet/sdk#51847
1221004	dotnet/aspnetcore	Checkout dotnet/aspnetcore@refs/pull/64656/merge to s	Log	dotnet/aspnetcore#64656
1221002	dotnet/aspnetcore	Checkout dotnet/aspnetcore@refs/pull/64656/merge to s	Log	dotnet/aspnetcore#64656
1221003	dotnet/aspnetcore	Checkout dotnet/aspnetcore@refs/pull/64656/merge to s	Log	dotnet/aspnetcore#64656
2853848	dotnet-sdk	Clone dotnet/sdk	Log	#55829
1220360	dotnet/runtime	Checkout dotnet/runtime@refs/pull/121389/merge to s	Log	dotnet/runtime#121389
1219486	dotnet/runtime	Checkout dotnet/runtime@refs/pull/122159/merge to s	Log	dotnet/runtime#122159
1219306	dotnet/arcade	Checkout dotnet/arcade@refs/pull/16364/merge to s	Log	#16364
1218998	dotnet/aspnetcore	Checkout dotnet/aspnetcore@refs/pull/64539/merge to s	Log	dotnet/aspnetcore#64539
1219000	dotnet/aspnetcore	Checkout dotnet/aspnetcore@refs/pull/64539/merge to s	Log	dotnet/aspnetcore#64539
1219001	dotnet/aspnetcore	Checkout dotnet/aspnetcore@refs/pull/64539/merge to s	Log	dotnet/aspnetcore#64539
1217310	dotnet/aspnetcore	Checkout dotnet/aspnetcore@refs/pull/64594/merge to s	Log	dotnet/aspnetcore#64594
1216606	dotnet/runtime	Checkout dotnet/runtime@refs/pull/122083/merge to s	Log	dotnet/runtime#122083
1214953	dotnet/arcade-services	Checkout dotnet/arcade-services@refs/pull/5578/merge to s	Log	dotnet/arcade-services#5578
1214933	dotnet/arcade-services	Checkout dotnet/arcade-services@refs/pull/5584/merge to s	Log	dotnet/arcade-services#5584
1213665	dotnet/runtime	Checkout dotnet/runtime@refs/pull/121664/merge to s	Log	dotnet/runtime#121664
1213649	dotnet/runtime	Checkout dotnet/runtime@refs/pull/121591/merge to s	Log	dotnet/runtime#121591
1213644	dotnet/runtime	Checkout dotnet/runtime@refs/pull/121589/merge to s	Log	dotnet/runtime#121589
1213657	dotnet/runtime	Checkout dotnet/runtime@refs/pull/121676/merge to s	Log	dotnet/runtime#121676
1213637	dotnet/runtime	Checkout dotnet/runtime@refs/pull/121588/merge to s	Log	dotnet/runtime#121588
1213641	dotnet/runtime	Checkout dotnet/runtime@refs/pull/121587/merge to s	Log	dotnet/runtime#121587
1213010	dotnet/sdk	Checkout dotnet/sdk@refs/pull/51795/merge to s	Log	dotnet/sdk#51795
1212984	dotnet/runtime	Checkout dotnet/runtime@refs/pull/121697/merge to s	Log	dotnet/runtime#121697
1212983	dotnet/runtime	Checkout dotnet/runtime@refs/pull/121697/merge to s	Log	dotnet/runtime#121697
1212931	dotnet/runtime	Checkout dotnet/runtime@refs/pull/121952/merge to s	Log	dotnet/runtime#121952
1212962	maestro-auth-test/helix-queue-insights-test	Checkout maestro-auth-test/helix-queue-insights-test@refs/pull/1092/merge to s	Log	maestro-auth-test/helix-queue-insights-test#1092
2846765	dotnet-sdk	Checkout dotnet-sdk@refs/pull/55515/merge to s	Log	#55515
2846742	dotnet-sdk	Checkout dotnet-sdk@refs/pull/55514/merge to s	Log	#55514
1209893	dotnet/sdk	Checkout dotnet/sdk@refs/pull/50705/merge to s	Log	dotnet/sdk#50705
1208812	dotnet/runtime	Checkout dotnet/runtime@refs/pull/115502/merge to s	Log	dotnet/runtime#115502
1207513	dotnet/runtime	Checkout dotnet/runtime@refs/pull/121743/merge to s		dotnet/runtime#121743
1206903	dotnet/runtime	Checkout dotnet/runtime@refs/pull/121646/merge to s	Log	dotnet/runtime#121646
1206915	dotnet/sdk	Checkout dotnet/sdk@release/8.0.4xx to s	Log
1206822	dotnet/runtime	Checkout dotnet/runtime@refs/pull/120589/merge to s	Log	dotnet/runtime#120589
1206922	dotnet/arcade	Checkout dotnet/arcade@refs/pull/16307/merge to s	Log	#16307
1206918	dotnet/runtime	Checkout dotnet/runtime@refs/pull/119432/merge to s	Log	dotnet/runtime#119432
1206917	dotnet/runtime	Checkout dotnet/runtime@refs/pull/119432/merge to s	Log	dotnet/runtime#119432
1206919	dotnet/runtime	Checkout dotnet/runtime@refs/pull/119432/merge to s	Log	dotnet/runtime#119432
1203938	dotnet/sdk	Checkout dotnet/sdk@refs/pull/51659/merge to s	Log	dotnet/sdk#51659
1206885	dotnet/runtime	Checkout dotnet/runtime@main to s	Log
1206908	dotnet/runtime	Checkout dotnet/runtime@main to s	Log
1206906	dotnet/runtime	Checkout dotnet/runtime@main to s	Log
1206901	dotnet/runtime	Checkout dotnet/runtime@refs/pull/121646/merge to s	Log	dotnet/runtime#121646
1206902	dotnet/runtime	Checkout dotnet/runtime@refs/pull/121646/merge to s	Log	dotnet/runtime#121646
1206859	dotnet/aspnetcore	Checkout dotnet/aspnetcore@refs/pull/64426/merge to s	Log	dotnet/aspnetcore#64426
1206892	dotnet/performance	Checkout dotnet/performance@refs/pull/5035/merge to s/performance	Log	dotnet/performance#5035
1206889	dotnet/aspire	Checkout dotnet/aspire@refs/pull/13036/merge to s	Log	dotnet/aspire#13036
1206801	dotnet/sdk	Checkout dotnet/sdk@refs/pull/51419/merge to s	Log	dotnet/sdk#51419
1206601	dotnet/arcade-services	Checkout dotnet/arcade-services@refs/pull/5518/merge to s	Log	dotnet/arcade-services#5518
1206531	dotnet/arcade-services	Checkout dotnet/arcade-services@refs/pull/5517/merge to s	Log	dotnet/arcade-services#5517
1206513	dotnet/arcade-services	Checkout dotnet/arcade-services@refs/pull/5513/merge to s	Log	dotnet/arcade-services#5513
1205901	dotnet/aspnetcore	Checkout dotnet/aspnetcore@refs/pull/64271/merge to s	Log	dotnet/aspnetcore#64271
1203492	dotnet/aspnetcore	Checkout dotnet/aspnetcore@refs/pull/64369/merge to s	Log	dotnet/aspnetcore#64369
1203494	dotnet/aspnetcore	Checkout dotnet/aspnetcore@refs/pull/64369/merge to s	Log	dotnet/aspnetcore#64369
1203422	dotnet/arcade	Checkout dotnet/arcade@refs/pull/16310/merge to s	Log	#16310
1202792	dotnet/runtime	Checkout dotnet/runtime@refs/pull/121603/merge to s	Log	dotnet/runtime#121603
1202736	dotnet/runtime	Checkout dotnet/runtime@refs/pull/121334/merge to s	Log	dotnet/runtime#121334
1202427	dotnet/aspnetcore	Checkout dotnet/aspnetcore@refs/pull/64350/merge to s	Log	dotnet/aspnetcore#64350
1200180	dotnet/aspnetcore	Checkout dotnet/aspnetcore@refs/pull/64313/merge to s	Log	dotnet/aspnetcore#64313
1200181	dotnet/aspnetcore	Checkout dotnet/aspnetcore@refs/pull/64313/merge to s	Log	dotnet/aspnetcore#64313
1197636	dotnet/runtime	Checkout dotnet/runtime@refs/pull/121396/merge to s	Log	dotnet/runtime#121396
1197105	dotnet/runtime	Checkout dotnet/runtime@refs/pull/121446/merge to s		dotnet/runtime#121446

Summary

24-Hour Hit Count	7-Day Hit Count	1-Month Count
0	15	64

Known issue validation

Build: 🔎 https://dev.azure.com/dnceng-public/public/_build/results?buildId=323960
Error message validated: Git fetch failed with exit code: 128
Result validation: ✅ Known issue matched with the provided build.
Validation performed at: 6/28/2023 9:15:14 PM UTC

[ChadNedzlek]

So this is... interesting. I agree this seems like essentially a random network failure that is not the fault of the code. Also... it doesn't seem like there is anything to be done other than retry the build.

@ulisesh and @AlitzelMendez. This seems like another instance of a "known issue" that would really benefit from the ability to get retried automatically. I think Carlos did the right thing by opening the known issue, but his PR (and any future PR that hits this) is still going to be blocked, since the build doesn't run at all in this case, because the checkout failed. And it would be great to help folks that hit this.

[ulisesh]

I created the issue to track how often it was happening and measure the impact of the problem. It might be hard to get traction on networking problems but having the number of impacted builds has great value in my opinion.

In the last 24 hours, we had 37 builds failed with the same error. cc @markwilkie who might find this data interesting.

I think automatic retries for known issues will be the next feature after we are done with analyzing builds retroactively, we can discuss this if people think we should change priorities.

[markwilkie]

So 37 builds were caught with the known issues feature @ulisesh ? Could I get the query please? (super exciting)

[ulisesh]

I saw 37 hits in the office but now I only see 33 🤔. There are 33 failed jobs in 2 different builds. Sorry if I misused the terms but I still think it is great that we can track these build breaks 😄

This is the query:

KnownIssues
| where IssueId == 9009 and IssueRepository == "dotnet/arcade"

[ViktorHofer]

This happened 36 times in the last 24 hours. Should we add a retry (assuming that's possible)?