Checkout failure: "Git fetch failed with exit code 128"

[carlossanlop]

Build

https://dev.azure.com/dnceng/public/_build/results?buildId=1706886&view=results

Build leg reported

Libraries Test Run release coreclr Linux_musl x64 Release

Pull Request

Rolling build for runtime

Action required for the engineering services team

To triage this issue (First Responder / @dotnet/dnceng):

• Open the failing build above and investigate
• Add a comment explaining your findings

If this is an issue that is causing build breaks across multiple builds and would get benefit from being listed on the build analysis check, follow the next steps:

1. Add the label "Known Build Error"
2. Edit this issue and add an error string in the Json below that can help us match this issue with future build breaks. You should use the known issues documentation

{
   "ErrorMessage" : "Git fetch failed with exit code: 128",
   "BuildRetry": true
}

Additional information about the issue reported

The Checkout dotnet/runtime@main to s step is the one that failed. Here is the end of the log output:

remote: Compressing objects:  29% (30070/103687)        
remote: Compressing objects:  30% (31107/103687)        
remote: Compressing objects:  31% (32143/103687)        
remote: Compressing objects:  31% (32322/103687)        
remote: Compressing objects:  32% (33180/103687)        
remote: Compressing objects:  33% (34217/103687)        
remote: Compressing objects:  33% (35247/103687)        
remote: Compressing objects:  34% (35254/103687)        
remote: Compressing objects:  35% (36291/103687)        
remote: Compressing objects:  36% (37328/103687)        
remote: Compressing objects:  37% (38365/103687)        
remote: Compressing objects:  37% (38926/103687)        
remote: Compressing objects:  38% (39402/103687)        
error: RPC failed; curl 18 transfer closed with outstanding read data remaining
fatal: protocol error: bad pack header
##[error]Git fetch failed with exit code: 128
Finishing: Checkout dotnet/runtime@main to s

Report

Build	Definition	Step Name	Console log	Pull Request
1409828	dotnet/runtime	Checkout dotnet/runtime@refs/pull/127819/merge to s	Log	dotnet/runtime#127819
1409469	dotnet/runtime	Checkout dotnet/runtime@refs/pull/127826/merge to s	Log	dotnet/runtime#127826
2967543	dotnet-maui	Checkout dotnet-maui@net11.0 to s	Log
1408595	dotnet/sdk	Checkout dotnet/sdk@refs/pull/53946/merge to s	Log	dotnet/sdk#53946
1408639	dotnet/sdk	Checkout dotnet/sdk@refs/pull/53954/merge to s	Log	dotnet/sdk#53954
1408359	dotnet/arcade-services	Checkout dotnet/arcade-services@refs/pull/6257/merge to s	Log	dotnet/arcade-services#6257
1408354	dotnet/arcade-services	Checkout dotnet/arcade-services@refs/pull/6253/merge to s	Log	dotnet/arcade-services#6253
1407663	dotnet/runtime	Checkout dotnet/runtime@refs/pull/127754/merge to s	Log	dotnet/runtime#127754
1407647	dotnet/runtime	Checkout dotnet/runtime@refs/pull/127760/merge to s	Log	dotnet/runtime#127760
1407588	dotnet/runtime	Checkout dotnet/runtime@refs/pull/127697/merge to s	Log	dotnet/runtime#127697
1407679	dotnet/runtime	Checkout dotnet/runtime@refs/pull/127763/merge to s	Log	dotnet/runtime#127763
1407587	dotnet/runtime	Checkout dotnet/runtime@refs/pull/127697/merge to s	Log	dotnet/runtime#127697
1407481	dotnet/runtime	Checkout dotnet/runtime@refs/pull/127748/merge to s	Log	dotnet/runtime#127748
1407609	dotnet/runtime	Checkout dotnet/runtime@refs/pull/127755/merge to s	Log	dotnet/runtime#127755
1407715	dotnet/maui	Checkout dotnet/maui@refs/pull/34621/merge to s	Log	dotnet/maui#34621
1407637	dotnet/runtime	Checkout dotnet/runtime@refs/pull/127744/merge to s	Log	dotnet/runtime#127744
1407405	dotnet/runtime	Checkout dotnet/runtime@refs/pull/127016/merge to s	Log	dotnet/runtime#127016
1407554	dotnet/runtime	Checkout dotnet/runtime@refs/pull/127743/merge to s	Log	dotnet/runtime#127743
1407595	dotnet/runtime	Checkout dotnet/runtime@refs/pull/127749/merge to s	Log	dotnet/runtime#127749
1407644	dotnet/sdk	Checkout dotnet/sdk@dev/nikolam/aot.fix to s	Log
1407563	dotnet/runtime	Checkout dotnet/runtime@refs/pull/127112/merge to s	Log	dotnet/runtime#127112
1407556	dotnet/runtime	Checkout dotnet/runtime@refs/pull/127743/merge to s	Log	dotnet/runtime#127743
1407372	dotnet/arcade-services	Checkout dotnet/arcade-services@refs/pull/6251/merge to s	Log	dotnet/arcade-services#6251
1406665	dotnet/runtime	Checkout dotnet/runtime@refs/pull/117261/merge to s	Log	dotnet/runtime#117261
2965456	dotnet-aspnetcore	Checkout dotnet-aspnetcore@refs/pull/60696/merge to s	Log	#60696
2965400	dotnet-aspnetcore	Checkout dotnet-aspnetcore@refs/pull/60692/merge to s	Log	#60692
1405511	dotnet/runtime	Checkout dotnet/runtime@refs/pull/127492/merge to s	Log	dotnet/runtime#127492
1405258	dotnet/aspnetcore	Checkout dotnet/aspnetcore@refs/pull/66563/merge to s	Log	dotnet/aspnetcore#66563
1404534	dotnet/runtime	Checkout dotnet/runtime@refs/pull/127256/merge to s	Log	dotnet/runtime#127256
2964373	dotnet-dotnet	Checkout dotnet-dotnet@refs/pull/60662/merge to s	Log	#60662
1403979	dotnet/sdk	Checkout dotnet/sdk@refs/pull/53847/merge to s	Log	dotnet/sdk#53847
1403764	dotnet/runtime	Checkout dotnet/runtime@refs/pull/127419/merge to s	Log	dotnet/runtime#127419
2964260	dotnet-runtime	Checkout dotnet-runtime@main to s	Log
1403371	dotnet/maui	Checkout dotnet/maui@refs/pull/35234/merge to s	Log	dotnet/maui#35234
1403417	dotnet/runtime	Checkout dotnet/runtime@refs/pull/127558/merge to s	Log	dotnet/runtime#127558
1402396	dotnet/sdk	Checkout dotnet/sdk@refs/pull/54149/merge to s	Log	dotnet/sdk#54149
1402458	dotnet/sdk	Checkout dotnet/sdk@refs/pull/54048/merge to s	Log	dotnet/sdk#54048
1402834	dotnet/runtime	Checkout dotnet/runtime@refs/pull/127587/merge to s	Log	dotnet/runtime#127587
2963673	dotnet-dotnet	Checkout dotnet-dotnet@refs/pull/60608/merge to s	Log	#60608
1401645	dotnet/runtime	Checkout dotnet/runtime@refs/pull/126649/merge to s	Log	dotnet/runtime#126649
1400945	dotnet/runtime	Checkout dotnet/runtime@refs/pull/127525/merge to s	Log	dotnet/runtime#127525
1400913	dotnet/runtime	Checkout dotnet/runtime@refs/pull/125955/merge to s	Log	dotnet/runtime#125955
1400574	dotnet/runtime	Checkout dotnet/runtime@refs/pull/127529/merge to s	Log	dotnet/runtime#127529
1400374	dotnet/sdk	Checkout dotnet/sdk@refs/pull/54131/merge to s	Log	dotnet/sdk#54131
1399187	dotnet/runtime	Checkout dotnet/runtime@refs/pull/125596/merge to s	Log	dotnet/runtime#125596
1399054	dotnet/runtime	Checkout dotnet/runtime@refs/pull/127481/merge to s	Log	dotnet/runtime#127481
1398675	dotnet/runtime	Checkout dotnet/runtime@refs/pull/127028/merge to s	Log	dotnet/runtime#127028
1398150	dotnet/arcade	Checkout dotnet/arcade@refs/pull/16754/merge to s	Log	#16754
2961186	dotnet-maui	Checkout dotnet-maui@main to s	Log
1397302	dotnet/sdk	Checkout dotnet/sdk@refs/pull/54074/merge to s	Log	dotnet/sdk#54074
1395978	dotnet/runtime	Checkout dotnet/runtime@refs/pull/127411/merge to s	Log	dotnet/runtime#127411
1395831	dotnet/runtime	Checkout dotnet/runtime@refs/pull/127403/merge to s	Log	dotnet/runtime#127403
1395700	dotnet/runtime	Checkout dotnet/runtime@refs/pull/127399/merge to s	Log	dotnet/runtime#127399
2959579	dotnet-dotnet	Checkout dotnet-dotnet@refs/pull/60428/merge to s	Log	#60428
1395170	dotnet/runtime	Checkout dotnet/runtime@refs/pull/127388/merge to s	Log	dotnet/runtime#127388
1395207	dotnet/runtime	Checkout dotnet/runtime@refs/pull/126701/merge to s	Log	dotnet/runtime#126701
1394113	dotnet/arcade	Checkout dotnet/arcade@refs/pull/16750/merge to s	Log	#16750
1394114	dotnet/arcade	Checkout dotnet/arcade@refs/pull/16751/merge to s	Log	#16751
1393975	dotnet/runtime	Checkout dotnet/runtime@refs/pull/127352/merge to s	Log	dotnet/runtime#127352
1394066	dotnet/installer	Checkout dotnet/installer@refs/pull/20856/merge to s	Log	dotnet/installer#20856
1394065	dotnet/installer	Checkout dotnet/installer@refs/pull/20855/merge to s	Log	dotnet/installer#20855
1394112	dotnet/arcade	Checkout dotnet/arcade@refs/pull/16749/merge to s	Log	#16749
1393800	dotnet/arcade	Checkout dotnet/arcade@refs/pull/16743/merge to s	Log	#16743
1391397	dotnet/runtime	Checkout dotnet/runtime@refs/pull/127288/merge to s	Log	dotnet/runtime#127288
1391392	dotnet/runtime	Checkout dotnet/runtime@refs/pull/127263/merge to s	Log	dotnet/runtime#127263
1391442	dotnet/sdk	Checkout dotnet/sdk@refs/pull/54043/merge to s	Log	dotnet/sdk#54043
1391181	dotnet/aspnetcore	Checkout dotnet/aspnetcore@refs/pull/66417/merge to s	Log	dotnet/aspnetcore#66417
1391145	dotnet/sdk	Checkout dotnet/sdk@refs/pull/54039/merge to s	Log	dotnet/sdk#54039
1391157	dotnet/runtime	Checkout dotnet/runtime@refs/pull/125651/merge to s	Log	dotnet/runtime#125651
1391130	dotnet/runtime	Checkout dotnet/runtime@refs/pull/127278/merge to s	Log	dotnet/runtime#127278
1390512	dotnet/runtime	Checkout dotnet/runtime@main to s	Log
1390331	dotnet/runtime	Checkout dotnet/runtime@refs/pull/127252/merge to s	Log	dotnet/runtime#127252
1389781	dotnet/runtime	Checkout dotnet/runtime@refs/pull/127233/merge to s	Log	dotnet/runtime#127233
1386132	dotnet/runtime	Checkout dotnet/runtime@refs/pull/127119/merge to s	Log	dotnet/runtime#127119
1384924	dotnet/runtime	Checkout dotnet/runtime@refs/pull/126596/merge to s	Log	dotnet/runtime#126596
1385054	dotnet/runtime	Checkout dotnet/runtime@refs/pull/127100/merge to s	Log	dotnet/runtime#127100
1384746	dotnet/runtime	Checkout dotnet/runtime@refs/pull/126752/merge to s	Log	dotnet/runtime#126752
1384205	maestro-auth-test/build-result-analysis-test	Checkout maestro-auth-test/build-result-analysis-test@refs/pull/2532/merge to s	Log	maestro-auth-test/build-result-analysis-test#2532
1384206	maestro-auth-test/build-result-analysis-test	Checkout maestro-auth-test/build-result-analysis-test@refs/pull/2532/merge to s	Log	maestro-auth-test/build-result-analysis-test#2532
1384202	maestro-auth-test/build-result-analysis-test	Checkout maestro-auth-test/build-result-analysis-test@refs/pull/2531/merge to s	Log	maestro-auth-test/build-result-analysis-test#2531
1384201	maestro-auth-test/build-result-analysis-test	Checkout maestro-auth-test/build-result-analysis-test@refs/pull/2531/merge to s	Log	maestro-auth-test/build-result-analysis-test#2531
1383814	dotnet/runtime	Checkout dotnet/runtime@refs/pull/127056/merge to s	Log	dotnet/runtime#127056
2953522	dotnet-dotnet	Checkout dotnet-dotnet@refs/pull/60202/merge to s	Log	#60202
1383137	dotnet/runtime	Checkout dotnet/runtime@refs/pull/127032/merge to s	Log	dotnet/runtime#127032
1382907	dotnet/installer	Checkout dotnet/installer@refs/pull/20848/merge to s	Log	dotnet/installer#20848
1382747	dotnet/runtime	Checkout dotnet/runtime@refs/pull/127024/merge to s	Log	dotnet/runtime#127024
1382359	dotnet/runtime	Checkout dotnet/runtime@refs/pull/127018/merge to s	Log	dotnet/runtime#127018
1382103	dotnet/runtime	Checkout dotnet/runtime@refs/pull/126959/merge to s	Log	dotnet/runtime#126959
1381750	dotnet/maui	Checkout dotnet/maui@refs/pull/34907/merge to s	Log	dotnet/maui#34907
1381977	dotnet/runtime	Checkout dotnet/runtime@refs/pull/127003/merge to s	Log	dotnet/runtime#127003
1381933	maestro-auth-test/build-result-analysis-test	Checkout maestro-auth-test/build-result-analysis-test@refs/pull/2527/merge to s	Log	maestro-auth-test/build-result-analysis-test#2527
1381932	maestro-auth-test/build-result-analysis-test	Checkout maestro-auth-test/build-result-analysis-test@refs/pull/2527/merge to s	Log	maestro-auth-test/build-result-analysis-test#2527
2952528	dotnet-dotnet	Checkout dotnet-dotnet@release/10.0.1xx to s	Log
1381131	dotnet/aspnetcore	Checkout dotnet/aspnetcore@refs/pull/66333/merge to s	Log	dotnet/aspnetcore#66333
1381038	dotnet/runtime	Checkout dotnet/runtime@refs/pull/126971/merge to s	Log	dotnet/runtime#126971
1380587	dotnet/runtime	Checkout dotnet/runtime@refs/pull/126959/merge to s	Log	dotnet/runtime#126959
1379927	dotnet/sdk	Checkout dotnet/sdk@refs/pull/53883/merge to s	Log	dotnet/sdk#53883
1379856	dotnet/runtime	Checkout dotnet/runtime@refs/pull/125516/merge to s	Log	dotnet/runtime#125516
1379734	dotnet/runtime	Checkout dotnet/runtime@refs/pull/126937/merge to s	Log	dotnet/runtime#126937
1378920	dotnet/aspnetcore	Checkout dotnet/aspnetcore@refs/pull/66317/merge to s	Log	dotnet/aspnetcore#66317
Displaying 100 of 120 results

Summary

24-Hour Hit Count	7-Day Hit Count	1-Month Count
2	39	120

Known issue validation

Build: 🔎 https://dev.azure.com/dnceng-public/public/_build/results?buildId=323960
Error message validated: Git fetch failed with exit code: 128
Result validation: ✅ Known issue matched with the provided build.
Validation performed at: 6/28/2023 9:15:14 PM UTC

[ChadNedzlek]

So this is... interesting. I agree this seems like essentially a random network failure that is not the fault of the code. Also... it doesn't seem like there is anything to be done other than retry the build.

@ulisesh and @AlitzelMendez. This seems like another instance of a "known issue" that would really benefit from the ability to get retried automatically. I think Carlos did the right thing by opening the known issue, but his PR (and any future PR that hits this) is still going to be blocked, since the build doesn't run at all in this case, because the checkout failed. And it would be great to help folks that hit this.

[ulisesh]

I created the issue to track how often it was happening and measure the impact of the problem. It might be hard to get traction on networking problems but having the number of impacted builds has great value in my opinion.

In the last 24 hours, we had 37 builds failed with the same error. cc @markwilkie who might find this data interesting.

I think automatic retries for known issues will be the next feature after we are done with analyzing builds retroactively, we can discuss this if people think we should change priorities.

[markwilkie]

So 37 builds were caught with the known issues feature @ulisesh ? Could I get the query please? (super exciting)

[ulisesh]

I saw 37 hits in the office but now I only see 33 🤔. There are 33 failed jobs in 2 different builds. Sorry if I misused the terms but I still think it is great that we can track these build breaks 😄

This is the query:

KnownIssues
| where IssueId == 9009 and IssueRepository == "dotnet/arcade"

[ViktorHofer]

This happened 36 times in the last 24 hours. Should we add a retry (assuming that's possible)?