HttpRequestException after update to .NET 8

[peroyhav]

After updating to .NET 8, we see that we get a new HttpRequestException on Grpc calls to Bigtable, not sure if the issue is in Grpc.net or in google Bigtable. This issue only exists after upgrading to .NET 8.

It seems the error is intermittent, and it looks like it happens during high request periods of our application.

What version of gRPC and what language are you using?

Grpc: 2.46.6
Grpc.Tools: 2.64.0
Google.Protobuf: 3.26.1
Google.Protobuf.Tools: 3.26.1
Google.Cloud.Bigtable.V2: 3.13.0

What operating system (Linux, Windows,...) and version?

Docker Ubuntu aspnet image

What runtime / compiler are you using (e.g. .NET Core SDK version dotnet --info)

Host:
Version: 8.0.6
Architecture: x64
Commit: 3b8b000a0e
RID: linux-x64

.NET SDKs installed:
No SDKs were found.

.NET runtimes installed:
Microsoft.AspNetCore.App 8.0.6 [/usr/share/dotnet/shared/Microsoft.AspNetCore.App]
Microsoft.NETCore.App 8.0.6 [/usr/share/dotnet/shared/Microsoft.NETCore.App]

Other architectures found:
None

Environment variables:
Not set

global.json file:
Not found

Learn more:
https://aka.ms/dotnet/info

Download .NET:
https://aka.ms/dotnet/download

What did you do?

Issue seem to only exists in kubernetes cluster, and seem to be intermittent. Not able to reproduce on Windows or Linux Mint while debugging. Have also tried to build Docker image on top of Focal with the same packages as .NET 6 runs on to check if it's an issue with updated linux packages as libssl3 instead of libssl1.1. Have tested

What did you expect to see?

No new issues with GRPC after upgrading to .NET 8

What did you see instead?

Exception as noted below

Error message:
Status(StatusCode="Internal", Detail="Error starting gRPC call. HttpRequestException: The HTTP/2 server sent invalid data on the connection. HTTP/2 error code 'PROTOCOL_ERROR' (0x1). (HttpProtocolError) HttpProtocolException: The HTTP/2 server sent invalid data on the connection. HTTP/2 error code 'PROTOCOL_ERROR' (0x1). (HttpProtocolError)", DebugException="System.Net.Http.HttpRequestException: The HTTP/2 server sent invalid data on the connection. HTTP/2 error code 'PROTOCOL_ERROR' (0x1). (HttpProtocolError)") The HTTP/2 server sent invalid data on the connection. HTTP/2 error code 'PROTOCOL_ERROR' (0x1). (HttpProtocolError) The HTTP/2 server sent invalid data on the connection. HTTP/2 error code 'PROTOCOL_ERROR' (0x1). (HttpProtocolError)

Stack trace:
Grpc.Core.RpcException:
   at Grpc.Net.Client.Internal.HttpContentClientStreamReader`2+<MoveNextCore>d__18.MoveNext (Grpc.Net.Client, Version=2.0.0.0, Culture=neutral, PublicKeyToken=d754f35622e28bad)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at Google.Api.Gax.Grpc.Gcp.GcpClientResponseStream`2+<MoveNext>d__7.MoveNext (Google.Api.Gax.Grpc, Version=4.8.0.0, Culture=neutral, PublicKeyToken=3ec5ea7f18953e47)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at Google.Cloud.Bigtable.V2.RowAsyncEnumerator+<MoveNextAsync>d__23.MoveNext (Google.Cloud.Bigtable.V2, Version=3.10.0.0, Culture=neutral, PublicKeyToken=185c282632e132a0)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Linq.AsyncEnumerable+SelectEnumerableAsyncIterator`2+<MoveNextCore>d__7.MoveNext (System.Linq.Async, Version=6.0.0.0, Culture=neutral, PublicKeyToken=94bc3704cddfc263: /_/Ix.NET/Source/System.Linq.Async/System/Linq/Operators/Select.cs:221)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Linq.AsyncIteratorBase`1+<MoveNextAsync>d__8.MoveNext (System.Linq.Async, Version=6.0.0.0, Culture=neutral, PublicKeyToken=94bc3704cddfc263: /_/Ix.NET/Source/System.Linq.Async/System/Linq/AsyncIterator.cs:70)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Linq.AsyncIteratorBase`1+<MoveNextAsync>d__8.MoveNext (System.Linq.Async, Version=6.0.0.0, Culture=neutral, PublicKeyToken=94bc3704cddfc263: /_/Ix.NET/Source/System.Linq.Async/System/Linq/AsyncIterator.cs:75)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Linq.AsyncEnumerable+WhereEnumerableAsyncIterator`1+<MoveNextCore>d__8.MoveNext (System.Linq.Async, Version=6.0.0.0, Culture=neutral, PublicKeyToken=94bc3704cddfc263: /_/Ix.NET/Source/System.Linq.Async/System/Linq/Operators/Where.cs:233)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Linq.AsyncIteratorBase`1+<MoveNextAsync>d__8.MoveNext (System.Linq.Async, Version=6.0.0.0, Culture=neutral, PublicKeyToken=94bc3704cddfc263: /_/Ix.NET/Source/System.Linq.Async/System/Linq/AsyncIterator.cs:70)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Linq.AsyncIteratorBase`1+<MoveNextAsync>d__8.MoveNext (System.Linq.Async, Version=6.0.0.0, Culture=neutral, PublicKeyToken=94bc3704cddfc263: /_/Ix.NET/Source/System.Linq.Async/System/Linq/AsyncIterator.cs:75)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Collections.Generic.AsyncEnumerableHelpers+<ToArrayWithLength>d__1`1.MoveNext (System.Linq.Async, Version=6.0.0.0, Culture=neutral, PublicKeyToken=94bc3704cddfc263: /_/Ix.NET/Source/System.Linq.Async/System/Linq/AsyncEnumerableHelpers.cs:49)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Collections.Generic.AsyncEnumerableHelpers+<ToArrayWithLength>d__1`1.MoveNext (System.Linq.Async, Version=6.0.0.0, Culture=neutral, PublicKeyToken=94bc3704cddfc263: /_/Ix.NET/Source/System.Linq.Async/System/Linq/AsyncEnumerableHelpers.cs:91)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Threading.Tasks.ValueTask`1.get_Result (System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Collections.Generic.AsyncEnumerableHelpers+<ToArray>d__0`1.MoveNext (System.Linq.Async, Version=6.0.0.0, Culture=neutral, PublicKeyToken=94bc3704cddfc263)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at Infrastructure.Repositories.ValueRepository+<GetValues>d__13.MoveNext (Infrastructure, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null: /app/src/Infrastructure/Repositories/ValueRepository.cs:257)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at Infrastructure.Repositories.ValuesRepository+<GetValues>d__7.MoveNext (Infrastructure, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null: /app/src/Infrastructure/Repositories/ValuesRepository.cs:130)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at Infrastructure.Repositories.ValuesRepository+<GetValues>d__6.MoveNext (Infrastructure, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null: /app/src/Infrastructure/ValuesRepository.cs:44)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at Core.Services.ValuesService+<ProcessMeterValues>d__5.MoveNext (Core, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null: /app/src/Core/Services/ValuesService.cs:54)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at Api.Services.Calculation.CalculationHandler+<HandleValueCalculation>d__3.MoveNext (Api, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null: /app/src/Api/Services/Calculation/CalculationHandler.cs:50)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at Api.Services.Calculation.Internal.AbstractCalculationHandler+<HandleValueCalculation>d__22.MoveNext (Api, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null: /app/src/Api/Services/Calculation/Internal/AbstractCalculationHandler.cs:57)
Inner exception System.Net.Http.HttpRequestException handled at Grpc.Net.Client.Internal.HttpContentClientStreamReader`2+<MoveNextCore>d__18.MoveNext:
   at System.Net.Http.Http2Connection+<SendAsync>d__111.MoveNext (System.Net.Http, Version=8.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Net.Http.HttpConnectionPool+<SendWithVersionDetectionAndRetryAsync>d__89.MoveNext (System.Net.Http, Version=8.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Net.Http.DiagnosticsHandler+<SendAsyncCore>d__10.MoveNext (System.Net.Http, Version=8.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Net.Http.RedirectHandler+<SendAsync>d__4.MoveNext (System.Net.Http, Version=8.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at Grpc.Net.Client.Balancer.Internal.BalancerHttpHandler+<SendAsync>d__11.MoveNext (Grpc.Net.Client, Version=2.0.0.0, Culture=neutral, PublicKeyToken=d754f35622e28bad)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at Grpc.Net.Client.Internal.GrpcCall`2+<RunCall>d__82.MoveNext (Grpc.Net.Client, Version=2.0.0.0, Culture=neutral, PublicKeyToken=d754f35622e28bad)
Inner exception System.Net.Http.HttpProtocolException handled at System.Net.Http.Http2Connection+<SendAsync>d__111.MoveNext:
   at System.Net.Http.Http2Connection.ThrowRequestAborted (System.Net.Http, Version=8.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a)
   at System.Net.Http.Http2Connection+Http2Stream.TryEnsureHeaders (System.Net.Http, Version=8.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a)
   at System.Net.Http.Http2Connection+Http2Stream+<ReadResponseHeadersAsync>d__74.MoveNext (System.Net.Http, Version=8.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Net.Http.Http2Connection+<SendAsync>d__111.MoveNext (System.Net.Http, Version=8.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a)

[pankajkumarjha-kongsbergdigital]

I am getting below error after upgrading my application to .Net8. This happens every time I run the test on ubuntu VM. The application works fine on docker hosted on windows. Looks to be related.

Grpc.Core.RpcException : Status(StatusCode="Internal", Detail="Error starting gRPC call. HttpRequestException: An HTTP/2 connection could not be established because the server did not complete the HTTP/2 handshake.", DebugException="System.Net.Http.HttpRequestException: An HTTP/2 connection could not be established because the server did not complete the HTTP/2 handshake.")
  ----> System.Net.Http.HttpRequestException : An HTTP/2 connection could not be established because the server did not complete the HTTP/2 handshake.

Is there anything we need in .Net8 ?

[pankajkumarjha-kongsbergdigital]

I am getting below error after upgrading my application to .Net8. This happens every time I run the test on ubuntu VM. The application works fine on docker hosted on windows. Looks to be related.

Grpc.Core.RpcException : Status(StatusCode="Internal", Detail="Error starting gRPC call. HttpRequestException: An HTTP/2 connection could not be established because the server did not complete the HTTP/2 handshake.", DebugException="System.Net.Http.HttpRequestException: An HTTP/2 connection could not be established because the server did not complete the HTTP/2 handshake.")
  ----> System.Net.Http.HttpRequestException : An HTTP/2 connection could not be established because the server did not complete the HTTP/2 handshake.

Is there anything we need in .Net8 ?

This error was due to breaking change in .NET 8 and got fixed now after implementing the recommended changes from the below link.
https://learn.microsoft.com/en-us/dotnet/core/compatibility/containers/8.0/aspnet-port#new-behavior

[WeihanLi]

Is there a minimized repro code snippet