A lot of time spent in Result execution with eventual exception in kestrel

[senioroman4uk]

Hi, I was trying to diagnose an issue that we observe for a small population of slow failing requests in production (less then 1%).
Our service is hosted on Kubernetes Linux (Azure Linux V2). From global traffic routing we use Azure Traffic Manager, so clients connect to the service directly.

We use mariner base images mcr.microsoft.com/dotnet/aspnet:8.0-cbl-mariner2.0

dotnet --info

Host:
  Version:      8.0.8
  Architecture: x64
  Commit:       08338fcaa5
  RID:          linux-x64
 
.NET SDKs installed:
  No SDKs were found.
 
.NET runtimes installed:
  Microsoft.AspNetCore.App 8.0.8 [/usr/share/dotnet/shared/Microsoft.AspNetCore.App]
  Microsoft.NETCore.App 8.0.8 [/usr/share/dotnet/shared/Microsoft.NETCore.App]
 
Other architectures found:
  None
 
Environment variables:
  Not set
 
global.json file:
  Not found

We log duration of various parts of MVC pipeline via filters (action, result and resource filter) and also measure an entire request duration via middleware. For these slow requests we observe a lot of time spent in result execution (for example 969520 ms) with an eventual exception from kestrel

 at System.Net.Security.SslStream.WriteAsyncInternal[TIOAdapter](ReadOnlyMemory`1 buffer, CancellationToken cancellationToken)
   at System.IO.Pipelines.StreamPipeWriter.FlushAsyncInternal(Boolean writeToStream, ReadOnlyMemory`1 data, CancellationToken cancellationToken)
   at System.Runtime.CompilerServices.PoolingAsyncValueTaskMethodBuilder`1.StateMachineBox`1.System.Threading.Tasks.Sources.IValueTaskSource<TResult>.GetResult(Int16 token)
   at System.IO.Pipelines.StreamPipeWriter.CompleteAsync(Exception exception)
   at System.IO.Pipelines.StreamPipeWriter.CompleteAsync(Exception exception)
   at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.DuplexPipeStreamAdapter`1.DisposeAsync()
   at Microsoft.AspNetCore.Server.Kestrel.Https.Internal.HttpsConnectionMiddleware.OnConnectionAsync(ConnectionContext context)
   at Microsoft.AspNetCore.Server.Kestrel.Https.Internal.HttpsConnectionMiddleware.OnConnectionAsync(ConnectionContext context)
   at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Infrastructure.KestrelConnection`1.ExecuteAsync()
InnerException type:System.Net.Sockets.SocketException message:No route to host stack:   at System.IO.Pipelines.Pipe.WriteAsync(ReadOnlyMemory`1 source, CancellationToken cancellationToken)
   at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.DuplexPipeStream.WriteAsync(ReadOnlyMemory`1 source, CancellationToken cancellationToken)
   at System.Net.Security.SslStream.WriteSingleChunk[TIOAdapter](ReadOnlyMemory`1 buffer, CancellationToken cancellationToken)
   at System.Net.Security.SslStream.WriteAsyncInternal[TIOAdapter](ReadOnlyMemory`1 buffer, CancellationToken cancellationToken)

I would like to get some help identifying the root cause of this issue

[captainsafia]

@senioroman4uk Can you share more details about whether or not this bottleneck occurs for a specific result type? I know that getting minimal repros for these kinds of issues can be hard but additional info about when it happens would be appreciated.

[dotnet-policy-service]

Hi @senioroman4uk. We have added the "Needs: Author Feedback" label to this issue, which indicates that we have an open question for you before we can take further action. This issue will be closed automatically in 7 days if we do not hear back from you by then - please feel free to re-open it if you come back to this issue after that time.