System.ObjectDisposedException exception in OwinHttpListener.ProcessRequestsAsync()

[VidyaKukke]

Using RuntimeVersion: v4.0.30319 for Microsoft.Owin, Microsoft.Owin.Host.HttpListener and Microsoft.Owin.Hosting.

We are seeing an objectdisposedexception in Owin that is causing a process crash with the following call stack

Description: The process was terminated due to an unhandled exception.
Exception Info: System.ObjectDisposedException
   at System.Net.HttpListenerResponse.CheckDisposed()
   at System.Net.HttpListenerResponse.set_StatusCode(Int32)
   at Microsoft.Owin.Host.HttpListener.RequestProcessing.OwinHttpListenerResponse.End()
   at Microsoft.Owin.Host.HttpListener.OwinHttpListener+<ProcessRequestAsync>d__5.MoveNext()
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(System.Threading.Tasks.Task)
   at Microsoft.Owin.Host.HttpListener.OwinHttpListener+<ProcessRequestsAsync>d__0.MoveNext()
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean)
   at System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean)
   at System.Threading.QueueUserWorkItemCallback.System.Threading.IThreadPoolWorkItem.ExecuteWorkItem()
   at System.Threading.ThreadPoolWorkQueue.Dispatch()

We suspect 2 places that might be triggering this:

1. We receive "shutdown" signals and as part of our shutdown process we dispose the serverHandle (returned by WebApp.Start).
2. We sometimes forcibly need close connections by doing:

var owinContext = request.GetOwinContext();
owinContext.Get<HttpListenerContext>("System.Net.HttpListenerContext").Response.Abort();

We are suspecting #2 as more likely the reason but not sure how to not make Owin crash? Any suggestions?

[Tratcher]

That's problematic. I agree it's probably the call to Response.Abort that puts the response in an unexpected state and later triggers the ODE on this call stack.

Can you share more about why you're aborting the response?

The main logic conflict is here:

AspNetKatana/src/Microsoft.Owin.Host.HttpListener/RequestProcessing/OwinHttpListenerResponse.cs

Lines 298 to 304 in d196e78

	int priorState = Interlocked.Exchange(ref _requestState, Completed);
	if (priorState == RequestInProgress)
	{
	// Premature ending, must be an error.
	// If the response has not started yet then we can send an error response before closing it.
	_context.Response.StatusCode = 500;

OwinHttpListenerResponse thinks the response is still active and that it can set an error status code, but HttpListenerResponse disagrees.

The simplest workaround seems to be setting your own error response and calling Flush on the response body to send it, and then throw an exception from your app for the server to catch and abort the response.

AspNetKatana/src/Microsoft.Owin.Host.HttpListener/OwinHttpListener.cs

Lines 261 to 269 in d196e78

	catch (Exception ex)
	{
	Interlocked.Decrement(ref _currentOutstandingRequests);
	LogHelper.LogException(_logger, "Exception during request processing.", ex);
	if (owinContext != null)
	{
	owinContext.End(ex);
	owinContext.Dispose();

[VidyaKukke]

Thanks Tratcher. The reason for introducing abort was that we have to terminate our connection with LoadBalancer when we can no longer process requests (something else in the system tells us to stop processing requests). Just informing the LoadBalancer does not terminate the connection and the only way to terminate the connection for sure was to Abort the connection.

[Tratcher]

The load balancer doesn't respect an HTTP/1.x Conneciton: close response header?

[VidyaKukke]

Based on our investigation didn't seem to. The connection continued to remain open. I might have to look into this again and see if there is a gap. In the meantime, will also try the earlier option of setting response and throwing an exception.

[VidyaKukke]

We are sending "Connection: close", however any existing connections were not being closed by the Client (here we are not sure if the client is a .NET Client/ Java client/other). LB we know does not close any existing connections. As a workaround, we wanted to forcibly close the connection from the service side and seems like we used an earlier thread to use the Abort() option. https://stackoverflow.com/questions/61995209/how-to-close-a-tcp-connection-of-the-http-request-from-self-hosted-owin-server-i. Turns out this has unexpected side-effect so will try the new option suggested earlier on this thread.