Prevent concurrent execution of the same mutable request object

[idelpivnitskiy]

Motivation:

Our HttpRequestMetaData object is mutable, and we expect users to create a new request every time they need to make a new call. Sequential retries are acceptable, but concurrent execution can corrupt internal state. While these expectations are more clear for HTTP users, with gRPC it gets less obvious that they can not subscribe to the same returned Single<Message> concurrently.

Modifications:

• Enhance FilterableClientToClient to protect users from concurrent execution of the same request, while still allowing sequential retries.
• Verify concurrent execution is not allowed for HTTP and gRPC.

Result:

Users get RejectedSubscribeException if they subscribe to the same Single that shares underlying meta-data object concurrently. This is the best effort to let users know they misused the client.

Risk:

The change may unexpectedly break existing use-cases. To give users some time to adjust their code, we temporarily introduce a system property to opt-out from this new behavior: -Dio.servicetalk.http.netty.skipConcurrentRequestCheck=true.

[bryce-anderson]

I don't really like the synchronization so if someone has a better idea I'm interested to hear it. However, I can't think of a better solution that doesn't break behavior so I'm fine with the compromises this makes since it's so easy to back out of the change (both in terms of flag and in terms of we can easily revert and go back to the drawing board if this isn't effective enough etc).

[mgodave]

I don't really like the synchronization so if someone has a better idea I'm interested to hear it. However, I can't think of a better solution that doesn't break behavior so I'm fine with the compromises this makes since it's so easy to back out of the change (both in terms of flag and in terms of we can easily revert and go back to the drawing board if this isn't effective enough etc).

My initial reaction was that I wasn't thrilled about the mutability and I didn't really have anywhere to go from there having not directly thought about this problem. I think this is probably about as good as we can achieve otherwise.