Default KeepAliveInterval gets "Detected stale connection to broker" and "Failed to write on connection" errors

[gunli]

Expected behavior

Default KeepAliveInterval should work well.

Actual behavior

Gets "Detected stale connection to broker" and "Failed to write on connection" errors

Steps to reproduce

1. I have about 6 producer services(about 100 Pods) that produce messages to different Pulsar server clusters;
2. All the producers' KeepAliveInterval are set to default(30s), and the values of the servers are the same;
3. Print debug log and increase an error metrics counter;
4. Start to produce messages;
5. Monitor the debug log and metrics;
6. Gets "Detected stale connection to broker" and "Failed to write on connection" errors quite often;
7. Sets KeepAliveInterval to 15s, errors continue;
8. When KeepAliveInterval sets to 5s, errors stop.

System configuration

Pulsar version: v2.11.0
Pulsar client go version: v0.10.0

[gunli]

errors are very often when `KeepAliveInterval ` are set to 15s or longer, and errors stops after `KeepAliveInterval ` are set to 5s.

[gunli]

errors we got.

[gunli]

It seems set to 5s still errors

[kaviyakrishnamoorthy]

is it resolved?how do we resolve this issue?