Describe the bug
During routine patching of our MSK Kafka clusters, we see a range of transient errors in our kafka producers (using the kafka-go Writer) as the brokers are patched. Many of these errors are retried by the logic in the kafka-go Writer, but we see a small volume of EOF errors which are not retried at all and result in an immediate permanent failure of those writes. I'm not sure if this is intentional due to certain behaviours of the kafka protocol, but from our perspective we would like to see these requests that result in an io.EOF error retried as well so that we don't lose those messages.
It looks like the related code snippets are:
|
if !isTemporary(err) && !isTransientNetworkError(err) { |
|
func isTemporary(err error) bool { |
|
func isTransientNetworkError(err error) bool { |
Kafka Version
Kafka version: 3.7
kafka-go version: v0.4.47
To Reproduce
We see this behaviour for a small number of writes every time there is security patching or other operations on our MSK cluster that result in broker rolling replacements/restarts. When the Writer sees an io.EOF error, this is not retried and the message write fails.
Expected Behavior
Ideally all errors that can be retried by the Writer are retried so that maintenance operations on a Kafka cluster are seamless and don't cause any messages to be lost.
Observed Behavior
If an EOF is received by the kafka Writer, this is not retried and the write fails immediately with the following error: Kafka write errors (1/1), errors: [kafka.(*Client).Produce: EOF]
Describe the bug
During routine patching of our MSK Kafka clusters, we see a range of transient errors in our kafka producers (using the kafka-go Writer) as the brokers are patched. Many of these errors are retried by the logic in the kafka-go Writer, but we see a small volume of EOF errors which are not retried at all and result in an immediate permanent failure of those writes. I'm not sure if this is intentional due to certain behaviours of the kafka protocol, but from our perspective we would like to see these requests that result in an
io.EOFerror retried as well so that we don't lose those messages.It looks like the related code snippets are:
kafka-go/writer.go
Line 1162 in a8e5eab
kafka-go/error.go
Line 601 in a8e5eab
kafka-go/error.go
Line 609 in a8e5eab
Kafka Version
Kafka version: 3.7
kafka-goversion: v0.4.47To Reproduce
We see this behaviour for a small number of writes every time there is security patching or other operations on our MSK cluster that result in broker rolling replacements/restarts. When the Writer sees an
io.EOFerror, this is not retried and the message write fails.Expected Behavior
Ideally all errors that can be retried by the Writer are retried so that maintenance operations on a Kafka cluster are seamless and don't cause any messages to be lost.
Observed Behavior
If an EOF is received by the kafka Writer, this is not retried and the write fails immediately with the following error:
Kafka write errors (1/1), errors: [kafka.(*Client).Produce: EOF]