Currently, kafka-go returns a NotLeaderForPartition error if a producer writes to a partition
whose leader info is stale. In other clients (e.g., Java Kafka client), this is handled automatically:
metadata is refreshed and the write is retried transparently.
Suggestion/Proposal : potentially add an optional WriterConfig flag, e.g., AutoRefreshMetadata bool,
which automatically retries after refreshing partition metadata ?
Description : The idea is to add a config to like
writer := kafka.NewWriter(kafka.WriterConfig{
Brokers: []string{"kafka1:9092", "kafka2:9092", "kafka3:9092"},
Topic: topic,
Balancer: &kafka.LeastBytes{},
AutoRefreshMetadata: true, // new config flag, may be false by default to avoid impact performance and B/w compatibility
})
With this enabled, the writer would:
Detect a NotLeaderForPartition error on a write.
Refresh partition metadata from the cluster.
Retry the write automatically (maybe a configurable number of times ? ).
Supporting documentation
https://issues.apache.org/jira/browse/KAFKA-16395
https://aiven.io/docs/products/kafka/troubleshooting/non-leader-for-partition#:~:text=Most%20producers%20libraries,in%20fact%20harmless.
I am happy to pick this if team is okay with the proposal. Thanks !
Currently, kafka-go returns a NotLeaderForPartition error if a producer writes to a partition
whose leader info is stale. In other clients (e.g., Java Kafka client), this is handled automatically:
metadata is refreshed and the write is retried transparently.
Suggestion/Proposal : potentially add an optional WriterConfig flag, e.g., AutoRefreshMetadata bool,
which automatically retries after refreshing partition metadata ?
Description : The idea is to add a config to like
writer := kafka.NewWriter(kafka.WriterConfig{
Brokers: []string{"kafka1:9092", "kafka2:9092", "kafka3:9092"},
Topic: topic,
Balancer: &kafka.LeastBytes{},
AutoRefreshMetadata: true, // new config flag, may be false by default to avoid impact performance and B/w compatibility
})
With this enabled, the writer would:
Detect a NotLeaderForPartition error on a write.
Refresh partition metadata from the cluster.
Retry the write automatically (maybe a configurable number of times ? ).
Supporting documentation
https://issues.apache.org/jira/browse/KAFKA-16395
https://aiven.io/docs/products/kafka/troubleshooting/non-leader-for-partition#:~:text=Most%20producers%20libraries,in%20fact%20harmless.
I am happy to pick this if team is okay with the proposal. Thanks !