Kafka 4.0 support

[maxwolf8852]

Updated deprecated protocol fields for Apache Kafka 4.0 support:

• JoinGroup: V1 -> V2
• CreateTopics: V0 -> V2
• DeleteTopics V0 -> V1

While Conn does not use Client, the "negotiateVersion" mechanism has been reused for backward compatibility.

[petedannemann]

Thank you for this! The main feedback is it looks like ThrottleTime was actually added to CreateTopics responses in v2, not v1. I think if possible it would be good to ensure our tests cover both supported versions with this change. Additionally, we should add Kafka 4.0 to our circleci testing matrix to make sure the tests do actually succeed there

[petedannemann]

Looks like there are some additional issues with tests. I think these may have started with kafka 3.X and I started trying to dig into them here #1381

[petedannemann]

Thank you for addressing my feedback though, this PR looks good I just need to figure out the test issues before we can merge this

[petedannemann]

What I am seeing in my PR is a lot of these errors in Kafka's logs that cause many requests to fail during tests

Timeout failed due to Java heap space

I've tried adding JVM_OPTS for increasing the heap size and I am still seeing the errors even with a huge amount of memory given to Kafka. I wonder if Kafka 4.0 is not stable yet, it seems like the heap size cannot be adjusted as expected and the memory usage is not predictable

[maxwolf8852]

What I am seeing in my PR is a lot of these errors in Kafka's logs that cause many requests to fail during tests

Timeout failed due to Java heap space

I've tried adding JVM_OPTS for increasing the heap size and I am still seeing the errors even with a huge amount of memory given to Kafka. I wonder if Kafka 4.0 is not stable yet, it seems like the heap size cannot be adjusted as expected and the memory usage is not predictable

Yes, as you can see, such errors only occur when using consumer group.
I also tried to reproduce these errors on a local device with more memory and they are still here. I'm currently trying to delve deeper into the problem.