backend: editing consumer offset in v3.7.4/v3.8.0 against Kafka 4.2 fails with UNKNOWN_TOPIC_ID

[makuche]

Hi,

I have noticed a bug when using Redpanda Console with v3.7.4/v3.8.0 against Apache Kafka 4.2: Any edit on the consumer group offsets returns the error Could not apply offsets for consumer group <group-id>. Error: Apply offsets failed with 1 errors. Broker-side, every partition is rejected with UNKNOWN_TOPIC_ID. Console ≤ v3.7.3 and brokers ≤ Kafka 4.1 are unaffected.

Root cause

#2452 bumped pkg/kmsg 1.12.0 → 1.13.1, changing the client's max OffsetCommit version from v9 to v10. v10 identifies topics by TopicID (KIP-848) instead of name.

Kafka added v10 in 4.1 with latestVersionUnstable and stabilized it in 4.2 (KAFKA-19186). Hence 4.2 is the first broker to advertise v10 by default, so the negotiated version becomes v10.

EditConsumerGroupOffsets (edit_consumer_group_offsets.go) builds the kmsg.OffsetCommitRequest by hand and only sets the topic name. At v10 the name is not serialized at all, so the broker receives an all-zero topic ID and answers UNKNOWN_TOPIC_ID for every partition.

Steps to reproduce

• Start Kafka 4.2 and Console v3.8.0
• Create a topic, produce and consume with a group so it has committed offsets
• Within the Redpanda Console UI: Consumer Groups -> <group-id> -> Edit group offsets -> set to earliest -> Apply.

Expected behavior is that offsets are reset, actual is the error described above with UNKNOWN_TOPIC_ID. Reproduced in my setup with confluentinc/cp-kafka:8.2.0 (bundles Kafka 4.2).

I will open a PR where I fetch the TopicID and map the ID back to the names, which fixes this behavior.