Kafka

Response message Kafka brokers communicate with clients through the Kafka wire protocol. The protocol uses a request-response model, where the client sends a request message and the broker replies with a response message. A Kafka response message has three parts:

1. message_size (32-bit signed integer)
2. Header
3. Body

Kafka has a few different header versions. The way Kafka determines which header version to use is a bit complicated and is outside the scope of this challenge. For more information, take a look at KIP-482 and this Stack Overflow answer. In this stage, you will use response header v0 (scroll down).

Response header v0 contains a single field: correlation_id. This field lets clients match responses to their original requests. Here's how it works:

1. The client generates a correlation ID.
2. The client sends a request that includes the correlation ID.
3. The broker sends a response that includes the same correlation ID.
4. The client receives the response and matches the correlation ID to the original request.

Kafka APIs Every Kafka request is an API call. The Kafka protocol defines over 70 different APIs, all of which do different things. Here are some examples:

• Produce writes events to partitions.
• CreateTopics creates new topics.
• ApiVersions returns the broker's supported API versions. A Kafka request specifies the API its calling by using the request_api_key header field.

Message body The schemas for the request and response bodies are determined by the API being called. For example, here are some of the fields that the Produce request body contains:

• The name of the topic to write to.
• The key of the partition to write to.
• The event data to write. On the other hand, the Produce response body contains a response code for each event. These response codes indicate if the writes succeeded. As a reminder, requests and responses both have the following format:

1. message_size
2. Header
3. Body

API versioning Each API supports multiple versions, to allow for different schemas. Here's how API versioning works:

• Requests use the header field request_api_version to specify the API version being requested.
• Responses always use the same API version as the request. For example, a Produce Request (Version: 3) will always get a Produce Response (Version: 3) back.
• Each API's version history is independent. So, different APIs with the same version are unrelated. For example, Produce Request (Version: 10) is not related to Fetch Request (Version: 10).

The ApiVersions API The ApiVersions API returns the broker's supported API versions. For example, ApiVersions may say that the broker supports Produce versions 5 to 11, Fetch versions 0 to 3, etc. In this stage, you'll begin to add support for ApiVersions version 4. For this stage, you only need to add support for the error_code field. You'll implement the other fields in later stages.

Note: As of Oct 30th 2024, version 4 of ApiVersions is still unreleased, so it isn't available in the Kafka docs yet. However, the request and response formats for ApiVersions version 4 are identical to those of version 3. The docs for version 4 will be available once Kafka 3.9 is released.

The ApiVersions response body begins with error_code, a 16-bit signed integer. This field indicates if an error occurred with the request. It's set to 0 if there was no error. To see all the possible values, consult the error codes chart. You only need to add support for error code 35, UNSUPPORTED_VERSION. This error code occurs when the version of ApiVersions requested by the client is not supported by the broker. Assume that your broker only supports versions 0 to 4.