KAFKA-19826: Implement coordinator adaptive batch linger time #20780

squah-confluent · 2025-10-27T22:00:53Z

Add support for an adaptive batch linger time in the group and share
coordinators. When an adaptive batch linger time is enabled, we no
longer create a timer to flush the current batch. Instead, we append a
flush operation at the end of the event queue so that any currently
queued operations are naturally collected into the batch.

To avoid double flushing from hitting the maximum batch size or
transactional writes, we number batches with an epoch to check whether
the batch has already been flushed.

The group.coordinator.append.linger.ms and
share.coordinator.append.linger.ms configs are extended to allow -1, to
specify an adaptive append linger time. The default for these configs is
also updated to -1.

Reviewers: PoAn Yang [email protected], TaiJuWu [email protected],
David Jacot [email protected]

NB: This refers to the CoordinatorRuntimeBuilder's default append linger time, not the config-level default.

squah-confluent · 2025-10-27T22:02:19Z

...r-common/src/test/java/org/apache/kafka/coordinator/common/runtime/DirectEventProcessor.java

+    private final Deque<CoordinatorEvent> queue;
+    private boolean inEvent;
+
+    public DirectEventProcessor() {
+        this.queue = new LinkedList<>();
+        this.inEvent = false;
+    }
+


We can now enqueue flush events while in the middle of another event. The flush event must be run after the current event, so we have to introduce a queue here.

squah-confluent · 2025-10-27T22:06:36Z

...common/src/test/java/org/apache/kafka/coordinator/common/runtime/CoordinatorRuntimeTest.java

                .withCoordinatorRuntimeMetrics(mock(CoordinatorRuntimeMetrics.class))
                .withCoordinatorMetrics(mock(CoordinatorMetrics.class))
                .withSerializer(new StringSerializer())
+                .withAppendLingerMs(0)


Many of the tests in this file run with an adaptive linger time now, except for those with an explicit linger time.

I updated a handful of tests using the ManualEventProcessor to use an append linger time of 0 (preserving the existing behavior) to avoid test churn.

squah-confluent · 2025-10-27T22:07:50Z

The changes are a little long, so it may be easier to review this PR commit by commit, at least for the first 5 commits.

FrankYang0529

Overall LGTM. Leave some minor comments.

FrankYang0529 · 2025-10-28T12:15:56Z

...tor-common/src/main/java/org/apache/kafka/coordinator/common/runtime/CoordinatorRuntime.java

        public Builder<S, U> withAppendLingerMs(int appendLingerMs) {
+            this.appendLingerMs = OptionalInt.of(appendLingerMs);
+            return this;
+        }


From the references, this function is only used by test code. Should we remove it and update test cases to use OptionalInt one?

Sure, we can remove the method.

FrankYang0529 · 2025-10-28T12:19:24Z

...tor-common/src/main/java/org/apache/kafka/coordinator/common/runtime/CoordinatorRuntime.java

+            if (appendLingerMs == null)
+                appendLingerMs = OptionalInt.empty();


From the production code, the appendLingerMs input cannot be null. How about we set a default value OptionalInt.empty() to appendLingerMs and remove appendLingerMs == null check?

I was following the existing pattern in the builder. We don't set a default value for logPrefix, logContext and compression and initialize them in build().

FrankYang0529 · 2025-10-28T12:20:35Z

...tor-common/src/main/java/org/apache/kafka/coordinator/common/runtime/CoordinatorRuntime.java

+            if (appendLingerMs.isPresent() && appendLingerMs.getAsInt() < -1)
+                throw new IllegalArgumentException("AppendLinger must be empty or >= 0");


If the appendLingerMs is -1, it will be input as OptionalInt.empty(). How about checking the value is >= 0 here?

Good catch! The check is supposed to reject -1.

squah-confluent · 2025-10-28T19:55:25Z

@FrankYang0529 Thanks for the review! I updated the code. I also added some tests for the config logic.

TaiJuWu

LGTM.

dajac

@squah-confluent Thanks for the patch. I left a few minor comments for consideration.

dajac · 2025-11-14T15:55:55Z

group-coordinator/src/main/java/org/apache/kafka/coordinator/group/GroupCoordinatorConfig.java

            ConfigDef.ValidList.in(false, Group.GroupType.documentValidValues()), MEDIUM, GROUP_COORDINATOR_REBALANCE_PROTOCOLS_DOC)
        .define(GROUP_COORDINATOR_NUM_THREADS_CONFIG, INT, GROUP_COORDINATOR_NUM_THREADS_DEFAULT, atLeast(1), HIGH, GROUP_COORDINATOR_NUM_THREADS_DOC)
-        .define(GROUP_COORDINATOR_APPEND_LINGER_MS_CONFIG, INT, GROUP_COORDINATOR_APPEND_LINGER_MS_DEFAULT, atLeast(0), MEDIUM, GROUP_COORDINATOR_APPEND_LINGER_MS_DOC)
+        .define(GROUP_COORDINATOR_APPEND_LINGER_MS_CONFIG, INT, GROUP_COORDINATOR_APPEND_LINGER_MS_DEFAULT, atLeast(-1), MEDIUM, GROUP_COORDINATOR_APPEND_LINGER_MS_DOC)


I wonder whether we could strengthen the validation here because -0.5 is not a valid value for instance. Would it be possible to accept -1 or atLeast(0)?

Thanks for the review. I tested -0.5 and it's already disallowed

org.apache.kafka.common.config.ConfigException: Invalid value -0.5 for configuration group.coordinator.append.linger.ms: Not a number of type INT at app//org.apache.kafka.common.config.ConfigDef.parseType(ConfigDef.java:776) at app//org.apache.kafka.common.config.ConfigDef.parseValue(ConfigDef.java:531) at app//org.apache.kafka.common.config.ConfigDef.parse(ConfigDef.java:524) at app//org.apache.kafka.common.config.AbstractConfig.<init>(AbstractConfig.java:118) at app//org.apache.kafka.server.config.AbstractKafkaConfig.<init>(AbstractKafkaConfig.java:78)

dajac · 2025-11-14T15:57:45Z

share-coordinator/src/main/java/org/apache/kafka/coordinator/share/ShareCoordinatorConfig.java

        .define(LOAD_BUFFER_SIZE_CONFIG, INT, LOAD_BUFFER_SIZE_DEFAULT, atLeast(1), HIGH, LOAD_BUFFER_SIZE_DOC)
        .define(STATE_TOPIC_COMPRESSION_CODEC_CONFIG, INT, (int) STATE_TOPIC_COMPRESSION_CODEC_DEFAULT.id, HIGH, STATE_TOPIC_COMPRESSION_CODEC_DOC)
-        .define(APPEND_LINGER_MS_CONFIG, INT, APPEND_LINGER_MS_DEFAULT, atLeast(0), MEDIUM, APPEND_LINGER_MS_DOC)
+        .define(APPEND_LINGER_MS_CONFIG, INT, APPEND_LINGER_MS_DEFAULT, atLeast(-1), MEDIUM, APPEND_LINGER_MS_DOC)


ditto about the validation.

dajac

lgtm, thanks.

dajac · 2025-11-14T17:28:24Z

@squah-confluent There are some checkstyle errors. Could you please fix them?

…d-linger-ms

dajac

lgtm

squah-confluent added 5 commits October 27, 2025 21:36

KAFKA-19826: Implement adaptive append linger in the coordinator runtime

4b96c79

KAFKA-19826: Add tests for adaptive append linger time

33a13a5

KAFKA-19826: Update config to allow append.linger.ms = -1

b93f790

KAFKA-19826: Set default CoordinatorRuntime append linger to adaptive

f13c265

NB: This refers to the CoordinatorRuntimeBuilder's default append linger time, not the config-level default.

KAFKA-19826: Move new tests to just after the batching tests

578b13a

github-actions bot added triage PRs from the community core Kafka Broker KIP-932 Queues for Kafka group-coordinator labels Oct 27, 2025

squah-confluent commented Oct 27, 2025

View reviewed changes

github-actions bot removed the triage PRs from the community label Oct 28, 2025

FrankYang0529 added the ci-approved label Oct 28, 2025

FrankYang0529 reviewed Oct 28, 2025

View reviewed changes

squah-confluent added 5 commits October 28, 2025 16:38

fixup: fix appendLingerMs range check

d4df111

fixup: fix GroupCoordinatorConfigTest

9cadfb4

fixup: add GroupCoordinatorConfigtest for appendLingerMs

68423f4

fixup: add ShareCoordinatorConfigTest for appendLingerMs

286ef96

fixup: remove withAppendLingerMs(int) overload

142f12f

squah-confluent requested a review from FrankYang0529 October 28, 2025 19:55

TaiJuWu approved these changes Oct 31, 2025

View reviewed changes

dajac self-requested a review November 3, 2025 08:50

dajac reviewed Nov 14, 2025

View reviewed changes

fixup: test that append.linger.ms = -0.5 is rejected

94be09a

dajac approved these changes Nov 14, 2025

View reviewed changes

squah-confluent added 2 commits November 14, 2025 17:45

Merge remote-tracking branch 'origin/trunk' into squah-adaptive-appen…

92634cd

…d-linger-ms

fixup: fix compile error and checkstyle

2fc2092

dajac approved these changes Nov 14, 2025

View reviewed changes

dajac merged commit b58aae2 into apache:trunk Nov 14, 2025
22 checks passed

dajac deleted the squah-adaptive-append-linger-ms branch November 14, 2025 20:16

		if (appendLingerMs == null)
		appendLingerMs = OptionalInt.empty();

		if (appendLingerMs.isPresent() && appendLingerMs.getAsInt() < -1)
		throw new IllegalArgumentException("AppendLinger must be empty or >= 0");

KAFKA-19826: Implement coordinator adaptive batch linger time #20780

KAFKA-19826: Implement coordinator adaptive batch linger time #20780

Conversation

squah-confluent commented Oct 27, 2025 • edited by github-actions bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

squah-confluent commented Oct 27, 2025

Uh oh!

FrankYang0529 left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

squah-confluent commented Oct 28, 2025

Uh oh!

TaiJuWu left a comment

Choose a reason for hiding this comment

Uh oh!

dajac left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

dajac left a comment

Choose a reason for hiding this comment

Uh oh!

dajac commented Nov 14, 2025

Uh oh!

dajac left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

squah-confluent commented Oct 27, 2025 •

edited by github-actions bot

Loading