Ability to trigger auto rebalance on imbalance feature by ShubhamRwt · Pull Request #13037 · strimzi/strimzi-kafka-operator

ShubhamRwt · 2026-08-07T09:15:58Z

Type of change

New feature

Description

This PR implements the proposal auto-rebalance-on-imbalance. This feature allows the operator to trigger rebalance automatically in case their are some imbalance in the cluster

Checklist

Please go through this checklist and make sure all applicable tasks have been done

Update documentation
Update CHANGELOG.md (if present)
Reference relevant issue(s) and close them after merging
Write tests
Make sure all tests pass
Try your changes inside a Kubernetes cluster, not just from unit tests
AI assistance was used to create this PR (see the Strimzi AI policy)

snyk-io · 2026-08-07T09:16:06Z

✅ Snyk checks have passed. No issues have been found so far.

Status	Scan Engine	Critical	High	Medium	Low	Total (0)
✅	Open Source Security	0	0	0	0	0 issues
✅	Licenses	0	0	0	0	0 issues
✅	Code Security	0	0	0	0	0 issues

💻 Catch issues earlier using the plugins for VS Code, JetBrains IDEs, Visual Studio, and Eclipse.

ppatierno · 2026-08-10T11:37:15Z

@ShubhamRwt can we get a green build before starting for an early review please? Failure are around the CruiseControlReconciler so maybe something to fix in the logic before taking a look. Thanks :-)

codecov · 2026-08-10T12:40:28Z

Codecov Report

❌ Patch coverage is 71.67139% with 100 lines in your changes missing coverage. Please review.
✅ Project coverage is 80.72%. Comparing base (f6776c2) to head (3ac94ed).
⚠️ Report is 1 commits behind head on main.

Files with missing lines	Patch %	Lines
...rator/assembly/KafkaAutoRebalancingReconciler.java	75.35%	25 Missing and 10 partials ⚠️
...r/resource/cruisecontrol/CruiseControlApiImpl.java	46.55%	19 Missing and 12 partials ⚠️
.../assembly/KafkaAutoRebalanceImbalanceDetector.java	84.87%	10 Missing and 8 partials ⚠️
...imzi/operator/common/metrics/AnomalyMetricKey.java	0.00%	12 Missing ⚠️
...urce/cruisecontrol/CruiseControlStateResponse.java	50.00%	1 Missing and 1 partial ⚠️
...uster/operator/assembly/KafkaAssemblyOperator.java	50.00%	1 Missing ⚠️
...strimzi/operator/common/metrics/MetricsHolder.java	0.00%	1 Missing ⚠️

Additional details and impacted files

@@             Coverage Diff              @@
##               main   #13037      +/-   ##
============================================
- Coverage     80.93%   80.72%   -0.21%     
- Complexity     6741     6821      +80     
============================================
  Files           355      358       +3     
  Lines         23072    23414     +342     
  Branches       3148     3185      +37     
============================================
+ Hits          18673    18902     +229     
- Misses         3175     3251      +76     
- Partials       1224     1261      +37

Files with missing lines	Coverage Δ
...r/assembly/KafkaAssemblyOperatorMetricsHolder.java	`100.00% <100.00%> (ø)`
...perator/assembly/KafkaRebalanceConfigMapUtils.java	`83.33% <100.00%> (ø)`
...ator/resource/cruisecontrol/GoalViolationInfo.java	`100.00% <100.00%> (ø)`
...uster/operator/assembly/KafkaAssemblyOperator.java	`84.17% <50.00%> (-0.31%)`	⬇️
...strimzi/operator/common/metrics/MetricsHolder.java	`30.18% <0.00%> (-0.59%)`	⬇️
...urce/cruisecontrol/CruiseControlStateResponse.java	`71.42% <50.00%> (-28.58%)`	⬇️
...imzi/operator/common/metrics/AnomalyMetricKey.java	`0.00% <0.00%> (ø)`
.../assembly/KafkaAutoRebalanceImbalanceDetector.java	`84.87% <84.87%> (ø)`
...r/resource/cruisecontrol/CruiseControlApiImpl.java	`69.93% <46.55%> (-6.20%)`	⬇️
...rator/assembly/KafkaAutoRebalancingReconciler.java	`76.50% <75.35%> (-5.00%)`	⬇️

... and 5 files with indirect coverage changes

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

ShubhamRwt · 2026-08-10T12:52:24Z

@ppatierno All the tests seems to be passing now

ppatierno

@ShubhamRwt I had a first quick pass but it needs more detailed review.

Signed-off-by: ShubhamRwt <shubhamrwt02@gmail.com>

ShubhamRwt · 2026-09-09T18:49:50Z

This PR is now ready for review

ppatierno · 2026-09-10T07:38:46Z

+                                                    });
+                                        } else {
+                                            LOGGER.infoCr(reconciliation, "Rebalancing completed, transitioning to Idle");
+                                            return updateRebalanceCompletionTime()


we are updating the rebalance completion time when the rebalancing completed but how does it relate with the proposal where at some point we said:

The ConfigMap's lastRebalanceCompletionTime now reflects the failure timestamp

So it looks like the timestamp should be updated when there is a failure too?

ppatierno · 2026-09-10T07:52:15Z

+     * @return CompletionStage with GoalViolationInfo if violations detected, null otherwise
+     */
+    public CompletionStage<GoalViolationInfo> checkForGoalViolations() {
+        if (kafkaCr.getSpec() == null || kafkaCr.getSpec().getCruiseControl() == null) {


why are we checking the CC is configured, I would assume this code at this deep level won't be never reached if CC is not configured. I mean, why I should check for goal violations if CC is not even configured?

ppatierno · 2026-09-10T07:55:06Z

+
+        if (imbalanceConfig.isEmpty()) {
+            return CompletableFuture.completedFuture(null);
+        }


The same applies to the above checks, why should I check for goal violation if auto-rebalance is not configured? Should this being called only when we are sure CC is enabled in the Kafka CR? Maybe a check would be needed somewhere in a upper level within the code?

ppatierno · 2026-09-10T07:57:46Z

+        return supplier.secretOperations.getAsync(reconciliation.namespace(), CruiseControlResources.secretName(reconciliation.name()))
+                .thenCompose(ccSecret -> {
+                    if (ccSecret == null) {
+                        LOGGER.warnCr(reconciliation, "Cruise Control secret not found, skipping anomaly detection");


The log is misleading, we are not skipping anomaly detection. We tried to detect anomalies but it failed because for some reason the CC Secret is not available. I would more use:

Cruise Control secret not found, goal violations cannot be detected

ppatierno · 2026-09-10T07:58:33Z

+                    }
+
+                    return supplier.secretOperations.getAsync(reconciliation.namespace(), CruiseControlResources.apiSecretName(reconciliation.name()))
+                            .thenCompose(ccApiSecret -> {


We checked for ccSecret to be null before but we are not doing the same here for ccApiSecret, why?

ppatierno · 2026-09-10T12:04:50Z

+    - DiskCapacityGoal
+    - DiskUsageDistributionGoal
+  skipHardGoalCheck: true
+----


you should take a look on this approved proposal https://github.com/strimzi/proposals/blob/main/154-kafkarebalance-custom-resource-consolidation.md that should be land at some point.
If it comes first, we need to update the examples with the new KafkaRebalance API.

ppatierno · 2026-09-10T12:06:15Z

+
+The operator tracks rebalance completion times in a ConfigMap named `<cluster_name>-auto-rebalance-imbalance-tracker`.
+This prevents the operator from retriggering rebalances for anomalies that were already addressed by a previous rebalance.
+The ConfigMap is automatically managed by the operator and deleted when the Kafka cluster is deleted.


maybe an example of the ConfigMap structure would be useful.

ppatierno · 2026-09-10T12:10:05Z

+
+== Failure recovery
+
+If an auto-rebalance fails (the `KafkaRebalance` transitions to `NotReady`), the operator transitions back to `Idle` and deletes the failed resource.


the operator transitions back to Idle

What is transitioning to Idle? The operator itself doesn't have Idle state. What are you referring to?

ppatierno · 2026-09-10T12:11:18Z

+
+If an auto-rebalance fails (the `KafkaRebalance` transitions to `NotReady`), the operator transitions back to `Idle` and deletes the failed resource.
+The operator does not immediately retry.
+If the underlying issue persists, Cruise Control detects the goal violation again in the next anomaly detection cycle and a fresh rebalance is triggered.


It's related to what I already mention about the detection interval in CC. It's different from the reconciliation period. This sentence let the user thinks that a new rebalance triggers when CC detect an anomaly while it's not true. It triggers when the operator ask to CC for anomaly on the next reconciliation.

ppatierno · 2026-09-10T12:11:40Z

+If the underlying issue persists, Cruise Control detects the goal violation again in the next anomaly detection cycle and a fresh rebalance is triggered.
+
+If the operator crashes while in `RebalanceOnImbalance` state, on restart it resumes monitoring from the current state.
+If the `KafkaRebalance` resource is missing (for example, deleted while the operator was not running), the operator transitions to `Idle` and relies on Cruise Control's next anomaly detection cycle to retrigger if violations persist.


Still this ...

operator transitions to Idle

scholzj added this to the 1.3.0 milestone Aug 7, 2026

ppatierno reviewed Aug 13, 2026

View reviewed changes

ppatierno reviewed Aug 14, 2026

View reviewed changes

ShubhamRwt force-pushed the autoRebalanceFeature branch 2 times, most recently from dcb35bc to 6ce0247 Compare September 1, 2026 10:35

tomncooper mentioned this pull request Sep 7, 2026

Generating suggested CR changes streamshub/console#2842

Open

ShubhamRwt added 9 commits September 9, 2026 14:22

Added changes

0cefd0d

Signed-off-by: ShubhamRwt <shubhamrwt02@gmail.com>

Fix tests

e2dc929

Signed-off-by: ShubhamRwt <shubhamrwt02@gmail.com>

Added changes

092f058

Signed-off-by: ShubhamRwt <shubhamrwt02@gmail.com>

More changes

7fa97ae

Signed-off-by: ShubhamRwt <shubhamrwt02@gmail.com>

Added changes

a3d0d40

Signed-off-by: ShubhamRwt <shubhamrwt02@gmail.com>

Added docs

4b732c6

Signed-off-by: ShubhamRwt <shubhamrwt02@gmail.com>

Added fix

e3c9998

Signed-off-by: ShubhamRwt <shubhamrwt02@gmail.com>

More changes

0562fa3

Signed-off-by: ShubhamRwt <shubhamrwt02@gmail.com>

Add more unit tests

977ae52

Signed-off-by: ShubhamRwt <shubhamrwt02@gmail.com>

ShubhamRwt force-pushed the autoRebalanceFeature branch from 49588ed to 39afd8d Compare September 9, 2026 08:55

ShubhamRwt marked this pull request as ready for review September 9, 2026 08:58

ShubhamRwt marked this pull request as draft September 9, 2026 08:59

ShubhamRwt force-pushed the autoRebalanceFeature branch from 8c5ceb9 to c8b11b5 Compare September 9, 2026 09:53

Final changes

3ac94ed

Signed-off-by: ShubhamRwt <shubhamrwt02@gmail.com>

ShubhamRwt force-pushed the autoRebalanceFeature branch from 8b2a441 to 3ac94ed Compare September 9, 2026 11:15

ShubhamRwt marked this pull request as ready for review September 9, 2026 12:30

ppatierno reviewed Sep 10, 2026

View reviewed changes


		== Failure recovery

		If an auto-rebalance fails (the `KafkaRebalance` transitions to `NotReady`), the operator transitions back to `Idle` and deletes the failed resource.

Uh oh!

Conversation

ShubhamRwt commented Aug 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Type of change

Description

Checklist

Uh oh!

snyk-io Bot commented Aug 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

✅ Snyk checks have passed. No issues have been found so far.

Uh oh!

ppatierno commented Aug 10, 2026

Uh oh!

codecov Bot commented Aug 10, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

ShubhamRwt commented Aug 10, 2026

Uh oh!

ppatierno left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

ShubhamRwt commented Sep 9, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

ShubhamRwt commented Aug 7, 2026 •

edited

Loading

snyk-io Bot commented Aug 7, 2026 •

edited

Loading

codecov Bot commented Aug 10, 2026 •

edited

Loading