Skip to content

Rebalance of topics with no config files - #245

Merged
dateutli merged 8 commits into
masterfrom
TUBE-3309/rebalaing-topcis-wo-config-files
Dec 8, 2025
Merged

Rebalance of topics with no config files#245
dateutli merged 8 commits into
masterfrom
TUBE-3309/rebalaing-topcis-wo-config-files

Conversation

@dateutli

@dateutli dateutli commented Dec 1, 2025

Copy link
Copy Markdown
Contributor

Description

This PR introduces the --generate-config flag for the rebalance command.

The flag would update the main process as outlined below:

  • From the inventoried config files we create a set to keep track of the existing config files
  • Bootstrap missing topic configs
  • Re-inventory config files to take into account the newly generated ones
  • Continue the process as usual (if a topic config file already existed it'll take precedence and the generated one will be dismissed)
  • Clean up bootstrapped topic configs

This PR also adds a wrapper function around the traversal of topic configs for reusability purposes.

Notes

  • This is effectively calling the main functionality of the bootstrap command in a limited capacity. We could potentially add support for all of the bootstrap arguments but that's outside the scope of this feature

Testing

  1. Picked topic check-topic--JNKfqh for testing
  2. $ topicctl get balance 'check-topic--JNKfqh' --zk-addr=localhost:2181

Broker replicas:

ID RACK POSITION 1 POSITION 2 TOTAL
1 zone1 2 1 3
2 zone1 1 2 3
3 zone2 1 2 3
4 zone2 2 1 3
5 zone3 1 2 3
6 zone3 2 1 3

Broker rack replicas:

RACK POSITION 1 POSITION 2 TOTAL
zone1 3 3 6
zone2 3 3 6
zone3 3 3 6
  1. Manually bootstrapped check-topic--JNKfqh in order to remove one broker

  2. $ topicctl apply 'examples/local-cluster/topics/check-topic--JNKfqh.yaml' --rebalance --to-remove 1

  3. $ topicctl get balance 'check-topic--JNKfqh' --zk-addr=localhost:2181

Broker replicas:

ID RACK POSITION 1 POSITION 2 TOTAL
1 zone1 0 0 0
2 zone1 2 2 4
3 zone2 1 2 3
4 zone2 2 2 4
5 zone3 2 2 4
6 zone3 2 1 3

Broker rack replicas:

RACK POSITION 1 POSITION 2 TOTAL
zone1 2 2 4
zone2 3 4 7
zone3 4 3 7
  1. Deleted manually bootstrapped topic config

  2. $ topicctl rebalance --cluster-config='examples/local-cluster/cluster.yaml' --path-prefix='examples/local-cluster/topics' --bootstrap-missing-configs

Broker replicas:

ID RACK POSITION 1 POSITION 2 TOTAL
1 zone1 2 1 3
2 zone1 1 2 3
3 zone2 1 2 3
4 zone2 1 2 3
5 zone3 2 1 3
6 zone3 2 1 3

Broker rack replicas:

RACK POSITION 1 POSITION 2 TOTAL
zone1 3 3 6
zone2 2 4 6
zone3 4 2 6

@dateutli
dateutli requested a review from a team as a code owner December 1, 2025 18:57
@pazhang-25

Copy link
Copy Markdown
Contributor

Sorry if you were already planning on doing this, but could you perhaps add a Description section on your PR and also a Testing section to show how you tested the code? :)

@dateutli

dateutli commented Dec 1, 2025

Copy link
Copy Markdown
Contributor Author

Sorry if you were already planning on doing this, but could you perhaps add a Description section on your PR and also a Testing section to show how you tested the code? :)

My bad, didn't mean to send it out for review just yet.

@dateutli
dateutli marked this pull request as draft December 1, 2025 19:48
David Teutli added 2 commits December 1, 2025 18:49
…existing generated files; wrap topic file and config traversal in reusable function
@dateutli
dateutli requested a review from hhahn-tw December 2, 2025 18:33
@dateutli
dateutli marked this pull request as ready for review December 2, 2025 18:34

@petedannemann petedannemann left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks pretty good, left two minor comments

Comment thread cmd/topicctl/subcmd/rebalance.go Outdated
Comment thread cmd/topicctl/subcmd/rebalance.go Outdated
David Teutli and others added 2 commits December 2, 2025 15:07
Co-authored-by: Peter Dannemann <28637185+petedannemann@users.noreply.github.com>
Comment thread cmd/topicctl/subcmd/rebalance.go Outdated
Comment thread cmd/topicctl/subcmd/rebalance.go Outdated
Comment thread cmd/topicctl/subcmd/rebalance.go Outdated
@hhahn-tw

hhahn-tw commented Dec 3, 2025

Copy link
Copy Markdown
Contributor

Please also provide update the README outlining this new feature

@dateutli dateutli changed the title [TUBE-3309] Rebalance of topics with no config files Rebalance of topics with no config files Dec 3, 2025
David Teutli added 2 commits December 4, 2025 10:07
@dateutli
dateutli requested a review from hhahn-tw December 4, 2025 17:23
Comment thread README.md Outdated
Co-authored-by: Peter Dannemann <28637185+petedannemann@users.noreply.github.com>

@hhahn-tw hhahn-tw left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

looks good!

@dateutli
dateutli merged commit 1a410cd into master Dec 8, 2025
9 checks passed
@dateutli
dateutli deleted the TUBE-3309/rebalaing-topcis-wo-config-files branch December 8, 2025 15:24
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants