Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

kafka replay speed: add support for metadata & source #9287

Open
wants to merge 2 commits into
base: dimitar/ingester/consume-latency-push-sharding
Choose a base branch
from

Conversation

dimitarvdimitrov
Copy link
Contributor

@dimitarvdimitrov dimitarvdimitrov commented Sep 12, 2024

What this PR does

  1. Parallelization based on:

    • The parallelStoragePusher now supports separate pushers for different write request sources (e.g., API, RULE) per tenant.
    • This is necessary because the ingester keep track of metrics based on the source of the request.
    • The tradeoff is that the config option for batch size isn't what it looks like - we have 2N batches per tenant, not the N that's configured.
  2. Metadata Handling:

    • Added support for pushing metadata alongside time series data.
    • Metadata is distributed across shards in a round-robin fashion.

Which issue(s) this PR fixes or relates to

Fixes #

Checklist

  • Tests updated.
  • Documentation added.
  • CHANGELOG.md updated - the order of entries should be [CHANGE], [FEATURE], [ENHANCEMENT], [BUGFIX].
  • about-versioning.md updated with experimental features.

@gotjosh gotjosh force-pushed the gotjosh/ingester/refactor-test branch 2 times, most recently from 08a724a to 8a8a7c2 Compare September 19, 2024 16:03
Base automatically changed from gotjosh/ingester/refactor-test to dimitar/ingester/consume-latency-push-sharding September 19, 2024 17:24
@dimitarvdimitrov dimitarvdimitrov force-pushed the dimitar/ingester/consume-latency-push-sharding branch from 16f8a12 to e28dbab Compare September 19, 2024 17:34
@dimitarvdimitrov dimitarvdimitrov force-pushed the dimitar/ingester/consume-latency-push-sharding-fixups/metadata-exemplars branch from f4f4a15 to f4b2c80 Compare September 19, 2024 21:57
@dimitarvdimitrov dimitarvdimitrov changed the title kafka replay speed: add support for metadata to concurrent pushing @dimitarvdimitrov kafka replay speed: add support for metadata & source Sep 19, 2024
@dimitarvdimitrov dimitarvdimitrov force-pushed the dimitar/ingester/consume-latency-push-sharding-fixups/metadata-exemplars branch from f4b2c80 to d144116 Compare September 19, 2024 21:59
@dimitarvdimitrov dimitarvdimitrov force-pushed the dimitar/ingester/consume-latency-push-sharding-fixups/metadata-exemplars branch from d144116 to 043c3b3 Compare September 19, 2024 22:31
@dimitarvdimitrov dimitarvdimitrov changed the title @dimitarvdimitrov kafka replay speed: add support for metadata & source kafka replay speed: add support for metadata & source Sep 19, 2024
@dimitarvdimitrov dimitarvdimitrov marked this pull request as ready for review September 19, 2024 22:37
@dimitarvdimitrov dimitarvdimitrov requested a review from a team as a code owner September 19, 2024 22:37
Signed-off-by: Dimitar Dimitrov <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant