processor: add log_sampling processor for rate limiting

[niedbalski]

This branch adds a new log_sampling processor for sampling log streams using some initial 3
strategies.

Strategies implemented on this PR

• Fixed window: samples first N logs per fixed time window
• Sliding window: maintains rolling rate limit over time
• Exponential decay: progressively reduces sampling rate

Configuration examples

Fixed window sampling

processors:
  logs:
    - name: log_sampling
      window_type: fixed
      window_size: 10  # 10 second windows
      max_logs_per_window: 100  # Keep first 100 logs per window

Sliding window sampling

processors:
  logs:
    - name: log_sampling
      window_type: sliding
      window_size: 30  # 30 second rolling window
      max_logs_per_window: 500  # Max 500 logs in any 30 second period

Exponential decay sampling

processors:
  logs:
    - name: log_sampling
      window_type: exponential
      decay_base_rate: 0.8  # Start at 80% sampling
      decay_factor: 0.7  # Reduce to 70% of previous rate
      decay_interval: 60  # Every 60 seconds

Testing

• Unit tests added in tests/internal/log_sampling.c
• Test configurations provided

Use cases

• Rate limiting high-volume debug logs
• Reducing ingestion costs while maintaining visibility
• Progressive sampling for long-running applications
• Protecting downstream systems from log bursts

Future work/ strategies

• LLM assisted sampling
• Random sampling
• Attribute based sampling

Summary by CodeRabbit

• New Features
  • Introduced a log sampling processor with configurable strategies (fixed window, sliding window, exponential decay).
  • Added user-configurable parameters for window type/size, max logs per window, and decay settings to tune sampling behavior.
• Chores
  • Added a build-time option to enable or disable the log sampling processor (enabled by default).
• Tests
  • Added comprehensive unit tests covering all sampling strategies and edge cases to ensure accuracy and reliability.

[coderabbitai]

Walkthrough

Introduces a new log sampling processor plugin with build-time enablement, plugin registration, implementation (with fixed, sliding, and exponential strategies), and unit tests. Updates CMake options and plugin lists to compile and include the processor and its tests when enabled.

Changes

Cohort / File(s)	Summary
Build option exposure cmake/plugins_options.cmake	Adds public option FLB_PROCESSOR_LOG_SAMPLING (default ON) under Processors.
Plugin registration plugins/CMakeLists.txt	Registers processor_log_sampling in PROCESSOR_PLUGIN list and adds its directory to build.
Processor plugin implementation plugins/processor_log_sampling/CMakeLists.txt, plugins/processor_log_sampling/log_sampling.c, plugins/processor_log_sampling/log_sampling.h	Adds new processor plugin with config map, init/process/exit callbacks, and three sampling strategies (fixed, sliding, exponential). Exposes exported helpers and context types.
Tests tests/internal/CMakeLists.txt, tests/internal/log_sampling.c	Gates unit tests behind FLB_PROCESSOR_LOG_SAMPLING and adds tests covering all strategies and edge cases.

Sequence Diagram(s)

sequenceDiagram
    participant C as Config
    participant FB as Fluent Bit
    participant PI as Processor Instance (log_sampling)
    participant STR as Sampling Strategy

    C->>FB: Enable FLB_PROCESSOR_LOG_SAMPLING + set params
    FB->>PI: Create instance (cb_init)
    PI->>PI: Parse config, allocate state

    loop For each log chunk
        FB->>PI: cb_process_logs(chunk)
        PI->>STR: Decide(sample?) per record (fixed/sliding/exponential)
        STR-->>PI: allow/drop decision
        PI-->>FB: Filtered records + stats
    end

    FB->>PI: cb_exit
    PI->>PI: Cleanup

Loading

Estimated code review effort

🎯 3 (Moderate) | ⏱️ ~20 minutes

Poem

I nibble logs with measured hop,
In windows fixed, I pause and stop;
I slide through buckets, ticking time,
Or fade like echoes, exponential rhyme.
New flags raised high, tests all in line—
Sampled carrots… er, records—crunch just fine! 🥕

Tip

🔌 Remote MCP (Model Context Protocol) integration is now available!

Pro plan users can now connect to remote MCP servers from the Integrations page. Connect with popular remote MCPs such as Notion and Linear to add more context to your reviews and chats.

✨ Finishing Touches

• 📝 Generate Docstrings

🧪 Generate unit tests

• Create PR with unit tests
• Post copyable unit tests in a comment
• Commit unit tests in branch feat/log_sampling

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

🪧 Tips

Chat

There are 3 ways to chat with CodeRabbit:

• Review comments: Directly reply to a review comment made by CodeRabbit. Example:
  • I pushed a fix in commit <commit_id>, please review it.
  • Open a follow-up GitHub issue for this discussion.
• Files and specific lines of code (under the "Files changed" tab): Tag @coderabbitai in a new review comment at the desired location with your query.
• PR comments: Tag @coderabbitai in a new PR comment to ask questions about the PR branch. For the best results, please provide a very specific query, as very limited context is provided in this mode. Examples:
  • @coderabbitai gather interesting stats about this repository and render them as a table. Additionally, render a pie chart showing the language distribution in the codebase.
  • @coderabbitai read the files in the src/scheduler package and generate a class diagram using mermaid and a README in the markdown format.

Support

Need help? Create a ticket on our support page for assistance with any issues or questions.

CodeRabbit Commands (Invoked using PR/Issue comments)

Type @coderabbitai help to get the list of available commands.

Other keywords and placeholders

• Add @coderabbitai ignore anywhere in the PR description to prevent this PR from being reviewed.
• Add @coderabbitai summary to generate the high-level summary at a specific location in the PR description.
• Add @coderabbitai anywhere in the PR title to generate the title automatically.

CodeRabbit Configuration File (.coderabbit.yaml)

• You can programmatically configure CodeRabbit by adding a .coderabbit.yaml file to the root of your repository.
• Please see the configuration documentation for more information.
• If your editor has YAML language server enabled, you can add the path at the top of this file to enable auto-completion and validation: # yaml-language-server: $schema=https://coderabbit.ai/integrations/schema.v2.json

Status, Documentation and Community

• Visit our Status Page to check the current availability of CodeRabbit.
• Visit our Documentation for detailed information on how to use CodeRabbit.
• Join our Discord Community to get help, request features, and share feedback.
• Follow us on X/Twitter for updates and announcements.

[niedbalski]

@edsiper can I get your push here? Thanks