Fix flaky: DSM Kafka consumer backlogs serialization

[p-datadog]

What does this PR do?

Fixes flaky test spec/datadog/data_streams/processor_spec.rb:294
(Datadog::DataStreams::Processor Kafka tracking methods #track_kafka_consume serializes consumer backlogs with type:kafka_commit tag)
by stopping the auto-spawned background worker thread at the start of the
test.

Motivation:

Flaky test reported in ruby-guild#281.
First-attempt failure on Ruby 3.2 / build & test (standard) [1];
attempt 2 (rerun) passed.

Failure:

expect(backlogs.length).to eq(2)
  expected: 2
       got: 1

Root cause: the auto-spawned background worker thread races with the
test thread. Processor.new schedules a worker that runs perform_loop's
first iteration immediately (loop_wait_before_first_iteration? is false).
If the OS schedules the worker after the first track_kafka_consume but
before the second:

1. Test pushes event1 to @event_buffer.
2. Worker wakes up, calls process_events → drains [event1] →
   @consumer_stats = [event1].
3. Worker continues to flush_stats → executes @consumer_stats.clear at
   processor.rb:313
   → @consumer_stats = [].
4. Test pushes event2.
5. Test calls process_events → drains [event2] →
   @consumer_stats = [event2].
6. serialize_consumer_backlogs returns one entry → expected 2, got 1.

Fix: Stop the worker at the start of the test so the test owns the
buffer and @consumer_stats lifecycle synchronously. Mirrors the fix
applied in PR #5715
(commit 76683752f7)
for the adjacent #flush_stats test, which has the same race class.

Change log entry

None.

How to test the change?

Reproducer is in companion PR — that PR's CI shows the race-forcing variant
(sleep 0.5 between pushes) failing deterministically. The validation
companion PR merges this fix with the reproducer and CI passes, proving
the fix neutralizes the forced failure.

Companion PRs:

• Reproducer: Reproduce flaky: DSM Kafka consumer backlogs serialization #5759
• Validation: Validate fix for flaky: DSM Kafka consumer backlogs serialization #5761

Validation results:

• Reproducer PR Reproduce flaky: DSM Kafka consumer backlogs serialization #5759 — failed deterministically across the full Ruby matrix: 9 Linux Unit Tests shards (Ruby 2.5 → 4.0), 6 macOS Test jobs (Ruby 3.0 → 4.0), and test-memcheck. All failures were the same assertion (expected 2, got 1) on this exact test. Closed.
• Validation PR Validate fix for flaky: DSM Kafka consumer backlogs serialization #5761 — merged reproducer + fix, CI green. Fix neutralizes the forced failure. Closed.

[datadog-datadog-prod-us1-2]

🎉 All green!

❄️ No new flaky tests detected
🧪 All tests passed

🎯 Code Coverage (details)
• Patch Coverage: 100.00%
• Overall Coverage: 97.15% (+0.00%)

_{This comment will be updated automatically if new data arrives.
🔗 Commit SHA: 5733f1b | Docs | Datadog PR Page | Give us feedback!}