[SPARK-52006][SQL][CORE] Exclude CollectMetricsExec accumulator from Spark UI + event logs + metric heartbeats

[ivoson]

What changes were proposed in this pull request?

CollectMetricsExec uses an AggregatingAccumulator for emitting metrics in a side-channel that can be collected to the driver. And there are a few issues with this:

• It is a named accumulator but not an internal accumulator, so its result is shown in the Spark UI. But that result isn’t meaningful to users because it’s the .toString representation of an UnsafeRow representing an aggregation buffer.
• This same string representation is also logged in event logs on a per-task basis which is not necessary.
• Last and the most important is that there could be a race condition issue serializing the accumulator between task result serialization and executor heartbeat (heartbeat serialize it firstly and then task result serialization happens) which would lead to task failures since TypedImperativeAggregate.serializeAggregateBufferInPlace is not idempotent.
  

To fix the issues, below changes are made in this PR:

• Mark the accumulator as internal metrics to exclude it from Spark UI.
• JsonProtocol to explicitly exclude this accumulator from event logs.
• Executor to explicitly exclude it from heartbeats.

Why are the changes needed?

Fixing the race condition issues and remove unnecessary information from Spark UI and event log.

Does this PR introduce any user-facing change?

No

How was this patch tested?

UT added.
Manually tested with below repro in standalone cluster since the issue can't be triggered with local mode:

import org.apache.spark.TaskContext

val df = spark.range(100)
  .mapPartitions { iter =>
    TaskContext.get().addTaskCompletionListener[Unit] { _ =>
      Thread.sleep(30000L)
    }
    iter
  }.toDF("id")
  .observe(
    name = "my_event",
    max($"id").as("max_val"),
    percentile_approx($"id", lit(0.5), lit(100)),
    percentile_approx($"id", lit(0.5), lit(100)),
    min($"id").as("min_val"))

df.collect()

Was this patch authored or co-authored using generative AI tooling?

No

[ivoson]

cc @cloud-fan can you please take a look? Thanks.

[cloud-fan]

thanks, merging to master/4.0!