Telemetry and Pipe Observability

[somombo]

A system built around data streaming should monitor the stream itself. Currently, impalab only records the duration_nanos that the executor self-reports.

• The Problem: If an executor handles a workload poorly, is it because the algorithm is slow, or because the process is suffering from memory bloat and garbage collection pauses?
• The Streaming Solution: Because impa is the orchestrator holding the process handles (exec_child), it is perfectly positioned to capture system-level telemetry. impa should independently measure:
  • Stream Throughput: (MB/s) passing through the Stdio::piped().
  • Process Metrics: Peak memory footprint (RSS) and CPU utilization of the executor while the stream is open.