Base Carbon behavior leads to silent data loss for critical systems #476
Description
In some systems that rely on Carbon as a datasource layer, any data loss is unacceptable — for example, services that must operate on a complete, continuous stream of blocks or events.
However, the current Carbon behavior does not provide any guarantees or even visibility into whether data was lost during datasource disconnections. When a datasource (Yellowstone gRPC or RPC WebSocket) silently hangs, disconnects, or reconnects, Carbon resumes consumption without indicating which part of the data stream was missed.
As a result:
• A portion of data may be skipped without any detection.
• There is no way to recover or even understand what exactly was lost.
• Consumers cannot implement compensating logic because Carbon provides no structured information about gaps or disconnections.
This makes Carbon unsuitable for critical scenarios where uninterrupted data continuity is required.
Why this PR is needed
The proposed fix adds full disconnection tracking and missed-slot gap detection across both Yellowstone gRPC and RPC WebSocket datasources, making it possible to build reliable systems on top of Carbon.
The new functionality provides:
• Structured DatasourceDisconnection events with source identification.
• Detection of hung connections via 30-second timeouts.
• Accurate calculation of gaps (missed slots) between disconnect and reconnect.
• Warning-level logging for visibility into disconnections.
• Optional notification channels so consumers can act on failures in real time.
• Persistent state across reconnection loops for correct gap computation.
With this feature, Carbon becomes suitable for high-reliability systems where data loss must be observable, traceable, and recoverable.