[Enhancement] Enhance the observability of routine load job #48511

sollhui · 2025-03-02T08:03:36Z

Search before asking

I had searched in the issues and found no similar issues.

Description

At present, the observability of routine load job is relatively poor. This issue plans to enhance the observability of routine load from three tasks:

Error msg can be displayed when show routine load job if exception happened. [improve](routine load) improve routine load observability #46238
Add more metrics to observe the routine load job. [improve](routine load) add more metrics to observe the routine load job #48209
Introduce routine load abnormal job monitor. For a job that requires long-term stable operation, ensuring high availability is important, and monitoring for abnormal jobs is also crucial. The goal is to quickly perceive and locate which job is at fault when an exception occurs. [improve](routine load) introduce routine load abnormal job monitor #48171

Solution

No response

Are you willing to submit PR?

Yes I am willing to submit a PR!

Code of Conduct

I agree to follow this project's Code of Conduct

…job (#48209) ### What problem does this PR solve? related #48511 Add more metrics to observe the routine load job: | Metrics | Module | Description | | ---------------------------------- | ------ | ------------------------------- | | routine_load_get_msg_latency | BE | Time to pull a Kafka message | | routine_load_get_msg_count | BE | Number of times pulling Kafka messages | | routine_load_consume_bytes | BE | Total data volume consumed from Kafka | | routine_load_consume_rows | BE | Total number of rows consumed from Kafka | | routine_load_task_execute_time | FE | Task execution time | | routine_load_task_execute_count | FE | Task execution count | | routine_load_get_meta_latency | FE | Delay in obtaining Kafka metadata | | routine_load_get_meta_count | FE | Number of times obtaining Kafka metadata | | routine_load_get_meta_fail_count | FE | Number of failures in obtaining metadata | | routine_load_received_bytes | FE | Total data volume consumed | | routine_load_received_rows | FE | Total number of rows consumed |

This was referenced Mar 2, 2025

[improve](routine load) improve routine load observability #46238

Merged

[improve](routine load) introduce routine load abnormal job monitor #48171

Open

[improve](routine load) add more metrics to observe the routine load job #48209

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Enhancement] Enhance the observability of routine load job #48511

[Enhancement] Enhance the observability of routine load job #48511

sollhui commented Mar 2, 2025 •

edited

Loading

[Enhancement] Enhance the observability of routine load job #48511

[Enhancement] Enhance the observability of routine load job #48511

Comments

sollhui commented Mar 2, 2025 • edited Loading

Search before asking

Description

Solution

Are you willing to submit PR?

Code of Conduct

sollhui commented Mar 2, 2025 •

edited

Loading