Detecting when a rule pipeline cannot keep up with the incoming data stream?

[milan-mkip]

Hi,

I'm setting up an eKuiper rule pipeline to handle potentially 100s or 1000s of tags from a Neuron data stream. The first rule runs an unnest on the batch, and latter rules handle the individual values and produce outputs.

The eKuiper instance is running on a device with limited hardware resources. As a result the pipeline is not keeping up with the data stream when the amount of tags or the update interval is too high. Some of the latter rules aren't being executed consistently or at all.

My question is if eKuiper provides any logs or other indicators that would allow me to detect when/if some rules aren't being triggered consistently? Even some indication of skipped values or high latency would be useful.

Or do I have to rely on monitoring the rule outputs to see when they are starved of hardware resources?

[ngjaying]

You can monitor a rule through metrics: https://ekuiper.org/docs/en/latest/operation/usage/monitor_with_prometheus.html#monitor-rule-status-with-prometheus.

[milan-mkip]

Thank you for the answer. The exception fields in the rule status combined with the timestamps should provide enough information.