Gobblin Metrics Performance

This document explains the performance impact of using Gobblin Metrics in applications.

Generalities

These are the main resources used by Gobblin Metrics:

• CPU time for updating metrics: scales with number of metrics and frequency of metric update
• CPU time for metric emission and lifecycle management: scales with number of metrics and frequency of emission
• Memory for storing metrics: scales with number of metrics and metric contexts
• I/O for reporting metrics: scales with number of metrics and frequency of emission
• External resources for metrics emission (e.g. HDFS space, Kafka queue space, etc.): scales with number of metrics and frequency of emission

Update Metrics Performance

Metric updates are the most common interaction with Gobblin Metrics in an application. Every time a counter is increased, a meter is marked, or entries are added to histograms and timers, an update happens. As such, metric updates are the most likely to impact application performance. We measured the max number of metric updates that can be executed per second. The performance of different metric types is different. Also, the performance of metrics depends on the depth in the Metric Context tree at which they are created. Metrics in the Root Metric Context are the fastest, while metrics deep in the tree are slower because they have to update all ancestors as well. The following table shows reference max QPS (updates per second) for each metric type in a i7 processor:

Metric	Root level	Depth: 1	Depth: 2	Depth: 3
Counter	76M	39M	29M	24M
Meter	11M	7M	4.5M	3.5M
Histogram	2.4M	2.4M	1.8M	1.3M
Timer	1.4M	1.4M	1M	1M