Add range response KV length as a metric #16881
Conversation
This adds a metric to allow us to alert on large range responses described in etcd-io#14809. Signed-off-by: Thomas Jungblut <[email protected]>
server/etcdserver/v3_server.go
Outdated
| @@ -136,6 +136,9 @@ func (s *EtcdServer) Range(ctx context.Context, r *pb.RangeRequest) (*pb.RangeRe | |||
| err = serr | |||
| return nil, err | |||
| } | |||
|
|
|||
| rangeResponseKvCount.WithLabelValues(string(r.Key)).Observe(float64(len(resp.Kvs))) | |||
There was a problem hiding this comment.
given the key label could cause a very large /metrics response, we might want to only emit this on n > 1000 or so. Also thought about adding this to the expensive trace request above.
Let me know what you guys think.
There was a problem hiding this comment.
Hey @tjungblu - Thanks for raising this idea. With the etcd project making tentative steps towards adopting the kubernetes enhancements process for new features or meaningful decisions this addition could be a potential item to first have a kep for sig/etcd.
Just an idea if you wanted a larger pool of eyes on the proposal. If you would rather just proceed here personally I am ok with it.
Is there any way you can provide some qualification of what impact this new metric would have on the response size? 🤔
Signed-off-by: Thomas Jungblut <[email protected]>
|
Closing for now, need to rethink this somewhat with our stuff downstream. |
This adds a metric to allow us to alert on large range responses described in #14809.
Please read https://github.com/etcd-io/etcd/blob/main/CONTRIBUTING.md#contribution-flow.