Support a pod authorization list limiting what pods can expose metrics

[jmazzitelli]

Allow the agent's config to support a pod authorization list. Pods on the list are allowed to be monitored. Any pods not on the list are ignored.

[jmazzitelli]

There is now a "authorized_pods" entry in the agent config yaml - a list of pod names (in the form "namespace/name") that must match if pods are to have metrics collected for them. If no authorized list is given, all pods are authorized. You can also specify this via K8S_AUTHORIZED_PODS environment variable (the entries are comma-separated).

The authorized pods are actually regexps - so you can say either "my-namespace/my-pod-name" or use regex expressions like ".*/my-pod-name" or "my-namespace/pod-[ABC]"

This is PR #13

[mwringe]

Pod names are too dynamic to be able use it this way. The approach you want is to use service account instead.

But even with this you don't want to actually store the list of 'trusted' service accounts in the agents config, this would need to be done at the OpenShift policy level (and is something we need to figure out more).

Please leave out this feature until we figure out a better solution for how to handle this with the OpenShift integration we already have.

[jmazzitelli]

ok, I'll reject this PR. Feel free to enter more comments in the others issues or implement other PRs for ways you think things should be done. I'm at the end of my knowledge of Open Shift :) so not sure how things should be done.

[vnugent]

Perhaps adding the ability to select pods to monitor by labels?

[mwringe]

We already have an annotation we use to determine if a pod has metrics to collect or not. This is not about discovery. but how to limit metric gathering to only a certain list of authorised pods.

[jmazzitelli]

NOTE: if that annotation is missing on a pod (or refers to a non-existent config map), that pod is not monitored. However, IIUC, we want something in the agent itself to limit what pods it monitors (otherwise, any Joe Schmo Admin can add an annotation to his pod to ask the agent to monitor it even if we don't want it to be monitored) This is what I think this issue is trying to address - provide a way for someone to tell the agent "don't accept any pod that wants to be monitored unless it is a pod on "this" list.

[mwringe]

We also need to think about how exactly we want to handle this.

We could handle this based on roles. So we would have the default role and a more privileged role.

For the default role, we could allow these pod to only be able to write to some set maximum, where the maximum could be zero or more metrics. The maximum here would be set at the cluster level for the agent. For example, this could be used to allow any pod to write up to 10 metrics each, and would give them an option to use custom metrics for say the HPA.

The more privileged role could have no limit applied. This could be used for something like an infrastructure component to be allowed to monitor any number of metrics (eg registry, router), etc

The role based approach is pretty simple, and ties more easily into how role based authentication works in OpenShift. Its fairly simple, easy to conceptualise but it a bit rigid in that you can't easily just give someone permission to write X many metrics.

I am not sure if a more robust solution would be any better though. We would have to do this more on our own (eg can no longer rely on OpenShift policies) to be able to grant individual accounts permission to write X many metrics. I have a feeling that 99% of the time, an admin is not going to want to have to grant individual accounts permissions for this, its going to be too difficult to administer (we could easily be dealing with thousands on service accounts) and they are just going to end up using essentially a role based setup anyways.

I think this issue falls more into how to balance preventing a rogue pod from using up too many system resources (eg by asking for too many metrics to be gathered) and having a desire to collect a lot of metrics from certain infrastructure components.