support different identity key-pairs per endpoint

[jmazzitelli]

There may be cases where the agent's identity needs to change in order to use different key-pairs in order to talk to different endpoints. This means the one global agent Identity (found in the agent config) isn't enough - there needs to be a way to specify a key-pair Identity per endpoint to override the agent's Identity when talking to that specific endpoint.

See the hawkular-dev thread: http://lists.jboss.org/pipermail/hawkular-dev/2016-December/003526.html and specifically this post: http://lists.jboss.org/pipermail/hawkular-dev/2016-December/003531.html

[jmazzitelli]

For completeness, this is the content of the thread post I linked above:

Below are the steps i am doing;

1. The certs for etcd are here (see path below), both server and client. I am only interested in the client for the agent:

[root at localhost master] # pwd
/var/lib/origin/openshift.local.config/master

[root at localhost master] # ls -ltr etc
-rw-rw-rw-. 1 root root 1078 Sep 23 18:31 master.etcd-client.crt
-rw-rw-rw-. 1 root root 1679 Sep 23 18:31 master.etcd-client.key
-rw-rw-rw-. 1 root root 1675 Sep 23 18:31 etcd.server.key
-rw-rw-rw-. 1 root root 2550 Sep 23 18:31 etcd.server.crt

2. I add the client certs as secrets and mount them to the agent:

oc project openshift-infra
oc secrets new etcd-client-crt master.etcd-client.crt
oc secrets new etcd-client-key master.etcd-client.key

oc volume rc/hawkular-openshift-agent --add --name=etcd-client-crt --type=secret --secret-name=etcd-client-crt --mount-path=/run/secrets/etcd-client-crt
oc volume rc/hawkular-openshift-agent --add --name=etcd-client-key --type=secret --secret-name=etcd-client-key --mount-path=/run/secrets/etcd-client-key

3. Then edit the config map of the agent and add in the below, which matches the above secret mounts:

oc edit configmap hawkular-openshift-agent-configuration

identity:
cert_file: /run/secrets/etcd-client-crt/master.etcd-client.crt
private_key_file: /run/secrets/etcd-client-key/master.etcd-client.key

4. Restart the pod to force a refresh and check the logs, which shows:

I1228 10:20:18.799687 1 prometheus_metrics_collector.go:97] DEBUG: Told to collect [2] Prometheus metrics from [https://172.17.0.8:9779/metrics]
I1228 10:20:18.984615 1 metrics_storage.go:152] DEBUG: Stored datapoints for [2] metrics

I now have a working agent collecting from etcd.

Since etcd is mutual auth - maybe thats what is causing the confusion, as you keep mentioning server-certs - i am not sure how generating my own client certs helps.

But since you said try it, i have with the below commands but as i would expect, got a TLS error:

openssl req -newkey rsa:2048 -nodes -keyout agent.key -out agent.csr -subj "/C=UK/ST=Yorkshire/L=Leeds/O=Home/CN=hawkular-agent"
openssl x509 -signkey agent.key -in agent.csr -req -days 365 -out agent.crt

Just incase there are cross wires; etcd in OCP requires mutual auth (thats how i understand it), so thats the reason i am adding in the etcd client certs to the identity section of the agent configmap. If i needed to monitor another endpoint which was also mutual auth, with the current setup i wouldn't be able to do that.

[jmazzitelli]

You would think the simple solution would be to add support for an optional "agent_identity" section (which would contain the paths to the key-pair locations) to the pod configmap's "endpoints" definitions.

But the problem is that these endpoints are defined by your pods (they are config maps on the projects where the pods are deployed) and the people doing those pod deployments are typically not the same admins who configure and set up the agent. So how would the pod configmap authors know the locations and names of the key-pairs or have privileges to install those keys on the agent (I'm thinking of the use case where OpenShift is running and developers/deployers are deploying their own applications and want to collect their own app metrics).

[garethahealy]

For my etcd example, i'd expect this to be setup by an admin (the certs are owned by root on the master) so the 'admin vs developer' problem you've highlighted isn't an issue for that use case. However, it might be for other use cases as you've stated, so happy to hear what other viewpoints are.

[mwringe]

Configmaps can store files.

But the Agent should not have different key-pair values when talking to an endpoint. The agent should have a more defined client certificate that it uses when communicating with an endpoint and that endpoint should be configured to accept connections from the agent. Configuring the agent to present a different certificate based on the endpoint its accessing seems wrong.

For mutual auth, we might have to include the something like the CA cert in the configmap (to support things like self signed certificates)

Overall I think we need to think about how to handle security better to support different endpoint types