Support API Proxy Access

[mwringe]

There are already some containers which are configured to be accessed via the API Proxy in OpenShift.

This means it would be nice to be able to specify that we want to be able to access the endpoint via the API Proxy URL instead of directly accessing the pod's ip address itself.

This is probably not a great idea (I don't think this will scale very well or be as performant as it could be). But this is probably a stop gap measure we will need to put in place until we have better adoption and can get our middleware containers updated.

[jmazzitelli]

documenting this mainly for me so I remember :)

A typical Jolokia URL over the API proxy would look like (this gets the heap memory usage from the Memory MBean):

https://master-ip:8443/api/v1/namespaces/openshift-infra/pods/http:hawkular-openshift-agent-example-jolokia-wildfly-r5rr6:8080/proxy/jolokia-war-1.3.5/read/java.lang:type=Memory/HeapMemoryUsage

Using curl and authenticating with the master API bearer token, this is what the response looks like (its just a Jolokia REST response):

curl -k -H "Authorization: Bearer `oc whoami -t`" https://master-ip:8443/api/v1/namespaces/openshift-infra/pods/http:hawkular-openshift-agent-example-jolokia-wildfly-r5rr6:8080/proxy/jolokia-war-1.3.5/read/java.lang:type=Memory/HeapMemoryUsage

{"request":{"mbean":"java.lang:type=Memory","attribute":"HeapMemoryUsage","type":"read"},"value":{"init":67108864,"committed":216530944,"max":477626368,"used":159009696},"timestamp":1484073697,"status":200}

So this means we need to:

1. talk to the master server over https and thus we either need to do so without verification (as I do above with curl's -k option) or we need the proper trusted CA file
2. authenticate with the master server (I do so above with the bearer token)
3. know the pod namespace (e.g. "openshift-infra"), pod name (e.g. "hawkular-openshift-agent-example-jolokia-wildfly-r5rr6"), the port ("8080"), the path (e.g. "/jolokia-war-1.3.5"), and the metric to collect (for Jolokia, that's the MBean name and its attribute - for Prometheus we'd just pull the /metrics endpoint which contains all the metrics). All of this the agent already knows either through watcher discovery or via the configmap.

The above example assumes we are connecting to the Jolokia endpoint over unsecured http. If its to be https, we need to know what certificate to use and what (if any) credentials there will be - though I don't think the proxy passes through credentials. Being able to support https in the backend pod was added a little over a year ago - see openshift/origin#4497 - we will need to configure the agent in a way so it knows what it has to pass via the proxy request (the proper cert and/or credentials of the backend pod).

[ghost]

Is there any way to impersonate the API proxy? I mean, if somehow we can get the master-proxy certificate and use it, maybe we can do something in the meantime.

For example I have an amq pod running on my OCP, and I have set a service and a route for jolokia port with passthrough.

If, from the master, execute:

curl --insecure --cert ./master.proxy-client.crt --key ./master.proxy-client.key https://amq-demo-app-amq-jolokia-amq-demo.apps.172.17.0.1.xip.io/jolokia/

I get:

{"request":{"type":"version"},"value":{"agent":"1.3.2","protocol":"7.2","config":{"maxDepth":"15","discoveryEnabled":"false","maxCollectionSize":"0","agentId":"172.17.0.11-1-2503dbd3-jvm","debug":"false","agentType":"jvm","historyMaxEntries":"10","agentContext":"/jolokia","maxObjects":"0","debugMaxEntries":"100"},"info":{"product":"activemq","vendor":"Apache","version":"5.11.0.redhat-621159"}},"timestamp":1487768223,"status":200}

But we need to know what does it implies getting this cert/key from inside if possible.

[mwringe]

Is there any way to impersonate the API proxy?

Yes, its not something we currently provide an easy mechanism to do currently. In essence you would need to add the master-proxy certificate and key to a secret, load the secret into your pod and tell the agent to use that client certificate.

@jmazzitelli what are the parameter names for the client certs for the agent?

[jmazzitelli]

what are the parameter names for the client certs for the agent?

If you are asking about the agent's certificate/private key it uses when it sends https requests to collect metrics from endpoints, you do this by assigning an identity to the agent via a cert file and private key file in the agent global config like this:

identity:
  private_key_file: <path to mounted private key file>
  cert_file: <path to mounted certificate file>

When the agent makes https requests to any endpoint, it uses those to identify itself to the remote endpoint.

If the endpoint additionally requires credentials when making these https requests (such as username/password or token), those are specified in the endpoint configurations (the configmaps).

[ghost]

Ok. That was exactly what I was trying. I has some advances, but now I am getting this error:

W0223 09:44:56.467586 1 metrics_collector_manager.go:181] Failed to collect metrics from [amq-demo|amq-demo-app-amq-2-jel3r|jolokia|https://172.17.0.11:8778/jolokia] at [Thu, 23 Feb 2017 09:44:56 +0000]. err=Failed to collect metrics from Jolokia endpoint [https://172.17.0.11:8778/jolokia]. err=Post https://172.17.0.11:8778/jolokia: x509: cannot validate certificate for 172.17.0.11 because it doesn't contain any IP SANs

I think it could be because of the server certificate. Is there any way to disable this validation? If you see from my "curl" example, I also needed to add the "--insecure" flag to work with.

[mwringe]

I think it could be because of the server certificate. Is there any way to disable this validation?

Its because the Jolokia endpoints are self signing there certificates and its not valid for direct IP address access (and probably fail the next check if the CA which signed the certificate is trusted or not).

Your pod can specify that it doesn't want its certificate to be validated by the agent. See https://github.com/hawkular/hawkular-openshift-agent/blob/master/examples/jolokia-wildfly-example/jolokia-wildfly.yaml#L77

[ghost]

Ok, now I am able to get metrics from jolokia of our amq image without any change. Steps:

First, create the master-proxy secret in openshift-infra namespace from master:

oc secrets new master-proxy $OPENSHIFT_CONFIG/master/master.proxy-client.crt $OPENSHIFT_CONFIG/master/master.proxy-client.key -n openshift-infra

Add this cert to mount in DaemonSet for HOSA:

spec:
  volumes:
    - name: master-proxy
      secret:
        secretName: master-proxy
        defaultMode: 420

...

  volumeMounts:
    - name: master-proxy
      mountPath: /master-proxy

Then, configure the HOSA identity with this cert:

identity:
  cert_file: /master-proxy/master.proxy-client.crt
  private_key_file: /master-proxy/master.proxy-client.key

And add to your AMQ dc a hawkular-openshift-agent configmap with:

endpoints:
- type: jolokia
  protocol: https
  port: 8778
  path: /jolokia/read
  tls:
    skip_certificate_validation: true

BTW, I can get the java.lang parameters, but the a-mq context in my case is:

org.apache.activemq:type=Broker,brokerName=amq-demo-app-amq-2-jel3r/TotalConnectionsCount

As you can see, the brokerName is the pod name. Is there any way to add this kind of parameters (like ${POD:name}) in the metrics? i.e:

  - name: org.apache.activemq:type=Broker,brokerName=${POD:name}#TotalConnectionsCount
    type: counter
    id:   total_connections_count

[jmazzitelli]

Is there any way to add this kind of parameters (like ${POD:name}) in the metrics?

Those are allowed in the tags: section and allowed in the "id" field (to let you define your own metric ID if you so choose) but not allowed in the name field itself. But I think that would be easy to add. If you feel it is something you require, enter another github issue.

[ghost]

Done. https://github.com/hawkular/hawkular-openshift-agent/issues/142