The following document covers various topics for configuring and integrating your Open Liberty runtime with monitoring tools in a Kubernetes environment. This document has been tested and is supported with Red Hat OpenShift Container Platform (RHOCP) versions 4.14, 4.15, 4.16, and 4.17.
Open Liberty logs inside a Kubernetes environment can be managed by deploying log aggregation tools onto a Kubernetes cluster. Log aggregation tools help you persist, search, and visualize the log data that is gathered from the pods across the cluster.
One choice for application logging with log aggregation is the open source Elasticsearch, Fluentd, and Kibana (EFK) stack. If you use the EFK stack, you can customize and deploy Kibana dashboards to monitor Open Liberty logging events.
Kibana dashboards are provided for visualizing events from the Open Liberty runtime.
To leverage these dashboards the logging events must be emitted in JSON format to standard-out. For Open Liberty Operator, JSON logging is enabled by default. For information regarding how to configure an Open Liberty image with JSON logging please see here.
Retrieve available Kibana dashboards for displaying Open Liberty logging events here.
For information regarding how to import Kibana dashboards see the official documentation here.
For effective management of logs emitted from applications, deploy your own Elasticsearch, Fluentd and Kibana (EFK) stack. If you are deploying the EFK stack on OpenShift, see the following guide. Note: The instructions in the guide for deploying the EFK stack on OpenShift are only valid up to RHOCP version 4.16.
Another choice for application logging with log aggregation is by using Loki and Vector. If you are deploying Loki and Vector on OpenShift, see the following guide.
Command-line JSON parsers, like JSON Query tool (jq), can be used to create human-readable views of JSON-formatted logs. You can create an alias for your jq command in the format you like. In the following example, the logs are piped through grep to ensure that the message field is there before jq parses the line:
alias prettylog="grep --line-buffered message | jq '.ibm_datetime + \" \" + .loglevel + \"\\t\" + \" \" + .message' -r"
kubectl logs -f pod_name -n namespace | prettylogA MicroProfile Metrics enabled Open Liberty runtime is capable of tracking and observing metrics from the JVM and Open Liberty server as well as tracking MicroProfile Metrics instrumented within a deployed application. The tracked metrics can then be scraped by Prometheus and visualized with Grafana.
The following steps outline how to manually create and modify a server.xml to add the MicroProfile Metrics feature that will be built as part of your Open Liberty image. You can use any version of the MicroProfile Metrics feature. If you use a version below mpMetrics-2.3, you should also include the monitor-1.0 feature to add Liberty server metrics. MicroProfile Metrics feature versions starting from mpMetrics-2.3 automatically load monitor-1.0 as a dependency.
-
Create an XML file named
server_mpMetrics.xmlwith the following contents and place it in the same directory as your Dockerfile:<?xml version="1.0" encoding="UTF-8"?> <server> <featureManager> <feature>mpMetrics-5.1</feature> <!-- include the monitor-1.0 feature below if you are using an mpMetrics version below 2.3 <feature>monitor-1.0</feature> --> </featureManager> <quickStartSecurity userName="${env.username}" userPassword="${env.password}"/> </server>
The above
server.xmlconfiguration secures access to the server with basic authentication using the<quickStartSecurity>element. The<quickStartSecurity>is used in the above example for simplicity. When configuring your server you may wish to use a basic registry or an LDAP registry for securing authenticated access to your server. When using Prometheus to scrape data from the/metricsendpoint only the Service Monitor approach can be configured to negotiate authentication with the Open Liberty server.The two environment variables in the
<quickStartSecurity>element,usernameandpassword, are used to avoid hardcoded, clear text authentication credentials in theserver.xml. Follow the steps in the following section, Using environment variables for basic authentication credentials, to set up these two environment variables with your username and password of choice. -
In your DockerFile, add the following line to copy the
server_mpMetrics.xmlfile into theconfigDropins/overridesdirectory:COPY --chown=1001:0 server_mpMetrics.xml /config/configDropins/overrides/
The following steps outline how to set up two environment variables that are used in the <quickStartSecurity> element for basic authentication, username and password, with your username and password of choice after your Open Liberty image is deployed onto your Kubernetes cluster.
-
Create a secret with your desired username and password in your Kubernetes cluster.
-
Modify your OpenLibertyApplication Custom Resource (CR) to add the
envFromdefinition with your secret referenced; in the following example, replacebasic-authwith your secret:spec: envFrom: - secretRef: name: basic-auth
-
Ensure your application container has access to the secret you created.
This envFrom configuration sets two environment variables for your application container, username and password, using your secret’s respective username and password values.
You will need to deploy Prometheus using the Prometheus Operator which will then utilize Service Monitors to monitor and scrape logs from target services. If you are deploying Prometheus on Openshift, see the following guide.
After deploying Prometheus, you must configure your application’s Service Monitor to use the basic authentication credentials specified in your server.xml when accessing the /metrics endpoint. Use the Service Monitor’s basicAuth definition with a secret that contains those credentials; this should be the same secret you created earlier in Step 1 of Using environment variables for basic authentication credentials.
Add the following basicAuth section to your OpenLibertyApplication CR under the monitoring definition, and replace basic-auth with your secret:
spec:
…
monitoring:
endpoints:
- basicAuth:
username:
key: username
name: basic-auth
password:
key: password
name: basic-auth
interval: 30s
tlsConfig:
insecureSkipVerify: trueThere are IBM provided Grafana dashboards that leverage metrics from the JVM as well as from the Open Liberty runtime. If you are deploying Grafana on OpenShift, see the following guide.
Sample Open Liberty Grafana dashboards are available for servers using either mpMetrics-1.x or mpMetrics-2.x here. Look in the featureManager section of the server.xml for either the mpMetrics feature or the umbrella microProfile feature to determine which dashboard to use.
Umbrella Feature |
mpMetrics Feature |
Dashboard |
microProfile-1.2 - microProfile 2.2 |
mpMetrics-1.x |
ibm-websphere-liberty-grafana-dashboard.json |
microProfile-3.0 - microProfile-6.1 |
mpMetrics-2.x - mpMetrics-5.1 |
ibm-websphere-liberty-grafana-dashboard-metrics-2.0.json |
MicroProfile Health allows services to report their readiness, liveness, and startup health statuses (i.e UP if it is ready/alive and DOWN if its not ready/alive) through three endpoints. The Health data will be available on the /health/live, /health/ready, and health/started endpoints for the liveness, readiness, and startup health checks respectively.
Readiness check allows third party services to know if the service is ready to process requests or not. e.g., dependency checks, such as database connections, application initialization, etc.
Liveness check allows third party services to determine if the service is running. This means that if this procedure fails the service can be discarded (terminated, shutdown).
Startup health check allows applications to define startup probes that are used for initial verification of the application before the liveness probe takes over.
MicroProfile Health reports an individual service’s status at the endpoints, and indicates the overall status as UP if all the services are UP. A service orchestrator can then use these health check statuses to make decisions.
The following steps outline how to manually create and modify a server.xml to add the MicroProfile Health feature that will be built as part of your Open Liberty image.
Configure the MicroProfile Health feature in your server.xml:
-
Create an XML file named
server_mpHealth.xml, with the following contents and place it in the same directory as your DockerFile:<?xml version="1.0" encoding="UTF-8"?> <server> <featureManager> <feature>mpHealth-4.0</feature> </featureManager> </server>
-
In your DockerFile, add the following line to copy the
server_mpHealth.xmlfile into theconfigDropins/overridesdirectory:COPY --chown=1001:0 server_mpHealth.xml /config/configDropins/overrides/
Configure the Kubernetes Liveness, Readiness, and Startup Probes to use the MicroProfile Health REST Endpoints
Kubernetes provides liveness, readiness, and startup probes that are used to check the health of your containers. These probes can check certain files in your containers, check a TCP socket, make HTTP requests, or protect slow starting containers.
Configure the readiness, liveness, and startup probes' fields to point to the MicroProfile Health REST endpoints. MicroProfile Health versions starting from mpHealth-3.1 include the addition of Kubernetes startup probes.
Modify the readiness and liveness probes' fields, if not configured, to point to the MicroProfile Health REST endpoints, in the OpenLibertyApplication Custom Resource (CR) for the Runtime Component Operator v0.8.0+:
spec:
applicationImage:
...
probes:
liveness:
failureThreshold: 12
httpGet:
path: /health/ready
port: 9443
scheme: HTTPS
initialDelaySeconds: 30
periodSeconds: 2
timeoutSeconds: 10
readiness:
failureThreshold: 12
httpGet:
path: /health/live
port: 9443
scheme: HTTPS
initialDelaySeconds: 30
periodSeconds: 2
timeoutSeconds: 10
...Modify the readiness, liveness, and startup probes' fields, if not configured, to point to the MicroProfile Health REST endpoints, in the OpenLibertyApplication Custom Resource (CR):
spec:
applicationImage:
...
probes:
liveness:
failureThreshold: 12
httpGet:
path: /health/ready
port: 9443
scheme: HTTPS
initialDelaySeconds: 30
periodSeconds: 2
timeoutSeconds: 10
readiness:
failureThreshold: 12
httpGet:
path: /health/live
port: 9443
scheme: HTTPS
initialDelaySeconds: 30
periodSeconds: 2
timeoutSeconds: 10
startup:
failureThreshold: 12
httpGet:
path: /health/started
port: 9443
scheme: HTTPS
initialDelaySeconds: 30
periodSeconds: 2
timeoutSeconds: 10
...Using the operator, you can enable the serviceability definition in your OpenLibertyApplication Custom Resource to create a PersistentVolumeClaim so that the logs from your application go to a single storage. Your cluster must either be configured to automatically bind the PersistentVolumeClaim to a PersistentVolume or you must bind it manually.
The serviceability.size definition in the following example will automatically create a PersistentVolumeClaim with the specified size and is shared between all pods of the OpenLibertyApplication instance. For more information on the serviceability definition provided by the operator, please see the following user guide.
Add the serviceability.size definition in your OpenLibertyApplication Custom Resource; the PersistentVolumeClaim should be created with the name <application_name>-serviceability:
spec:
applicationImage:
...
serviceability:
size: 1Gi