[8.1.x][ANSIENG-4730] | Implement logrotate for prometheus & alertmanager

[abhijeet2096-confluent]

Description

Please include a summary of the change and which issue is fixed. Please also include relevant motivation and context. List any dependencies that are required for this change.

Fixes # (issue)

Type of change

• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• Breaking change (fix or feature that would cause existing functionality to not work as expected)
• This change requires a documentation update

How Has This Been Tested?

Please describe the tests that you ran to verify your changes. Provide instructions so we can reproduce. Please also list any relevant details for your test configuration

Checklist:

• Any variable/code changes have been validated to be backwards compatible (doesn't break upgrade)
• I have added tests that prove my fix is effective or that my feature works
• If required, I have ensured the changes can be discovered by cp-ansible discovery codebase
• My code follows the style guidelines of this project
• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation
• My changes generate no new warnings
• Any dependent changes have been merged and published in downstream modules

[abhijeet2096-confluent]

Demo & Troubleshooting

root@control-center-next-gen:/var/log/confluent/control-center# date
Tue Oct 28 10:39:02 UTC 2025

root@control-center-next-gen:/var/log/confluent/control-center# sudo journalctl -u cron -f
Oct 28 10:20:01 control-center-next-gen.confluent CRON[31030]: (root) CMD (/usr/local/bin/logrotate-control-center-next-gen-alertmanager.sh >> /tmp/control-center-next-gen-alertmanager-rotate.log 2>&1)
Oct 28 10:20:01 control-center-next-gen.confluent CRON[31031]: (root) CMD (/usr/local/bin/logrotate-control-center-next-gen-prometheus.sh >> /tmp/control-center-next-gen-prometheus-rotate.log 2>&1)
Oct 28 10:20:01 control-center-next-gen.confluent CRON[31028]: pam_unix(cron:session): session closed for user root
Oct 28 10:20:01 control-center-next-gen.confluent CRON[31029]: pam_unix(cron:session): session closed for user root
Oct 28 10:30:01 control-center-next-gen.confluent CRON[31055]: pam_unix(cron:session): session opened for user root(uid=0) by root(uid=0)
Oct 28 10:30:01 control-center-next-gen.confluent CRON[31056]: pam_unix(cron:session): session opened for user root(uid=0) by root(uid=0)
Oct 28 10:30:01 control-center-next-gen.confluent CRON[31058]: (root) CMD (/usr/local/bin/logrotate-control-center-next-gen-alertmanager.sh >> /tmp/control-center-next-gen-alertmanager-rotate.log 2>&1)
Oct 28 10:30:01 control-center-next-gen.confluent CRON[31057]: (root) CMD (/usr/local/bin/logrotate-control-center-next-gen-prometheus.sh >> /tmp/control-center-next-gen-prometheus-rotate.log 2>&1)
Oct 28 10:30:01 control-center-next-gen.confluent CRON[31055]: pam_unix(cron:session): session closed for user root
Oct 28 10:30:01 control-center-next-gen.confluent CRON[31056]: pam_unix(cron:session): session closed for user root
Oct 28 10:40:01 control-center-next-gen.confluent CRON[31149]: pam_unix(cron:session): session opened for user root(uid=0) by root(uid=0)
Oct 28 10:40:01 control-center-next-gen.confluent CRON[31148]: pam_unix(cron:session): session opened for user root(uid=0) by root(uid=0)
Oct 28 10:40:01 control-center-next-gen.confluent CRON[31150]: (root) CMD (/usr/local/bin/logrotate-control-center-next-gen-alertmanager.sh >> /tmp/control-center-next-gen-alertmanager-rotate.log 2>&1)
Oct 28 10:40:01 control-center-next-gen.confluent CRON[31151]: (root) CMD (/usr/local/bin/logrotate-control-center-next-gen-prometheus.sh >> /tmp/control-center-next-gen-prometheus-rotate.log 2>&1)
Oct 28 10:40:01 control-center-next-gen.confluent CRON[31148]: pam_unix(cron:session): session closed for user root
Oct 28 10:40:01 control-center-next-gen.confluent CRON[31149]: pam_unix(cron:session): session closed for user root

root@control-center-next-gen:/var/log/confluent/control-center# ls -lh /var/log/confluent/control-center/
total 8.0M
-rw-r-----. 1 cp-control-center confluent 2.9K Oct 28 09:13 alertmanager.log
-rw-r--r--. 1 cp-control-center confluent 1.3M Oct 28 10:34 control-center-kafka.log
-rw-r--r--. 1 cp-control-center confluent 547K Oct 28 10:40 control-center-streams.log
-rw-r--r--. 1 cp-control-center confluent 4.9M Oct 28 10:40 control-center.log
-rw-r-----. 1 cp-control-center confluent    0 Oct 28 10:40 prometheus.log
-rw-r-----. 1 cp-control-center confluent 8.3K Oct 28 10:40 prometheus.log.1
root@control-center-next-gen:/var/log/confluent/control-center# 

root@control-center-next-gen:/var/log/confluent/control-center# cat /etc/logrotate.d/control-center-next-gen-prometheus 
/var/log/confluent/control-center/prometheus.log {
    size 1K
    rotate 5
    compress
    delaycompress
    missingok
    notifempty
    copytruncate
    su cp-control-center confluent
}

To force log rotation use following command

   sudo /usr/local/bin/logrotate-control-center-next-gen-prometheus.sh

[rkunwar-28]

Code changes look fine. How have we tested this? Ideally, we should spin up a setup with a very small value for file size limit and make sure that new log files are getting created everytime that file size is breached.

[abhijeet2096-confluent]

Yes testing details here #2306 (comment)

[rkunwar-28]

Missed the demo screenshot above, this is fine. Prometheus log file size is set to 1K and it is getting rotated properly

[rkunwar-28]

LGTM, thanks! Good idea to incorporate the validations, we can add similar validations for USM Agent as well.