Unauthorized metrics endpoint

[poligraph]

Hello,

in deploy yaml file I see that service has following annotation:

prometheus.io/scrape: "true"

But when my prometheus tries to scrape metrics, it says Unauthorized

I tried to check it with curl and got the same

curl -k https://10.1.41.10:8443/metrics
Unauthorized

Could you please explain, why /metrics endpoint has authorization and how to disable it?

BTW, I did not find list of exposed metrics.

Could please provide?

Thanks.

[rosmo]

I don't think we expose any metrics at the moment - the metrics endpoint is probably from kubebuilder scaffolding. Are there some specific metrics that you would be interested in?

[romankor]

I see there are metrics exposed on /metrics

# HELP certwatcher_read_certificate_errors_total Total number of certificate read errors
# TYPE certwatcher_read_certificate_errors_total counter
certwatcher_read_certificate_errors_total 0
# HELP certwatcher_read_certificate_total Total number of certificate reads
# TYPE certwatcher_read_certificate_total counter
certwatcher_read_certificate_total 0
# HELP controller_runtime_active_workers Number of currently used workers per controller
# TYPE controller_runtime_active_workers gauge
controller_runtime_active_workers{controller="service"} 1
# HELP controller_runtime_max_concurrent_reconciles Maximum number of concurrent reconciles per controller
# TYPE controller_runtime_max_concurrent_reconciles gauge
controller_runtime_max_concurrent_reconciles{controller="service"} 1
# HELP controller_runtime_reconcile_errors_total Total number of reconciliation errors per controller
# TYPE controller_runtime_reconcile_errors_total counter
controller_runtime_reconcile_errors_total{controller="service"} 1
# HELP controller_runtime_reconcile_time_seconds Length of time per reconciliation per controller
# TYPE controller_runtime_reconcile_time_seconds histogram
controller_runtime_reconcile_time_seconds_bucket{controller="service",le="0.005"} 845
controller_runtime_reconcile_time_seconds_bucket{controller="service",le="0.01"} 846
controller_runtime_reconcile_time_seconds_bucket{controller="service",le="0.025"} 846
controller_runtime_reconcile_time_seconds_bucket{controller="service",le="0.05"} 846
controller_runtime_reconcile_time_seconds_bucket{controller="service",le="0.1"} 846
controller_runtime_reconcile_time_seconds_bucket{controller="service",le="0.15"} 846
controller_runtime_reconcile_time_seconds_bucket{controller="service",le="0.2"} 846
controller_runtime_reconcile_time_seconds_bucket{controller="service",le="0.25"} 846
controller_runtime_reconcile_time_seconds_bucket{controller="service",le="0.3"} 846
controller_runtime_reconcile_time_seconds_bucket{controller="service",le="0.35"} 846
controller_runtime_reconcile_time_seconds_bucket{controller="service",le="0.4"} 846
controller_runtime_reconcile_time_seconds_bucket{controller="service",le="0.45"} 846
controller_runtime_reconcile_time_seconds_bucket{controller="service",le="0.5"} 846
controller_runtime_reconcile_time_seconds_bucket{controller="service",le="0.6"} 846
controller_runtime_reconcile_time_seconds_bucket{controller="service",le="0.7"} 846
controller_runtime_reconcile_time_seconds_bucket{controller="service",le="0.8"} 846
controller_runtime_reconcile_time_seconds_bucket{controller="service",le="0.9"} 846
controller_runtime_reconcile_time_seconds_bucket{controller="service",le="1"} 846
controller_runtime_reconcile_time_seconds_bucket{controller="service",le="1.25"} 846
controller_runtime_reconcile_time_seconds_bucket{controller="service",le="1.5"} 846
controller_runtime_reconcile_time_seconds_bucket{controller="service",le="1.75"} 846
controller_runtime_reconcile_time_seconds_bucket{controller="service",le="2"} 846
controller_runtime_reconcile_time_seconds_bucket{controller="service",le="2.5"} 846
controller_runtime_reconcile_time_seconds_bucket{controller="service",le="3"} 846
controller_runtime_reconcile_time_seconds_bucket{controller="service",le="3.5"} 846
controller_runtime_reconcile_time_seconds_bucket{controller="service",le="4"} 846
controller_runtime_reconcile_time_seconds_bucket{controller="service",le="4.5"} 846
controller_runtime_reconcile_time_seconds_bucket{controller="service",le="5"} 846
controller_runtime_reconcile_time_seconds_bucket{controller="service",le="6"} 846
controller_runtime_reconcile_time_seconds_bucket{controller="service",le="7"} 846
controller_runtime_reconcile_time_seconds_bucket{controller="service",le="8"} 846
controller_runtime_reconcile_time_seconds_bucket{controller="service",le="9"} 846
controller_runtime_reconcile_time_seconds_bucket{controller="service",le="10"} 846
controller_runtime_reconcile_time_seconds_bucket{controller="service",le="15"} 846
controller_runtime_reconcile_time_seconds_bucket{controller="service",le="20"} 847
controller_runtime_reconcile_time_seconds_bucket{controller="service",le="25"} 847
controller_runtime_reconcile_time_seconds_bucket{controller="service",le="30"} 847
controller_runtime_reconcile_time_seconds_bucket{controller="service",le="40"} 847
controller_runtime_reconcile_time_seconds_bucket{controller="service",le="50"} 847
controller_runtime_reconcile_time_seconds_bucket{controller="service",le="60"} 847
controller_runtime_reconcile_time_seconds_bucket{controller="service",le="+Inf"} 847
controller_runtime_reconcile_time_seconds_sum{controller="service"} 18.14909732
controller_runtime_reconcile_time_seconds_count{controller="service"} 847
# HELP controller_runtime_reconcile_total Total number of reconciliations per controller
# TYPE controller_runtime_reconcile_total counter
controller_runtime_reconcile_total{controller="service",result="error"} 1
controller_runtime_reconcile_total{controller="service",result="requeue"} 0
controller_runtime_reconcile_total{controller="service",result="requeue_after"} 0
controller_runtime_reconcile_total{controller="service",result="success"} 846
# HELP go_gc_duration_seconds A summary of the pause duration of garbage collection cycles.
# TYPE go_gc_duration_seconds summary
go_gc_duration_seconds{quantile="0"} 3.202e-05
go_gc_duration_seconds{quantile="0.25"} 3.898e-05
go_gc_duration_seconds{quantile="0.5"} 4.874e-05
go_gc_duration_seconds{quantile="0.75"} 8.407e-05
go_gc_duration_seconds{quantile="1"} 0.005923069
go_gc_duration_seconds_sum 0.212160145
go_gc_duration_seconds_count 2997
# HELP go_goroutines Number of goroutines that currently exist.
# TYPE go_goroutines gauge
go_goroutines 45
# HELP go_info Information about the Go environment.
# TYPE go_info gauge
go_info{version="go1.21.0"} 1
# HELP go_memstats_alloc_bytes Number of bytes allocated and still in use.
# TYPE go_memstats_alloc_bytes gauge
go_memstats_alloc_bytes 7.187576e+06
# HELP go_memstats_alloc_bytes_total Total number of bytes allocated, even if freed.
# TYPE go_memstats_alloc_bytes_total counter
go_memstats_alloc_bytes_total 9.902860224e+09
# HELP go_memstats_buck_hash_sys_bytes Number of bytes used by the profiling bucket hash table.
# TYPE go_memstats_buck_hash_sys_bytes gauge
go_memstats_buck_hash_sys_bytes 1.704292e+06
# HELP go_memstats_frees_total Total number of frees.
# TYPE go_memstats_frees_total counter
go_memstats_frees_total 1.10731608e+08
# HELP go_memstats_gc_sys_bytes Number of bytes used for garbage collection system metadata.
# TYPE go_memstats_gc_sys_bytes gauge
go_memstats_gc_sys_bytes 4.744552e+06
# HELP go_memstats_heap_alloc_bytes Number of heap bytes allocated and still in use.
# TYPE go_memstats_heap_alloc_bytes gauge
go_memstats_heap_alloc_bytes 7.187576e+06
# HELP go_memstats_heap_idle_bytes Number of heap bytes waiting to be used.
# TYPE go_memstats_heap_idle_bytes gauge
go_memstats_heap_idle_bytes 9.035776e+06
# HELP go_memstats_heap_inuse_bytes Number of heap bytes that are in use.
# TYPE go_memstats_heap_inuse_bytes gauge
go_memstats_heap_inuse_bytes 1.0985472e+07
# HELP go_memstats_heap_objects Number of allocated objects.
# TYPE go_memstats_heap_objects gauge
go_memstats_heap_objects 23179
# HELP go_memstats_heap_released_bytes Number of heap bytes released to OS.
# TYPE go_memstats_heap_released_bytes gauge
go_memstats_heap_released_bytes 5.029888e+06
# HELP go_memstats_heap_sys_bytes Number of heap bytes obtained from system.
# TYPE go_memstats_heap_sys_bytes gauge
go_memstats_heap_sys_bytes 2.0021248e+07
# HELP go_memstats_last_gc_time_seconds Number of seconds since 1970 of last garbage collection.
# TYPE go_memstats_last_gc_time_seconds gauge
go_memstats_last_gc_time_seconds 1.69359500169754e+09
# HELP go_memstats_lookups_total Total number of pointer lookups.
# TYPE go_memstats_lookups_total counter
go_memstats_lookups_total 0
# HELP go_memstats_mallocs_total Total number of mallocs.
# TYPE go_memstats_mallocs_total counter
go_memstats_mallocs_total 1.10754787e+08
# HELP go_memstats_mcache_inuse_bytes Number of bytes in use by mcache structures.
# TYPE go_memstats_mcache_inuse_bytes gauge
go_memstats_mcache_inuse_bytes 4800
# HELP go_memstats_mcache_sys_bytes Number of bytes used for mcache structures obtained from system.
# TYPE go_memstats_mcache_sys_bytes gauge
go_memstats_mcache_sys_bytes 15600
# HELP go_memstats_mspan_inuse_bytes Number of bytes in use by mspan structures.
# TYPE go_memstats_mspan_inuse_bytes gauge
go_memstats_mspan_inuse_bytes 177240
# HELP go_memstats_mspan_sys_bytes Number of bytes used for mspan structures obtained from system.
# TYPE go_memstats_mspan_sys_bytes gauge
go_memstats_mspan_sys_bytes 228144
# HELP go_memstats_next_gc_bytes Number of heap bytes when next garbage collection will take place.
# TYPE go_memstats_next_gc_bytes gauge
go_memstats_next_gc_bytes 1.2783728e+07
# HELP go_memstats_other_sys_bytes Number of bytes used for other system allocations.
# TYPE go_memstats_other_sys_bytes gauge
go_memstats_other_sys_bytes 751132
# HELP go_memstats_stack_inuse_bytes Number of bytes in use by the stack allocator.
# TYPE go_memstats_stack_inuse_bytes gauge
go_memstats_stack_inuse_bytes 950272
# HELP go_memstats_stack_sys_bytes Number of bytes obtained from system for stack allocator.
# TYPE go_memstats_stack_sys_bytes gauge
go_memstats_stack_sys_bytes 950272
# HELP go_memstats_sys_bytes Number of bytes obtained from system.
# TYPE go_memstats_sys_bytes gauge
go_memstats_sys_bytes 2.841524e+07
# HELP go_threads Number of OS threads created.
# TYPE go_threads gauge
go_threads 10
# HELP leader_election_master_status Gauge of if the reporting system is master of the relevant lease, 0 indicates backup, 1 indicates master. 'name' is the string used to identify the lease. Please make sure to group by name.
# TYPE leader_election_master_status gauge
leader_election_master_status{name="9fe89c94.controller.autoneg.dev"} 1
# HELP process_cpu_seconds_total Total user and system CPU time spent in seconds.
# TYPE process_cpu_seconds_total counter
process_cpu_seconds_total 364.13
# HELP process_max_fds Maximum number of open file descriptors.
# TYPE process_max_fds gauge
process_max_fds 1.048576e+06
# HELP process_open_fds Number of open file descriptors.
# TYPE process_open_fds gauge
process_open_fds 15
# HELP process_resident_memory_bytes Resident memory size in bytes.
# TYPE process_resident_memory_bytes gauge
process_resident_memory_bytes 4.919296e+07
# HELP process_start_time_seconds Start time of the process since unix epoch in seconds.
# TYPE process_start_time_seconds gauge
process_start_time_seconds 1.69323937661e+09
# HELP process_virtual_memory_bytes Virtual memory size in bytes.
# TYPE process_virtual_memory_bytes gauge
process_virtual_memory_bytes 1.305399296e+09
# HELP process_virtual_memory_max_bytes Maximum amount of virtual memory available in bytes.
# TYPE process_virtual_memory_max_bytes gauge
process_virtual_memory_max_bytes 1.8446744073709552e+19
# HELP rest_client_requests_total Number of HTTP requests, partitioned by status code, method, and host.
# TYPE rest_client_requests_total counter
rest_client_requests_total{code="200",host="10.27.128.1:443",method="GET"} 177747
rest_client_requests_total{code="200",host="10.27.128.1:443",method="PUT"} 176954
rest_client_requests_total{code="201",host="10.27.128.1:443",method="POST"} 2
# HELP workqueue_adds_total Total number of adds handled by workqueue
# TYPE workqueue_adds_total counter
workqueue_adds_total{name="service"} 848
# HELP workqueue_depth Current depth of workqueue
# TYPE workqueue_depth gauge
workqueue_depth{name="service"} 0
# HELP workqueue_longest_running_processor_seconds How many seconds has the longest running processor for workqueue been running.
# TYPE workqueue_longest_running_processor_seconds gauge
workqueue_longest_running_processor_seconds{name="service"} 18.763401986
# HELP workqueue_queue_duration_seconds How long in seconds an item stays in workqueue before being requested
# TYPE workqueue_queue_duration_seconds histogram
workqueue_queue_duration_seconds_bucket{name="service",le="1e-08"} 0
workqueue_queue_duration_seconds_bucket{name="service",le="1e-07"} 0
workqueue_queue_duration_seconds_bucket{name="service",le="1e-06"} 0
workqueue_queue_duration_seconds_bucket{name="service",le="9.999999999999999e-06"} 7
workqueue_queue_duration_seconds_bucket{name="service",le="9.999999999999999e-05"} 11
workqueue_queue_duration_seconds_bucket{name="service",le="0.001"} 244
workqueue_queue_duration_seconds_bucket{name="service",le="0.01"} 848
workqueue_queue_duration_seconds_bucket{name="service",le="0.1"} 848
workqueue_queue_duration_seconds_bucket{name="service",le="1"} 848
workqueue_queue_duration_seconds_bucket{name="service",le="10"} 848
workqueue_queue_duration_seconds_bucket{name="service",le="+Inf"} 848
workqueue_queue_duration_seconds_sum{name="service"} 1.472508323
workqueue_queue_duration_seconds_count{name="service"} 848
# HELP workqueue_retries_total Total number of retries handled by workqueue
# TYPE workqueue_retries_total counter
workqueue_retries_total{name="service"} 1
# HELP workqueue_unfinished_work_seconds How many seconds of work has been done that is in progress and hasn't been observed by work_duration. Large values indicate stuck threads. One can deduce the number of stuck threads by observing the rate at which this increases.
# TYPE workqueue_unfinished_work_seconds gauge
workqueue_unfinished_work_seconds{name="service"} 18.763401986
# HELP workqueue_work_duration_seconds How long in seconds processing an item from workqueue takes.
# TYPE workqueue_work_duration_seconds histogram
workqueue_work_duration_seconds_bucket{name="service",le="1e-08"} 0
workqueue_work_duration_seconds_bucket{name="service",le="1e-07"} 0
workqueue_work_duration_seconds_bucket{name="service",le="1e-06"} 0
workqueue_work_duration_seconds_bucket{name="service",le="9.999999999999999e-06"} 5
workqueue_work_duration_seconds_bucket{name="service",le="9.999999999999999e-05"} 810
workqueue_work_duration_seconds_bucket{name="service",le="0.001"} 844
workqueue_work_duration_seconds_bucket{name="service",le="0.01"} 846
workqueue_work_duration_seconds_bucket{name="service",le="0.1"} 846
workqueue_work_duration_seconds_bucket{name="service",le="1"} 846
workqueue_work_duration_seconds_bucket{name="service",le="10"} 846
workqueue_work_duration_seconds_bucket{name="service",le="+Inf"} 847
workqueue_work_duration_seconds_sum{name="service"} 18.151012388
workqueue_work_duration_seconds_count{name="service"} 847