bug: lua_shared_dict prometheus-metrics overflow

[DrJSAnD]

Current Behavior

We deploy apisix in K8s cluster and have problem with prometheus metrics.
We noticed that lua_shared_dict prometheus-metrics overflows, then the number of apisix_nginx_metric_errors_total errors starts to grow and all metrics stop displaying correctly.

We try increase the prometheus-metrics parameter to 40m in the ConfigMap (config.yaml), but after 2 months this lua_shared_dict was full on all pods and errors started to occur again.

nginx_config:    # config for render the template to genarate nginx.conf
  error_log: "/dev/stderr"
  error_log_level: "warn"    # warn,error
  worker_processes: "auto"
  enable_cpu_affinity: true
  worker_rlimit_nofile: 20480  # the number of files a worker process can open, should be larger than worker_connections
  event:
    worker_connections: 10620
  http:
    enable_access_log: true
    access_log: "/dev/stdout"
    access_log_format: '$remote_addr - $remote_user [$time_local] $http_host \"$request\" $status $body_bytes_sent $request_time \"$http_referer\" \"$http_user_agent\" $upstream_addr $upstream_status $upstream_response_time \"$upstream_scheme://$upstream_host$upstream_uri\"'
    access_log_format_escape: default
    keepalive_timeout: "60s"
    client_header_timeout: 60s     # timeout for reading client request header, then 408 (Request Time-out) error is returned to the client
    client_body_timeout: 60s       # timeout for reading client request body, then 408 (Request Time-out) error is returned to the client
    send_timeout: 10s              # timeout for transmitting a response to the client.then the connection is closed
    underscores_in_headers: "on"   # default enables the use of underscores in client request header fields
    real_ip_header: "X-Real-IP"    # http://nginx.org/en/docs/http/ngx_http_realip_module.html#real_ip_header
    real_ip_from:                  # http://nginx.org/en/docs/http/ngx_http_realip_module.html#set_real_ip_from
      - 127.0.0.1
      - 'unix:'
    lua_shared_dict:
      prometheus-metrics: 40m

Current Apisix state

• Deployment via Helm chart: https://github.com/apache/apisix-helm-chart
• Helm Chart version: 2.10.0
• K8s pods: 3
• Pod CPU limits: 15 (usage 4%)
• Pod Memory limits: 60Gb (usage 35 GiB)
• Total requests per second: 2500 - 3000
• Active connections: 2000+
• Upstreams: 100+
• Routes: 120+
• Consumers: 60+
• Plugins: basic-auth and kafka-logger on all routes

Expected Behavior

No response

Error Logs

No response

Steps to Reproduce

1. Run apisix with default lua_shared_dict: prometheus-metrics
2. After 2-3 weeks prometheus-metrics overflows and apisix_nginx_metric_errors_total errors starts to grow and all metrics stop displaying correctly
3. Change lua_shared_dict: prometheus-metrics to 40m
4. After 2-3 months lua_shared_dict overflows again and we get a similar problem with displaying metrics

Environment

• APISIX version (run apisix version): 3.10.0
• Operating system (run uname -a): Linux apisix-69cfdc5fbf-m7k27 5.14.0-362.13.1.el9_3.x86_64 SMP PREEMPT_DYNAMIC Fri Nov 24 01:57:57 EST 2023 x86_64 GNU/Linux
• OpenResty / Nginx version (run openresty -V or nginx -V): openresty/1.25.3.2
• etcd version, if relevant (run curl http://127.0.0.1:9090/v1/server_info): 3.5.0
• APISIX Dashboard version, if relevant: 3.0.0
• Plugin runner version, for issues related to plugin runners:
• LuaRocks version, for installation issues (run luarocks --version):

[qizhendong1]

You can use curl 127.0.0.1:9091/apisix/prometheus/metrics to test the prometheus plugin endpoint if it works properly

[DrJSAnD]

Prometheus plugin works on all pods and it returns 69000+ rows from each pod.
We display metrics from Prometheus in Grafana (https://github.com/apache/apisix/blob/master/docs/assets/other/json/apisix-grafana-dashboard.json).
When apisix_shared_dict_free_space_bytes{name="prometheus-metrics"} reached value "0" then apisix_nginx_metric_errors_total start grows up and all apisix metrics show incorrect values.

[shihuncl]

K8S部署的apisix怎么修改lua_shared_dict的大小，试了很多地方都是修改不成功

[DrJSAnD]

K8S部署的apisix怎么修改lua_shared_dict的大小，试了很多地方都是修改不成功

In ConfigMap - config.yaml

nginx_config:
    http:
        lua_shared_dict:
            prometheus-metrics: 40m

[bzp2010]

Generally this has to do with how discrete the requests are on your API service, which is related to multiple metrics such as route_id, resp_code, or even consumer_id, etc. (https://apisix.apache.org/docs/apisix/next/plugins/prometheus/#labels) Multiply them together and that's the number of record entries you'll get, which in the prometheus or observability is called a high cardinality.
If they're really large, you'll need to increase the metrics size.

[DrJSAnD]

Generally this has to do with how discrete the requests are on your API service, which is related to multiple metrics such as route_id, resp_code, or even consumer_id, etc. (https://apisix.apache.org/docs/apisix/next/plugins/prometheus/#labels) Multiply them together and that's the number of record entries you'll get, which in the prometheus or observability is called a high cardinality. If they're really large, you'll need to increase the metrics size.

Will it be okay if we increase the lua_shared_dict - prometheus-metrics parameter value to 512m or 1024m?