"error encoding and sending metric family" on HTTP/2 HEAD request

[es-korneev]

Hi!

I have encountered probably undesirable behaviour of node_exporter when running HTTP/2. node_exporter complained about HTTP/2 write errors when HEAD request method was used. Steps to reproduce:

1. Run node_exporter with TLS:

$ ./node_exporter --web.listen-address=:9101 --web.config.file=node_exp.yml --log.level=error

# node_exp.yml
---
tls_server_config:
  cert_file: node_exp.crt
  key_file: node_exp.key

2. Send a request with curl:

$ curl https://localhost:9101 -Ik

curl will display 200 OK and exit cleanly, though node_exporter will write 299 lines like this:

ts=2024-03-23T18:08:58.844Z caller=stdlib.go:105 level=error msg="error encoding and sending metric family: http2: request body closed due to handler exiting"

3. The error will show up in promhttp_metric_handler_errors_total metric:

$ curl https://localhost:9101/metrics -sk | grep encoding
promhttp_metric_handler_errors_total{cause="encoding"} 299

I have tracked errors down to prometheus/[email protected]/expfmt/text_create.go:96, http2 connection complained about invalid writes len(metric_families) times. HTTP/1.1 deals OK with such requests.

Not sure whether it is a problem related to HTTP/2 implementation in Go or error handling issue in promhttp, but I feel like some sort of error handling can be added to the library.

Thank you for any comments!

Some info about environment:

Host operating system: output of uname -a

Linux host 6.2.0-35-generic #35~22.04.1-Ubuntu SMP PREEMPT_DYNAMIC Fri Oct  6 10:23:26 UTC 2 x86_64 x86_64 x86_64 GNU/Linux

node_exporter version: output of node_exporter --version

node_exporter, version 1.7.0 (branch: master, revision: 6425f079d162ebd22d4c6c4e4d7e4a36ebbe2239)
  build user:       mint@mint-Vostro-14-5410
  build date:       20240323-17:52:29
  go version:       go1.22.1
  platform:         linux/amd64
  tags:             netgo osusergo static_build

node_exporter command line flags

./node_exporter --web.listen-address=:9101 --web.config.file=node_exp.yml --log.level=error

# node_exp.yml
---
tls_server_config:
  cert_file: node_exp.crt
  key_file: node_exp.key

node_exporter log output

# repeated 299 times
ts=2024-03-23T18:08:58.844Z caller=stdlib.go:105 level=error msg="error encoding and sending metric family: http2: request body closed due to handler exiting"

Are you running node_exporter in Docker?

No.

What did you do that produced an error?

curl https://localhost:9101 -Ik

curl --version

curl 7.81.0 (x86_64-pc-linux-gnu) libcurl/7.81.0 OpenSSL/3.0.2 zlib/1.2.11 brotli/1.0.9 zstd/1.4.8 libidn2/2.3.2 libpsl/0.21.0 (+libidn2/2.3.2) libssh/0.9.6/openssl/zlib nghttp2/1.43.0 librtmp/2.3 OpenLDAP/2.5.16
Release-Date: 2022-01-05
Protocols: dict file ftp ftps gopher gophers http https imap imaps ldap ldaps mqtt pop3 pop3s rtmp rtsp scp sftp smb smbs smtp smtps telnet tftp 
Features: alt-svc AsynchDNS brotli GSS-API HSTS HTTP2 HTTPS-proxy IDN IPv6 Kerberos Largefile libz NTLM NTLM_WB PSL SPNEGO SSL TLS-SRP UnixSockets zstd