(Metrics) Is this expected that Akka HTTP instrumentation generates histogram with a route label? (high cardinality)

[gaeljw]

Hello folks,

I noticed that Akka HTTP auto instrumentation generates a very high number of metrics (or I should say one metric with high cardinality labels): http_server_duration_milliseconds_bucket.

I may be wrong (still relatively new to the topic) but isn't it against metrics best practices?

For instance, with only 5 routes called I get this bunch of metrics (Prometheus format):

http_server_duration_milliseconds_count{otel_scope_name="io.opentelemetry.akka-http-10.0",otel_scope_version="1.29.0-alpha",http_method="GET",http_route="/docs/swagger.json",http_scheme="http",http_status_code="200",net_host_name="mysrv.mycompany.net",net_host_port="9001",net_protocol_name="http",net_protocol_version="1.1"} 1.0 1701100607547
http_server_duration_milliseconds_sum{otel_scope_name="io.opentelemetry.akka-http-10.0",otel_scope_version="1.29.0-alpha",http_method="GET",http_route="/docs/swagger.json",http_scheme="http",http_status_code="200",net_host_name="mysrv.mycompany.net",net_host_port="9001",net_protocol_name="http",net_protocol_version="1.1"} 50.605375 1701100607547
http_server_duration_milliseconds_bucket{otel_scope_name="io.opentelemetry.akka-http-10.0",otel_scope_version="1.29.0-alpha",http_method="GET",http_route="/docs/swagger.json",http_scheme="http",http_status_code="200",net_host_name="mysrv.mycompany.net",net_host_port="9001",net_protocol_name="http",net_protocol_version="1.1",le="0.0"} 0.0 1701100607547
http_server_duration_milliseconds_bucket{otel_scope_name="io.opentelemetry.akka-http-10.0",otel_scope_version="1.29.0-alpha",http_method="GET",http_route="/docs/swagger.json",http_scheme="http",http_status_code="200",net_host_name="mysrv.mycompany.net",net_host_port="9001",net_protocol_name="http",net_protocol_version="1.1",le="5.0"} 0.0 1701100607547
http_server_duration_milliseconds_bucket{otel_scope_name="io.opentelemetry.akka-http-10.0",otel_scope_version="1.29.0-alpha",http_method="GET",http_route="/docs/swagger.json",http_scheme="http",http_status_code="200",net_host_name="mysrv.mycompany.net",net_host_port="9001",net_protocol_name="http",net_protocol_version="1.1",le="10.0"} 0.0 1701100607547
http_server_duration_milliseconds_bucket{otel_scope_name="io.opentelemetry.akka-http-10.0",otel_scope_version="1.29.0-alpha",http_method="GET",http_route="/docs/swagger.json",http_scheme="http",http_status_code="200",net_host_name="mysrv.mycompany.net",net_host_port="9001",net_protocol_name="http",net_protocol_version="1.1",le="25.0"} 0.0 1701100607547
http_server_duration_milliseconds_bucket{otel_scope_name="io.opentelemetry.akka-http-10.0",otel_scope_version="1.29.0-alpha",http_method="GET",http_route="/docs/swagger.json",http_scheme="http",http_status_code="200",net_host_name="mysrv.mycompany.net",net_host_port="9001",net_protocol_name="http",net_protocol_version="1.1",le="50.0"} 0.0 1701100607547
http_server_duration_milliseconds_bucket{otel_scope_name="io.opentelemetry.akka-http-10.0",otel_scope_version="1.29.0-alpha",http_method="GET",http_route="/docs/swagger.json",http_scheme="http",http_status_code="200",net_host_name="mysrv.mycompany.net",net_host_port="9001",net_protocol_name="http",net_protocol_version="1.1",le="75.0"} 1.0 1701100607547
http_server_duration_milliseconds_bucket{otel_scope_name="io.opentelemetry.akka-http-10.0",otel_scope_version="1.29.0-alpha",http_method="GET",http_route="/docs/swagger.json",http_scheme="http",http_status_code="200",net_host_name="mysrv.mycompany.net",net_host_port="9001",net_protocol_name="http",net_protocol_version="1.1",le="100.0"} 1.0 1701100607547
http_server_duration_milliseconds_bucket{otel_scope_name="io.opentelemetry.akka-http-10.0",otel_scope_version="1.29.0-alpha",http_method="GET",http_route="/docs/swagger.json",http_scheme="http",http_status_code="200",net_host_name="mysrv.mycompany.net",net_host_port="9001",net_protocol_name="http",net_protocol_version="1.1",le="250.0"} 1.0 1701100607547
http_server_duration_milliseconds_bucket{otel_scope_name="io.opentelemetry.akka-http-10.0",otel_scope_version="1.29.0-alpha",http_method="GET",http_route="/docs/swagger.json",http_scheme="http",http_status_code="200",net_host_name="mysrv.mycompany.net",net_host_port="9001",net_protocol_name="http",net_protocol_version="1.1",le="500.0"} 1.0 1701100607547
http_server_duration_milliseconds_bucket{otel_scope_name="io.opentelemetry.akka-http-10.0",otel_scope_version="1.29.0-alpha",http_method="GET",http_route="/docs/swagger.json",http_scheme="http",http_status_code="200",net_host_name="mysrv.mycompany.net",net_host_port="9001",net_protocol_name="http",net_protocol_version="1.1",le="750.0"} 1.0 1701100607547
http_server_duration_milliseconds_bucket{otel_scope_name="io.opentelemetry.akka-http-10.0",otel_scope_version="1.29.0-alpha",http_method="GET",http_route="/docs/swagger.json",http_scheme="http",http_status_code="200",net_host_name="mysrv.mycompany.net",net_host_port="9001",net_protocol_name="http",net_protocol_version="1.1",le="1000.0"} 1.0 1701100607547
http_server_duration_milliseconds_bucket{otel_scope_name="io.opentelemetry.akka-http-10.0",otel_scope_version="1.29.0-alpha",http_method="GET",http_route="/docs/swagger.json",http_scheme="http",http_status_code="200",net_host_name="mysrv.mycompany.net",net_host_port="9001",net_protocol_name="http",net_protocol_version="1.1",le="2500.0"} 1.0 1701100607547
http_server_duration_milliseconds_bucket{otel_scope_name="io.opentelemetry.akka-http-10.0",otel_scope_version="1.29.0-alpha",http_method="GET",http_route="/docs/swagger.json",http_scheme="http",http_status_code="200",net_host_name="mysrv.mycompany.net",net_host_port="9001",net_protocol_name="http",net_protocol_version="1.1",le="5000.0"} 1.0 1701100607547
http_server_duration_milliseconds_bucket{otel_scope_name="io.opentelemetry.akka-http-10.0",otel_scope_version="1.29.0-alpha",http_method="GET",http_route="/docs/swagger.json",http_scheme="http",http_status_code="200",net_host_name="mysrv.mycompany.net",net_host_port="9001",net_protocol_name="http",net_protocol_version="1.1",le="7500.0"} 1.0 1701100607547
http_server_duration_milliseconds_bucket{otel_scope_name="io.opentelemetry.akka-http-10.0",otel_scope_version="1.29.0-alpha",http_method="GET",http_route="/docs/swagger.json",http_scheme="http",http_status_code="200",net_host_name="mysrv.mycompany.net",net_host_port="9001",net_protocol_name="http",net_protocol_version="1.1",le="10000.0"} 1.0 1701100607547
http_server_duration_milliseconds_bucket{otel_scope_name="io.opentelemetry.akka-http-10.0",otel_scope_version="1.29.0-alpha",http_method="GET",http_route="/docs/swagger.json",http_scheme="http",http_status_code="200",net_host_name="mysrv.mycompany.net",net_host_port="9001",net_protocol_name="http",net_protocol_version="1.1",le="+Inf"} 1.0 1701100607547
http_server_duration_milliseconds_count{otel_scope_name="io.opentelemetry.akka-http-10.0",otel_scope_version="1.29.0-alpha",http_method="GET",http_route="/docs/",http_scheme="http",http_status_code="200",net_host_name="mysrv.mycompany.net",net_host_port="9001",net_protocol_name="http",net_protocol_version="1.1"} 1.0 1701100607547
http_server_duration_milliseconds_sum{otel_scope_name="io.opentelemetry.akka-http-10.0",otel_scope_version="1.29.0-alpha",http_method="GET",http_route="/docs/",http_scheme="http",http_status_code="200",net_host_name="mysrv.mycompany.net",net_host_port="9001",net_protocol_name="http",net_protocol_version="1.1"} 284.781024 1701100607547
http_server_duration_milliseconds_bucket{otel_scope_name="io.opentelemetry.akka-http-10.0",otel_scope_version="1.29.0-alpha",http_method="GET",http_route="/docs/",http_scheme="http",http_status_code="200",net_host_name="mysrv.mycompany.net",net_host_port="9001",net_protocol_name="http",net_protocol_version="1.1",le="0.0"} 0.0 1701100607547
http_server_duration_milliseconds_bucket{otel_scope_name="io.opentelemetry.akka-http-10.0",otel_scope_version="1.29.0-alpha",http_method="GET",http_route="/docs/",http_scheme="http",http_status_code="200",net_host_name="mysrv.mycompany.net",net_host_port="9001",net_protocol_name="http",net_protocol_version="1.1",le="5.0"} 0.0 1701100607547
http_server_duration_milliseconds_bucket{otel_scope_name="io.opentelemetry.akka-http-10.0",otel_scope_version="1.29.0-alpha",http_method="GET",http_route="/docs/",http_scheme="http",http_status_code="200",net_host_name="mysrv.mycompany.net",net_host_port="9001",net_protocol_name="http",net_protocol_version="1.1",le="10.0"} 0.0 1701100607547
http_server_duration_milliseconds_bucket{otel_scope_name="io.opentelemetry.akka-http-10.0",otel_scope_version="1.29.0-alpha",http_method="GET",http_route="/docs/",http_scheme="http",http_status_code="200",net_host_name="mysrv.mycompany.net",net_host_port="9001",net_protocol_name="http",net_protocol_version="1.1",le="25.0"} 0.0 1701100607547
http_server_duration_milliseconds_bucket{otel_scope_name="io.opentelemetry.akka-http-10.0",otel_scope_version="1.29.0-alpha",http_method="GET",http_route="/docs/",http_scheme="http",http_status_code="200",net_host_name="mysrv.mycompany.net",net_host_port="9001",net_protocol_name="http",net_protocol_version="1.1",le="50.0"} 0.0 1701100607547
http_server_duration_milliseconds_bucket{otel_scope_name="io.opentelemetry.akka-http-10.0",otel_scope_version="1.29.0-alpha",http_method="GET",http_route="/docs/",http_scheme="http",http_status_code="200",net_host_name="mysrv.mycompany.net",net_host_port="9001",net_protocol_name="http",net_protocol_version="1.1",le="75.0"} 0.0 1701100607547
http_server_duration_milliseconds_bucket{otel_scope_name="io.opentelemetry.akka-http-10.0",otel_scope_version="1.29.0-alpha",http_method="GET",http_route="/docs/",http_scheme="http",http_status_code="200",net_host_name="mysrv.mycompany.net",net_host_port="9001",net_protocol_name="http",net_protocol_version="1.1",le="100.0"} 0.0 1701100607547
http_server_duration_milliseconds_bucket{otel_scope_name="io.opentelemetry.akka-http-10.0",otel_scope_version="1.29.0-alpha",http_method="GET",http_route="/docs/",http_scheme="http",http_status_code="200",net_host_name="mysrv.mycompany.net",net_host_port="9001",net_protocol_name="http",net_protocol_version="1.1",le="250.0"} 0.0 1701100607547
http_server_duration_milliseconds_bucket{otel_scope_name="io.opentelemetry.akka-http-10.0",otel_scope_version="1.29.0-alpha",http_method="GET",http_route="/docs/",http_scheme="http",http_status_code="200",net_host_name="mysrv.mycompany.net",net_host_port="9001",net_protocol_name="http",net_protocol_version="1.1",le="500.0"} 1.0 1701100607547
http_server_duration_milliseconds_bucket{otel_scope_name="io.opentelemetry.akka-http-10.0",otel_scope_version="1.29.0-alpha",http_method="GET",http_route="/docs/",http_scheme="http",http_status_code="200",net_host_name="mysrv.mycompany.net",net_host_port="9001",net_protocol_name="http",net_protocol_version="1.1",le="750.0"} 1.0 1701100607547
http_server_duration_milliseconds_bucket{otel_scope_name="io.opentelemetry.akka-http-10.0",otel_scope_version="1.29.0-alpha",http_method="GET",http_route="/docs/",http_scheme="http",http_status_code="200",net_host_name="mysrv.mycompany.net",net_host_port="9001",net_protocol_name="http",net_protocol_version="1.1",le="1000.0"} 1.0 1701100607547
http_server_duration_milliseconds_bucket{otel_scope_name="io.opentelemetry.akka-http-10.0",otel_scope_version="1.29.0-alpha",http_method="GET",http_route="/docs/",http_scheme="http",http_status_code="200",net_host_name="mysrv.mycompany.net",net_host_port="9001",net_protocol_name="http",net_protocol_version="1.1",le="2500.0"} 1.0 1701100607547
http_server_duration_milliseconds_bucket{otel_scope_name="io.opentelemetry.akka-http-10.0",otel_scope_version="1.29.0-alpha",http_method="GET",http_route="/docs/",http_scheme="http",http_status_code="200",net_host_name="mysrv.mycompany.net",net_host_port="9001",net_protocol_name="http",net_protocol_version="1.1",le="5000.0"} 1.0 1701100607547
http_server_duration_milliseconds_bucket{otel_scope_name="io.opentelemetry.akka-http-10.0",otel_scope_version="1.29.0-alpha",http_method="GET",http_route="/docs/",http_scheme="http",http_status_code="200",net_host_name="mysrv.mycompany.net",net_host_port="9001",net_protocol_name="http",net_protocol_version="1.1",le="7500.0"} 1.0 1701100607547
http_server_duration_milliseconds_bucket{otel_scope_name="io.opentelemetry.akka-http-10.0",otel_scope_version="1.29.0-alpha",http_method="GET",http_route="/docs/",http_scheme="http",http_status_code="200",net_host_name="mysrv.mycompany.net",net_host_port="9001",net_protocol_name="http",net_protocol_version="1.1",le="10000.0"} 1.0 1701100607547
http_server_duration_milliseconds_bucket{otel_scope_name="io.opentelemetry.akka-http-10.0",otel_scope_version="1.29.0-alpha",http_method="GET",http_route="/docs/",http_scheme="http",http_status_code="200",net_host_name="mysrv.mycompany.net",net_host_port="9001",net_protocol_name="http",net_protocol_version="1.1",le="+Inf"} 1.0 1701100607547
http_server_duration_milliseconds_count{otel_scope_name="io.opentelemetry.akka-http-10.0",otel_scope_version="1.29.0-alpha",http_method="GET",http_route="/docs/$file<.+>",http_scheme="http",http_status_code="200",net_host_name="mysrv.mycompany.net",net_host_port="9001",net_protocol_name="http",net_protocol_version="1.1"} 4.0 1701100607547
http_server_duration_milliseconds_sum{otel_scope_name="io.opentelemetry.akka-http-10.0",otel_scope_version="1.29.0-alpha",http_method="GET",http_route="/docs/$file<.+>",http_scheme="http",http_status_code="200",net_host_name="mysrv.mycompany.net",net_host_port="9001",net_protocol_name="http",net_protocol_version="1.1"} 257.491083 1701100607547
http_server_duration_milliseconds_bucket{otel_scope_name="io.opentelemetry.akka-http-10.0",otel_scope_version="1.29.0-alpha",http_method="GET",http_route="/docs/$file<.+>",http_scheme="http",http_status_code="200",net_host_name="mysrv.mycompany.net",net_host_port="9001",net_protocol_name="http",net_protocol_version="1.1",le="0.0"} 0.0 1701100607547
http_server_duration_milliseconds_bucket{otel_scope_name="io.opentelemetry.akka-http-10.0",otel_scope_version="1.29.0-alpha",http_method="GET",http_route="/docs/$file<.+>",http_scheme="http",http_status_code="200",net_host_name="mysrv.mycompany.net",net_host_port="9001",net_protocol_name="http",net_protocol_version="1.1",le="5.0"} 0.0 1701100607547
http_server_duration_milliseconds_bucket{otel_scope_name="io.opentelemetry.akka-http-10.0",otel_scope_version="1.29.0-alpha",http_method="GET",http_route="/docs/$file<.+>",http_scheme="http",http_status_code="200",net_host_name="mysrv.mycompany.net",net_host_port="9001",net_protocol_name="http",net_protocol_version="1.1",le="10.0"} 0.0 1701100607547
http_server_duration_milliseconds_bucket{otel_scope_name="io.opentelemetry.akka-http-10.0",otel_scope_version="1.29.0-alpha",http_method="GET",http_route="/docs/$file<.+>",http_scheme="http",http_status_code="200",net_host_name="mysrv.mycompany.net",net_host_port="9001",net_protocol_name="http",net_protocol_version="1.1",le="25.0"} 0.0 1701100607547
http_server_duration_milliseconds_bucket{otel_scope_name="io.opentelemetry.akka-http-10.0",otel_scope_version="1.29.0-alpha",http_method="GET",http_route="/docs/$file<.+>",http_scheme="http",http_status_code="200",net_host_name="mysrv.mycompany.net",net_host_port="9001",net_protocol_name="http",net_protocol_version="1.1",le="50.0"} 1.0 1701100607547
http_server_duration_milliseconds_bucket{otel_scope_name="io.opentelemetry.akka-http-10.0",otel_scope_version="1.29.0-alpha",http_method="GET",http_route="/docs/$file<.+>",http_scheme="http",http_status_code="200",net_host_name="mysrv.mycompany.net",net_host_port="9001",net_protocol_name="http",net_protocol_version="1.1",le="75.0"} 2.0 1701100607547
http_server_duration_milliseconds_bucket{otel_scope_name="io.opentelemetry.akka-http-10.0",otel_scope_version="1.29.0-alpha",http_method="GET",http_route="/docs/$file<.+>",http_scheme="http",http_status_code="200",net_host_name="mysrv.mycompany.net",net_host_port="9001",net_protocol_name="http",net_protocol_version="1.1",le="100.0"} 4.0 1701100607547
http_server_duration_milliseconds_bucket{otel_scope_name="io.opentelemetry.akka-http-10.0",otel_scope_version="1.29.0-alpha",http_method="GET",http_route="/docs/$file<.+>",http_scheme="http",http_status_code="200",net_host_name="mysrv.mycompany.net",net_host_port="9001",net_protocol_name="http",net_protocol_version="1.1",le="250.0"} 4.0 1701100607547
http_server_duration_milliseconds_bucket{otel_scope_name="io.opentelemetry.akka-http-10.0",otel_scope_version="1.29.0-alpha",http_method="GET",http_route="/docs/$file<.+>",http_scheme="http",http_status_code="200",net_host_name="mysrv.mycompany.net",net_host_port="9001",net_protocol_name="http",net_protocol_version="1.1",le="500.0"} 4.0 1701100607547
http_server_duration_milliseconds_bucket{otel_scope_name="io.opentelemetry.akka-http-10.0",otel_scope_version="1.29.0-alpha",http_method="GET",http_route="/docs/$file<.+>",http_scheme="http",http_status_code="200",net_host_name="mysrv.mycompany.net",net_host_port="9001",net_protocol_name="http",net_protocol_version="1.1",le="750.0"} 4.0 1701100607547
http_server_duration_milliseconds_bucket{otel_scope_name="io.opentelemetry.akka-http-10.0",otel_scope_version="1.29.0-alpha",http_method="GET",http_route="/docs/$file<.+>",http_scheme="http",http_status_code="200",net_host_name="mysrv.mycompany.net",net_host_port="9001",net_protocol_name="http",net_protocol_version="1.1",le="1000.0"} 4.0 1701100607547
http_server_duration_milliseconds_bucket{otel_scope_name="io.opentelemetry.akka-http-10.0",otel_scope_version="1.29.0-alpha",http_method="GET",http_route="/docs/$file<.+>",http_scheme="http",http_status_code="200",net_host_name="mysrv.mycompany.net",net_host_port="9001",net_protocol_name="http",net_protocol_version="1.1",le="2500.0"} 4.0 1701100607547
http_server_duration_milliseconds_bucket{otel_scope_name="io.opentelemetry.akka-http-10.0",otel_scope_version="1.29.0-alpha",http_method="GET",http_route="/docs/$file<.+>",http_scheme="http",http_status_code="200",net_host_name="mysrv.mycompany.net",net_host_port="9001",net_protocol_name="http",net_protocol_version="1.1",le="5000.0"} 4.0 1701100607547
http_server_duration_milliseconds_bucket{otel_scope_name="io.opentelemetry.akka-http-10.0",otel_scope_version="1.29.0-alpha",http_method="GET",http_route="/docs/$file<.+>",http_scheme="http",http_status_code="200",net_host_name="mysrv.mycompany.net",net_host_port="9001",net_protocol_name="http",net_protocol_version="1.1",le="7500.0"} 4.0 1701100607547
http_server_duration_milliseconds_bucket{otel_scope_name="io.opentelemetry.akka-http-10.0",otel_scope_version="1.29.0-alpha",http_method="GET",http_route="/docs/$file<.+>",http_scheme="http",http_status_code="200",net_host_name="mysrv.mycompany.net",net_host_port="9001",net_protocol_name="http",net_protocol_version="1.1",le="10000.0"} 4.0 1701100607547
http_server_duration_milliseconds_bucket{otel_scope_name="io.opentelemetry.akka-http-10.0",otel_scope_version="1.29.0-alpha",http_method="GET",http_route="/docs/$file<.+>",http_scheme="http",http_status_code="200",net_host_name="mysrv.mycompany.net",net_host_port="9001",net_protocol_name="http",net_protocol_version="1.1",le="+Inf"} 4.0 1701100607547
http_server_duration_milliseconds_count{otel_scope_name="io.opentelemetry.akka-http-10.0",otel_scope_version="1.29.0-alpha",http_method="GET",http_route="/api/message/$messageId<[^/]+>",http_scheme="http",http_status_code="404",net_host_name="mysrv.mycompany.net",net_host_port="9001",net_protocol_name="http",net_protocol_version="1.1"} 2.0 1701100607547
http_server_duration_milliseconds_sum{otel_scope_name="io.opentelemetry.akka-http-10.0",otel_scope_version="1.29.0-alpha",http_method="GET",http_route="/api/message/$messageId<[^/]+>",http_scheme="http",http_status_code="404",net_host_name="mysrv.mycompany.net",net_host_port="9001",net_protocol_name="http",net_protocol_version="1.1"} 884.285349 1701100607547
http_server_duration_milliseconds_bucket{otel_scope_name="io.opentelemetry.akka-http-10.0",otel_scope_version="1.29.0-alpha",http_method="GET",http_route="/api/message/$messageId<[^/]+>",http_scheme="http",http_status_code="404",net_host_name="mysrv.mycompany.net",net_host_port="9001",net_protocol_name="http",net_protocol_version="1.1",le="0.0"} 0.0 1701100607547
http_server_duration_milliseconds_bucket{otel_scope_name="io.opentelemetry.akka-http-10.0",otel_scope_version="1.29.0-alpha",http_method="GET",http_route="/api/message/$messageId<[^/]+>",http_scheme="http",http_status_code="404",net_host_name="mysrv.mycompany.net",net_host_port="9001",net_protocol_name="http",net_protocol_version="1.1",le="5.0"} 0.0 1701100607547
http_server_duration_milliseconds_bucket{otel_scope_name="io.opentelemetry.akka-http-10.0",otel_scope_version="1.29.0-alpha",http_method="GET",http_route="/api/message/$messageId<[^/]+>",http_scheme="http",http_status_code="404",net_host_name="mysrv.mycompany.net",net_host_port="9001",net_protocol_name="http",net_protocol_version="1.1",le="10.0"} 0.0 1701100607547
http_server_duration_milliseconds_bucket{otel_scope_name="io.opentelemetry.akka-http-10.0",otel_scope_version="1.29.0-alpha",http_method="GET",http_route="/api/message/$messageId<[^/]+>",http_scheme="http",http_status_code="404",net_host_name="mysrv.mycompany.net",net_host_port="9001",net_protocol_name="http",net_protocol_version="1.1",le="25.0"} 0.0 1701100607547
http_server_duration_milliseconds_bucket{otel_scope_name="io.opentelemetry.akka-http-10.0",otel_scope_version="1.29.0-alpha",http_method="GET",http_route="/api/message/$messageId<[^/]+>",http_scheme="http",http_status_code="404",net_host_name="mysrv.mycompany.net",net_host_port="9001",net_protocol_name="http",net_protocol_version="1.1",le="50.0"} 0.0 1701100607547
http_server_duration_milliseconds_bucket{otel_scope_name="io.opentelemetry.akka-http-10.0",otel_scope_version="1.29.0-alpha",http_method="GET",http_route="/api/message/$messageId<[^/]+>",http_scheme="http",http_status_code="404",net_host_name="mysrv.mycompany.net",net_host_port="9001",net_protocol_name="http",net_protocol_version="1.1",le="75.0"} 1.0 1701100607547
http_server_duration_milliseconds_bucket{otel_scope_name="io.opentelemetry.akka-http-10.0",otel_scope_version="1.29.0-alpha",http_method="GET",http_route="/api/message/$messageId<[^/]+>",http_scheme="http",http_status_code="404",net_host_name="mysrv.mycompany.net",net_host_port="9001",net_protocol_name="http",net_protocol_version="1.1",le="100.0"} 1.0 1701100607547
http_server_duration_milliseconds_bucket{otel_scope_name="io.opentelemetry.akka-http-10.0",otel_scope_version="1.29.0-alpha",http_method="GET",http_route="/api/message/$messageId<[^/]+>",http_scheme="http",http_status_code="404",net_host_name="mysrv.mycompany.net",net_host_port="9001",net_protocol_name="http",net_protocol_version="1.1",le="250.0"} 1.0 1701100607547
http_server_duration_milliseconds_bucket{otel_scope_name="io.opentelemetry.akka-http-10.0",otel_scope_version="1.29.0-alpha",http_method="GET",http_route="/api/message/$messageId<[^/]+>",http_scheme="http",http_status_code="404",net_host_name="mysrv.mycompany.net",net_host_port="9001",net_protocol_name="http",net_protocol_version="1.1",le="500.0"} 1.0 1701100607547
http_server_duration_milliseconds_bucket{otel_scope_name="io.opentelemetry.akka-http-10.0",otel_scope_version="1.29.0-alpha",http_method="GET",http_route="/api/message/$messageId<[^/]+>",http_scheme="http",http_status_code="404",net_host_name="mysrv.mycompany.net",net_host_port="9001",net_protocol_name="http",net_protocol_version="1.1",le="750.0"} 1.0 1701100607547
http_server_duration_milliseconds_bucket{otel_scope_name="io.opentelemetry.akka-http-10.0",otel_scope_version="1.29.0-alpha",http_method="GET",http_route="/api/message/$messageId<[^/]+>",http_scheme="http",http_status_code="404",net_host_name="mysrv.mycompany.net",net_host_port="9001",net_protocol_name="http",net_protocol_version="1.1",le="1000.0"} 2.0 1701100607547
http_server_duration_milliseconds_bucket{otel_scope_name="io.opentelemetry.akka-http-10.0",otel_scope_version="1.29.0-alpha",http_method="GET",http_route="/api/message/$messageId<[^/]+>",http_scheme="http",http_status_code="404",net_host_name="mysrv.mycompany.net",net_host_port="9001",net_protocol_name="http",net_protocol_version="1.1",le="2500.0"} 2.0 1701100607547
http_server_duration_milliseconds_bucket{otel_scope_name="io.opentelemetry.akka-http-10.0",otel_scope_version="1.29.0-alpha",http_method="GET",http_route="/api/message/$messageId<[^/]+>",http_scheme="http",http_status_code="404",net_host_name="mysrv.mycompany.net",net_host_port="9001",net_protocol_name="http",net_protocol_version="1.1",le="5000.0"} 2.0 1701100607547
http_server_duration_milliseconds_bucket{otel_scope_name="io.opentelemetry.akka-http-10.0",otel_scope_version="1.29.0-alpha",http_method="GET",http_route="/api/message/$messageId<[^/]+>",http_scheme="http",http_status_code="404",net_host_name="mysrv.mycompany.net",net_host_port="9001",net_protocol_name="http",net_protocol_version="1.1",le="7500.0"} 2.0 1701100607547
http_server_duration_milliseconds_bucket{otel_scope_name="io.opentelemetry.akka-http-10.0",otel_scope_version="1.29.0-alpha",http_method="GET",http_route="/api/message/$messageId<[^/]+>",http_scheme="http",http_status_code="404",net_host_name="mysrv.mycompany.net",net_host_port="9001",net_protocol_name="http",net_protocol_version="1.1",le="10000.0"} 2.0 1701100607547
http_server_duration_milliseconds_bucket{otel_scope_name="io.opentelemetry.akka-http-10.0",otel_scope_version="1.29.0-alpha",http_method="GET",http_route="/api/message/$messageId<[^/]+>",http_scheme="http",http_status_code="404",net_host_name="mysrv.mycompany.net",net_host_port="9001",net_protocol_name="http",net_protocol_version="1.1",le="+Inf"} 2.0 1701100607547
http_server_duration_milliseconds_count{otel_scope_name="io.opentelemetry.akka-http-10.0",otel_scope_version="1.29.0-alpha",http_method="GET",http_route="/docs/swagger-initializer.js",http_scheme="http",http_status_code="200",net_host_name="mysrv.mycompany.net",net_host_port="9001",net_protocol_name="http",net_protocol_version="1.1"} 1.0 1701100607547
http_server_duration_milliseconds_sum{otel_scope_name="io.opentelemetry.akka-http-10.0",otel_scope_version="1.29.0-alpha",http_method="GET",http_route="/docs/swagger-initializer.js",http_scheme="http",http_status_code="200",net_host_name="mysrv.mycompany.net",net_host_port="9001",net_protocol_name="http",net_protocol_version="1.1"} 83.999574 1701100607547
http_server_duration_milliseconds_bucket{otel_scope_name="io.opentelemetry.akka-http-10.0",otel_scope_version="1.29.0-alpha",http_method="GET",http_route="/docs/swagger-initializer.js",http_scheme="http",http_status_code="200",net_host_name="mysrv.mycompany.net",net_host_port="9001",net_protocol_name="http",net_protocol_version="1.1",le="0.0"} 0.0 1701100607547
http_server_duration_milliseconds_bucket{otel_scope_name="io.opentelemetry.akka-http-10.0",otel_scope_version="1.29.0-alpha",http_method="GET",http_route="/docs/swagger-initializer.js",http_scheme="http",http_status_code="200",net_host_name="mysrv.mycompany.net",net_host_port="9001",net_protocol_name="http",net_protocol_version="1.1",le="5.0"} 0.0 1701100607547
http_server_duration_milliseconds_bucket{otel_scope_name="io.opentelemetry.akka-http-10.0",otel_scope_version="1.29.0-alpha",http_method="GET",http_route="/docs/swagger-initializer.js",http_scheme="http",http_status_code="200",net_host_name="mysrv.mycompany.net",net_host_port="9001",net_protocol_name="http",net_protocol_version="1.1",le="10.0"} 0.0 1701100607547
http_server_duration_milliseconds_bucket{otel_scope_name="io.opentelemetry.akka-http-10.0",otel_scope_version="1.29.0-alpha",http_method="GET",http_route="/docs/swagger-initializer.js",http_scheme="http",http_status_code="200",net_host_name="mysrv.mycompany.net",net_host_port="9001",net_protocol_name="http",net_protocol_version="1.1",le="25.0"} 0.0 1701100607547
http_server_duration_milliseconds_bucket{otel_scope_name="io.opentelemetry.akka-http-10.0",otel_scope_version="1.29.0-alpha",http_method="GET",http_route="/docs/swagger-initializer.js",http_scheme="http",http_status_code="200",net_host_name="mysrv.mycompany.net",net_host_port="9001",net_protocol_name="http",net_protocol_version="1.1",le="50.0"} 0.0 1701100607547
http_server_duration_milliseconds_bucket{otel_scope_name="io.opentelemetry.akka-http-10.0",otel_scope_version="1.29.0-alpha",http_method="GET",http_route="/docs/swagger-initializer.js",http_scheme="http",http_status_code="200",net_host_name="mysrv.mycompany.net",net_host_port="9001",net_protocol_name="http",net_protocol_version="1.1",le="75.0"} 0.0 1701100607547
http_server_duration_milliseconds_bucket{otel_scope_name="io.opentelemetry.akka-http-10.0",otel_scope_version="1.29.0-alpha",http_method="GET",http_route="/docs/swagger-initializer.js",http_scheme="http",http_status_code="200",net_host_name="mysrv.mycompany.net",net_host_port="9001",net_protocol_name="http",net_protocol_version="1.1",le="100.0"} 1.0 1701100607547
http_server_duration_milliseconds_bucket{otel_scope_name="io.opentelemetry.akka-http-10.0",otel_scope_version="1.29.0-alpha",http_method="GET",http_route="/docs/swagger-initializer.js",http_scheme="http",http_status_code="200",net_host_name="mysrv.mycompany.net",net_host_port="9001",net_protocol_name="http",net_protocol_version="1.1",le="250.0"} 1.0 1701100607547
http_server_duration_milliseconds_bucket{otel_scope_name="io.opentelemetry.akka-http-10.0",otel_scope_version="1.29.0-alpha",http_method="GET",http_route="/docs/swagger-initializer.js",http_scheme="http",http_status_code="200",net_host_name="mysrv.mycompany.net",net_host_port="9001",net_protocol_name="http",net_protocol_version="1.1",le="500.0"} 1.0 1701100607547
http_server_duration_milliseconds_bucket{otel_scope_name="io.opentelemetry.akka-http-10.0",otel_scope_version="1.29.0-alpha",http_method="GET",http_route="/docs/swagger-initializer.js",http_scheme="http",http_status_code="200",net_host_name="mysrv.mycompany.net",net_host_port="9001",net_protocol_name="http",net_protocol_version="1.1",le="750.0"} 1.0 1701100607547
http_server_duration_milliseconds_bucket{otel_scope_name="io.opentelemetry.akka-http-10.0",otel_scope_version="1.29.0-alpha",http_method="GET",http_route="/docs/swagger-initializer.js",http_scheme="http",http_status_code="200",net_host_name="mysrv.mycompany.net",net_host_port="9001",net_protocol_name="http",net_protocol_version="1.1",le="1000.0"} 1.0 1701100607547
http_server_duration_milliseconds_bucket{otel_scope_name="io.opentelemetry.akka-http-10.0",otel_scope_version="1.29.0-alpha",http_method="GET",http_route="/docs/swagger-initializer.js",http_scheme="http",http_status_code="200",net_host_name="mysrv.mycompany.net",net_host_port="9001",net_protocol_name="http",net_protocol_version="1.1",le="2500.0"} 1.0 1701100607547
http_server_duration_milliseconds_bucket{otel_scope_name="io.opentelemetry.akka-http-10.0",otel_scope_version="1.29.0-alpha",http_method="GET",http_route="/docs/swagger-initializer.js",http_scheme="http",http_status_code="200",net_host_name="mysrv.mycompany.net",net_host_port="9001",net_protocol_name="http",net_protocol_version="1.1",le="5000.0"} 1.0 1701100607547
http_server_duration_milliseconds_bucket{otel_scope_name="io.opentelemetry.akka-http-10.0",otel_scope_version="1.29.0-alpha",http_method="GET",http_route="/docs/swagger-initializer.js",http_scheme="http",http_status_code="200",net_host_name="mysrv.mycompany.net",net_host_port="9001",net_protocol_name="http",net_protocol_version="1.1",le="7500.0"} 1.0 1701100607547
http_server_duration_milliseconds_bucket{otel_scope_name="io.opentelemetry.akka-http-10.0",otel_scope_version="1.29.0-alpha",http_method="GET",http_route="/docs/swagger-initializer.js",http_scheme="http",http_status_code="200",net_host_name="mysrv.mycompany.net",net_host_port="9001",net_protocol_name="http",net_protocol_version="1.1",le="10000.0"} 1.0 1701100607547
http_server_duration_milliseconds_bucket{otel_scope_name="io.opentelemetry.akka-http-10.0",otel_scope_version="1.29.0-alpha",http_method="GET",http_route="/docs/swagger-initializer.js",http_scheme="http",http_status_code="200",net_host_name="mysrv.mycompany.net",net_host_port="9001",net_protocol_name="http",net_protocol_version="1.1",le="+Inf"} 1.0 1701100607547

With a real application having a few dozens of routes, I believe we quickly reach "high cardinality", don't we?

[laurit]

Our akka http instrumentation does not fill http.route yet. Perhaps you are using https://github.com/ScalaConsultants/mesmer Anyway having these metrics per route is expected. If you don't wan't it to include http.route you could try configuring a metrics view https://github.com/open-telemetry/opentelemetry-java/tree/main/sdk-extensions/incubator that does not include http.route.

[gaeljw]

I confirm I saw this using OTEL java agent (1.29.0). You can see it from the otel_scope_name="io.opentelemetry.akka-http-10.0",otel_scope_version="1.29.0-alpha" labels.

Where can this come from if not OTEL instrumentation? 🤔

I'll look into configuring a view if I really need to though. Thanks for the tip @laurit .

[trask]

Where can this come from if not OTEL instrumentation?

The span could be coming from akka instrumentation, but the http.route attribute should be coming from another (otel or other) instrumentation, e.g. Spring MVC, etc...

[gaeljw]

Ah I had not realized attributes could be coming from another span. Got it!

This must be Play MVC OTEL instrumentation then in my case.

[matthughes]

Typically instrumentation like this should let you modify and simplify these label names. You might buckets all your routes into 5-10 labels instead of a label per unique uri.