benchmark update

[margaretjgu]

Goal is to have more consistent builds

use cases tried:

1. use existing core image and take median of several runs to determine thresholds docker.elastic.co/ci-agent-images/ems/buildkite-agent-node22

ran benchmarks with no changes and saw irregular numbers ^

• update to use dedicated agent

[margaretjgu]

initial run metrics:

💚 Build Succeeded

• Buildkite Build
• Commit: c9fde3a

Performance benchmark

Benchmark details (Ubuntu)

Benchmark Comparison

Summary

4 metric(s) exceeded threshold

Metric	Change
Transport#constructor - UndiciConnection > WeightedConnectionPool p99	+35.9%
Search request: defaults + compression opsPerSec	+4.8%
Search request: defaults + compression external	+86.1%
Search request: defaults + compression d external	+2723.9%

2 warning(s)

Metric	Change
Transport#constructor - UndiciConnection > WeightedConnectionPool p75	+16.3%
Search request: defaults heapTotal	+39.7%

Performance Benchmarks

Transport#constructor - UndiciConnection

Pool	Metric	Base	PR	Change
WeightedConnectionPool	p75	426.8us	496.2us	+16.3% [WARN]
	p99	519.9us	706.5us	+35.9% [FAIL]
	avg	388.8us	420.3us	+8.1%
ClusterConnectionPool	p75	380.9us	374.0us	-1.8%
	p99	419.1us	413.4us	-1.4%
	avg	368.5us	359.6us	-2.4%
CloudConnectionPool	p75	379.0us	353.8us	-6.7%
	p99	427.3us	400.2us	-6.3%
	avg	361.1us	341.5us	-5.4%

Transport#constructor - HttpConnection

Pool	Metric	Base	PR	Change
WeightedConnectionPool	p75	49.3us	40.8us	-17.3% [OK]
	p99	51.3us	42.4us	-17.3% [OK]
	avg	45.4us	41.2us	-9.2%
ClusterConnectionPool	p75	42.5us	41.2us	-3.0%
	p99	49.5us	50.6us	+2.2%
	avg	43.1us	42.1us	-2.3%
CloudConnectionPool	p75	42.2us	43.0us	+1.9%
	p99	42.9us	45.4us	+5.9%
	avg	39.7us	41.3us	+4.1%

GC & Memory Benchmarks

Search request: defaults

Metric	Base	PR	Change
Throughput
ops/sec	31,064	44,842	+44.4% [OK]
avg latency	32.2us	22.3us	-30.7% [OK]
total time	160.96ms	111.50ms	-30.7%
iterations	5,000	5,000	-
Garbage Collection
GC events	0	0	none
Memory (after test)
heap used	10.8 MB	10.9 MB	+0.3%
heap total	20.2 MB	28.2 MB	+39.7% [WARN]
external	3.3 MB	3.3 MB	+0.0%
array buffers	66.4 KB	66.4 KB	+0.0%
Memory delta
d heap used	+85.1 KB	+85.6 KB	+0.6%
d heap total	+0.0 B	+256.0 KB	n/a
d external	+0.0 B	+0.0 B	n/a

Search request: defaults + compression

Metric	Base	PR	Change
Throughput
ops/sec	5,934	6,216	+4.8% [FAIL]
avg latency	168.5us	160.9us	-4.5%
total time	842.57ms	804.30ms	-4.5%
iterations	5,000	5,000	-
Garbage Collection
GC events	18.0	9.00	-50.0% [OK]
GC total time	81.5 ms	15.2 ms	-81.3% [OK]
GC avg time	4.53 ms	1.69 ms	-62.7% [OK]
GC max pause	64.0 ms	4.38 ms	-93.2% [OK]
Memory (after test)
heap used	11.4 MB	11.5 MB	+0.2%
heap total	28.9 MB	28.9 MB	+0.0%
external	5.2 MB	9.7 MB	+86.1% [FAIL]
array buffers	90.4 KB	90.4 KB	+0.0%
Memory delta
d heap used	+389.2 KB	+391.8 KB	+0.7%
d heap total	+8.3 MB	+256.0 KB	-97.0% [OK]
d external	+168.1 KB	+4.6 MB	+2723.9% [FAIL]

History

• 💔 Build #24 failed 726cec4
• 💔 Build #23 failed 2b4e643
• 💚 Build #21 succeeded 9df14a3
• 💔 Build #20 failed ad4f4ac
• 💚 Build #19 succeeded 91fc630

[elasticmachine]

💚 Build Succeeded

• Buildkite Build
• Commit: f427559

Performance benchmark

Benchmark details (Ubuntu)

Benchmark Comparison

Summary

2 metric(s) exceeded threshold

Metric	Change
Search request: defaults opsPerSec	+4.4%
Search request: defaults + compression opsPerSec	+9.6%

Performance Benchmarks

Transport#constructor - UndiciConnection

Pool	Metric	Base	PR	Change
WeightedConnectionPool	p75	370.3us	371.7us	+0.4%
	p99	450.3us	453.7us	+0.8%
	avg	345.3us	347.0us	+0.5%
ClusterConnectionPool	p75	365.4us	373.3us	+2.2%
	p99	436.5us	429.0us	-1.7%
	avg	339.9us	353.0us	+3.9%
CloudConnectionPool	p75	45.1us	46.8us	+3.7%
	p99	52.5us	47.3us	-9.8%
	avg	44.6us	44.6us	-0.0%

Transport#constructor - HttpConnection

Pool	Metric	Base	PR	Change
WeightedConnectionPool	p75	36.5us	38.1us	+4.6%
	p99	41.4us	40.5us	-2.1%
	avg	34.4us	35.7us	+3.6%
ClusterConnectionPool	p75	37.5us	38.4us	+2.3%
	p99	38.7us	39.8us	+2.9%
	avg	35.6us	36.6us	+2.8%
CloudConnectionPool	p75	34.0us	33.5us	-1.4%
	p99	36.6us	35.1us	-4.2%
	avg	33.7us	31.4us	-6.8%

GC & Memory Benchmarks

Search request: defaults

Metric	Base	PR	Change
Throughput
ops/sec	45,017	46,979	+4.4% [FAIL]
avg latency	22.2us	21.3us	-4.2%
total time	111.07ms	106.43ms	-4.2%
iterations	5,000	5,000	-
Garbage Collection
GC events	0	0	none
Memory (after test)
heap used	10.8 MB	10.9 MB	+0.3%
heap total	28.2 MB	28.4 MB	+0.9%

Search request: defaults + compression

Metric	Base	PR	Change
Throughput
ops/sec	7,229	7,925	+9.6% [FAIL]
avg latency	138.3us	126.2us	-8.8%
total time	691.56ms	630.85ms	-8.8%
iterations	5,000	5,000	-
Garbage Collection
GC events	16.0	14.0	-12.5%
GC total time	22.2 ms	20.7 ms	-6.8%
GC avg time	1.67 ms	1.60 ms	-4.4%
GC max pause	10.6 ms	5.83 ms	-44.9%
Memory (after test)
heap used	11.4 MB	11.5 MB	+0.3%
heap total	29.4 MB	29.4 MB	+0.0%

History

• 💚 Build #25 succeeded c9fde3a
• 💔 Build #24 failed 726cec4
• 💔 Build #23 failed 2b4e643
• 💚 Build #21 succeeded 9df14a3
• 💔 Build #20 failed ad4f4ac
• 💚 Build #19 succeeded 91fc630