Consider adopting @opentelemetry/instrumentation-runtime-node

[gibson042]

What is the Problem Being Solved?

kernel-stats.js includes a note about the lack of off-the-shelf Node.js OpenTelemetry instrumentation:

agoric-sdk/packages/cosmic-swingset/src/kernel-stats.js

Lines 468 to 509 in 0ca4b57

	// TODO: We probably shouldn't roll our own Node.js process metrics, but a
	// cursory search for "opentelemetry node.js VM instrumentation" didn't reveal
	// anything useful.
	let heapStatsLast = 0;
	let heapStatsCache = v8.getHeapStatistics();
	const getHeapStats = () => {
	const now = Date.now();
	if (now - heapStatsLast >= 800) {
	heapStatsLast = now;
	heapStatsCache = v8.getHeapStatistics();
	}
	return heapStatsCache;
	};
	for (const key of Object.keys(heapStatsCache)) {
	const name = `heapStats_${key}`;
	const options = { description: 'v8 kernel heap statistic' };
	const counter = metricMeter.createObservableUpDownCounter(name, options);
	counter.addCallback(observableResult => {
	observableResult.observe(getHeapStats()[key], attributes);
	});
	}
	let memoryUsageLast = 0;
	let memoryUsageCache = process.memoryUsage();
	const getMemoryUsage = () => {
	const now = Date.now();
	if (now - memoryUsageLast >= 800) {
	memoryUsageLast = now;
	memoryUsageCache = process.memoryUsage();
	}
	return memoryUsageCache;
	};
	for (const key of Object.keys(memoryUsageCache)) {
	const name = `memoryUsage_${key}`;
	const options = { description: 'kernel process memory statistic' };
	const counter = metricMeter.createObservableUpDownCounter(name, options);
	counter.addCallback(observableResult => {
	observableResult.observe(getMemoryUsage()[key], attributes);
	});
	}

Since that time, @opentelemetry/instrumentation-runtime-node has been published.

Description of the Design

We need to decide what combination of data we actually want. Metrics published by the new package have significant overlap with our custom metrics and are generally richer, but are also incomplete in some ways that may or may not be worth continuing to report.

Description	v8.getHeapStatistics()	process.memoryUsage()	instrumentation-runtime-node v0.12.1
the number of bytes V8 has allocated for the heap	total_heap_size	heapTotal	v8js_memory_heap_limit (partitioned by new/old/code/…)
the portion of the heap that can contain executable code	total_heap_size_executable	-	v8js_memory_heap_limit {v8js_heap_space_name="code_space"}
actual physical memory used by the V8 heap	total_physical_size	-	v8js_memory_heap_space_physical_size (partitioned by new/old/code/…)
memory available to the V8 heap (how much more memory V8 can use before it exceeds the heap limit)	total_available_size	-	-
number of bytes currently being used by V8’s JavaScript objects	used_heap_size	heapUsed	v8js_memory_heap_used (partitioned by new/old/code/…)
maximum size of the V8 heap	heap_size_limit	-	-
number of bytes allocated through malloc by V8	malloced_memory	-	-
peak number of bytes allocated through malloc by V8 during the lifetime of the process	peak_malloced_memory	-	-
signifies whether the --zap_code_space option is enabled	does_zap_garbage	-	-
the number of the top-level contexts currently active	number_of_native_contexts	-	-
the number of contexts that were detached and not yet garbage collected	number_of_detached_contexts	-	-
total memory size of V8 global handles	total_global_handles_size	-	-
used memory size of V8 global handles	used_global_handles_size	-	-
memory size of array buffers/external strings/C++ objects bound to V8-managed JavaScript objects	external_memory (inclusive of strings)	external (inclusive of arrayBuffers, exclusive of strings)	-
"Heap space available size"	? (not total_available_size, which is far too big)	-	v8js_memory_heap_space_available_size
Event loop utilization	-	-	nodejs_eventloop_utilization (value between 0 and 1)
cumulative duration of event loop time	-	-	nodejs_eventloop_time (partitioned by idle/active)
Event loop delay	-	-	nodejs_eventloop_delay_${stat}, (stat in {min,max,mean,stddev,p50,p90,p99})
Garbage collection duration	-	-	v8js_gc_duration_{count,sum,bucket} histogram (partitioned by major/minor/incremental/weakcb)
Resident Set Size	-	rss	-

Recommendation

Adopt the package and stop producing most existing custom metrics:

Non-subsumed Metric	Keep	Notes
heapStats_heap_size_limit	-	seems to be static ("determined by system resources, or… --max_old_space_size")
heapStats_total_available_size	-	difference of used memory and heap_size_limit
heapStats_malloced_memory	-
heapStats_peak_malloced_memory	-
heapStats_does_zap_garbage	-	static
heapStats_number_of_native_contexts	yes (but maybe rename)	"Increase of this number over time indicates a memory leak"
heapStats_number_of_detached_contexts	yes (but maybe rename)	"This number being non-zero indicates a potential memory leak"
heapStats_total_global_handles_size	-
heapStats_used_global_handles_size	-
heapStats_external_memory	-
memoryUsage_rss	yes (but maybe rename)	maybe available elsewhere right now, but not after "split-brain" Go/JS process separation
memoryUsage_heapTotal	-
memoryUsage_heapUsed	-
memoryUsage_external	-
memoryUsage_arrayBuffers	-

Security Considerations

None known.

Scaling Considerations

Metrics production shouldn't be expensive, but our use of Node.js is definitely unusual. We should verify that there's no obvious significant performance penalty before adopting the package.

Test Plan

The new metrics should be covered by changes to prometheus.test.js.

Upgrade Considerations

Existing consumption of heapStats_${key} and memoryUsage_${key} must be prepared for the new names before any change that stops producing them.