Add new awsebsnvmereceiver #1603

duhminick · 2025-03-18T14:54:55Z

Note to Reviewers

This PR does not enable this new receiver. This is strictly just adding a new receiver that will be enabled later.

Changes 2:

Use fmt.Sscanf instead of the custom parsing logic for the device file name
Enable only one metric by default. This is for the upcoming translation changes -- this just simplifies some of the logic. We need to keep at least one metric enabled by default or else the generated tests from mdatagen fail
Rename Resources to Devices in the receiver config

Description of the issue

Elastic Block Storage (EBS) exposes performance statistics for EBS volumes attached to EC2 instances as NVMe devices in a vendor unique log page. The log page can be retrieved by making a system call to the NVMe device. CloudWatch Agent (CWA) is going to collect the retrieved metrics and emit them to CloudWatch.

Description of changes

Main Scraper Implementation (scraper.go):
- Implements the core scraping logic through the nvmeScraper struct
- Uses a MetricsBuilder to construct standardized metrics
  - MetricsBuilder is generated using mdatagen with the schema defined in metadata.yaml
- Handles device discovery and metric collection for EBS NVMe devices
  - Includes the ability to scrape all devices, or specific devices.
- Implements safety checks for integer overflow protection
NVMe Metrics Collection (internal/nvme):
- Implements low-level NVMe device interaction through Linux ioctl calls
- Provides structured metric collection through EBSMetrics struct
- Collects key metrics including:
  - Read/Write operations and bytes
  - Total read/write times
  - IOPS and throughput exceeded counters
  - Queue length
  - Read/Write latency histograms (though this is not being collected atm)
Generated components (internal/metadata);

All generated by mdatagen using the schema defined in metadata.yaml

The receiver collects the following metrics from EBS NVMe devices:

diskio_ebs_total_read_ops
diskio_ebs_total_write_ops
diskio_ebs_total_read_bytes
diskio_ebs_total_write_bytes
diskio_ebs_total_read_time
diskio_ebs_total_write_time
diskio_ebs_volume_performance_exceeded_iops
diskio_ebs_volume_performance_exceeded_tp
diskio_ebs_ec2_instance_performance_exceeded_iops
diskio_ebs_ec2_instance_performance_exceeded_tp
diskio_ebs_volume_queue_length

License

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

Tests

Manually updated the YAML config to have a new instance of the EBS receiver as well as update the existing host delta metrics pipeline to have the new receiver.

Attached an EBS volume to the instance

The EC2 instance that the manual tests were ran on have two EBS volumes attached (nvme0 and nvme1)

Resource is explicitly empty

2025-03-18T19:35:03Z D! {"caller":"awsebsnvmereceiver/scraper.go:54","msg":"Began scraping for NVMe metrics","kind":"receiver","name":"awsebsnvmereceiver","data_type":"metrics"}
2025-03-18T19:35:03Z D! {"caller":"awsebsnvmereceiver/scraper.go:133","msg":"skipping un-allowed device","kind":"receiver","name":"awsebsnvmereceiver","data_type":"metrics","device":"nvme0"}
2025-03-18T19:35:03Z D! {"caller":"awsebsnvmereceiver/scraper.go:133","msg":"skipping un-allowed device","kind":"receiver","name":"awsebsnvmereceiver","data_type":"metrics","device":"nvme0n1"}

One device (nvme0) is in resources

2025-03-18T19:36:24Z D! {"caller":"awsebsnvmereceiver/scraper.go:54","msg":"Began scraping for NVMe metrics","kind":"receiver","name":"awsebsnvmereceiver","data_type":"metrics"}
2025-03-18T19:36:24Z D! {"caller":"awsebsnvmereceiver/scraper.go:133","msg":"skipping un-allowed device","kind":"receiver","name":"awsebsnvmereceiver","data_type":"metrics","device":"nvme0n1"}
2025-03-18T19:36:24Z D! {"caller":"awsebsnvmereceiver/scraper.go:133","msg":"skipping un-allowed device","kind":"receiver","name":"awsebsnvmereceiver","data_type":"metrics","device":"nvme0n1p1"}
2025-03-18T19:36:24Z D! {"caller":"awsebsnvmereceiver/scraper.go:101","msg":"emitted metrics for nvme device with controller id","kind":"receiver","name":"awsebsnvmereceiver","data_type":"metrics","controllerID":0}

`*` for resources

2025-03-18T19:38:30Z D! {"caller":"awsebsnvmereceiver/scraper.go:54","msg":"Began scraping for NVMe metrics","kind":"receiver","name":"awsebsnvmereceiver","data_type":"metrics"}
2025-03-18T19:38:30Z D! {"caller":"awsebsnvmereceiver/scraper.go:101","msg":"emitted metrics for nvme device with controller id","kind":"receiver","name":"awsebsnvmereceiver","data_type":"metrics","controllerID":0}
2025-03-18T19:38:30Z D! {"caller":"awsebsnvmereceiver/scraper.go:101","msg":"emitted metrics for nvme device with controller id","kind":"receiver","name":"awsebsnvmereceiver","data_type":"metrics","controllerID":1}

Sample Config

receivers:
    awsebsnvmereceiver:
        collection_interval: 1m0s
        initial_delay: 1s
        metrics:
            diskio_ebs_ec2_instance_performance_exceeded_iops:
                enabled: false
            diskio_ebs_ec2_instance_performance_exceeded_tp:
                enabled: false
            diskio_ebs_total_read_bytes:
                enabled: true
            diskio_ebs_total_read_ops:
                enabled: false
            diskio_ebs_total_read_time:
                enabled: false
            diskio_ebs_total_write_bytes:
                enabled: true
            diskio_ebs_total_write_ops:
                enabled: false
            diskio_ebs_total_write_time:
                enabled: false
            diskio_ebs_volume_performance_exceeded_iops:
                enabled: false
            diskio_ebs_volume_performance_exceeded_tp:
                enabled: false
            diskio_ebs_volume_queue_length:
                enabled: false
        resource_attributes:
            VolumeId:
                enabled: true
        devices:
            - nvme0n1
        timeout: 0s

Requirements

Before commit the code, please do the following steps.

Run make fmt and make fmt-sh
Run make lint

receiver/awsebsnvmereceiver/config.go

receiver/awsebsnvmereceiver/internal/metadata/generated_config.go

receiver/awsebsnvmereceiver/internal/metadata/testdata/config.yaml

receiver/awsebsnvmereceiver/internal/nvme/device_file_attributes.go

receiver/awsebsnvmereceiver/internal/nvme/ebs_metrics.go

receiver/awsebsnvmereceiver/scraper.go

lisguo · 2025-03-25T13:53:01Z

"This PR should be merged into the ebs branch"

Is this still true? We are just adding the receiver not enabling it so putting it to main seems fine

duhminick · 2025-03-25T14:59:02Z

Is this still true? We are just adding the receiver not enabling it so putting it to main seems fine

Oh, got it. Then I'll leave the destination branch as is

duhminick · 2025-03-25T16:04:17Z

agent.json
output.txt

Attaching the agent config and translated configs

lisguo

Can you also provide a sample yaml config of the receiver?

lisguo · 2025-03-26T15:01:07Z

receiver/awsebsnvmereceiver/internal/nvme/device_file_attributes.go

+	namespace := -1
+	partition := -1
+
+	fmt.Sscanf(device, "nvme%dn%dp%d", &controller, &namespace, &partition)


can we check if this returns an error? if so then we are unable to parse the device

I opted to ignore the error and chose to throw an error if we couldn't at least get the controller ID. The reason is that this function parses the three different device file name patterns (nvme{id}, nvme{id}n{namespace}, nvme{id}n{namespace}p{partition}). If the input isn't the third pattern, then there's an "EOF" error.

I could instead just do something like check for nvme%dn%dp%d, then nvme%dn%d, and lastly nvme%d. I guess that'll give a better error message but it seems like a lot of excess work. In which then I kind of like the implementation I had before.

Or, we could do a regular expression like: ^nvme(\d+)(?:n(\d+))?(?:p(\d+))?$. https://regex101.com/r/5TjfEE/1

nit: Make it clear that it can throw an error and we're explicitly ignoring it.

Suggested change

fmt.Sscanf(device, "nvme%dn%dp%d", &controller, &namespace, &partition)

_, _ = fmt.Sscanf(device, "nvme%dn%dp%d", &controller, &namespace, &partition)

lisguo · 2025-03-26T15:19:51Z

receiver/awsebsnvmereceiver/scraper.go

+		}
+
+		// Check if all devices should be collected. Otherwise check if defined by user
+		_, hasAsterisk := s.allowedDevices["*"]


do we have plans on filtering by devices?

yep, it's possible to filter by devices already. it's just below the line you highlighted 😄

amazon-cloudwatch-agent/receiver/awsebsnvmereceiver/scraper.go

Lines 131 to 134 in 9d4a053

if _, isAllowed := s.allowedDevices[deviceName]; !isAllowed {

s.logger.Debug("skipping un-allowed device", zap.String("device", deviceName))

continue

}

jefchien · 2025-03-28T19:15:22Z

receiver/awsebsnvmereceiver/internal/nvme/util_stub.go

nit: Typically, in Go conventon, the file would be named something like util_notunix.go like https://github.com/aws/amazon-cloudwatch-agent/blob/main/plugins/inputs/windows_event_log/windows_event_log_notwindows.go

Oh. I was following what I found in the Go src. Example: https://github.com/golang/go/blob/master/src/net/cgo_stub.go. I can change it to what we do for the agent to keep it consistent though.

jefchien · 2025-03-28T19:18:13Z

receiver/awsebsnvmereceiver/internal/nvme/util.go

+	IsEbsDevice(device *DeviceFileAttributes) (bool, error)
+}
+
+type Util struct {


nit: I know we use Util as a name all over the place, but it doesn't help me understand what this is. Would rather it be called like DeviceInfoProvider or something more descriptive.

nit: Why is this a struct if it doesn't have a state? Why not just have the functions exported as package level functions?

jefchien · 2025-03-28T19:20:56Z

receiver/awsebsnvmereceiver/internal/nvme/util.go

+
+package nvme
+
+type UtilInterface interface {


nit: In Go, Interface typically isn't included in interface names. The main purpose of an interface is to define a set of functions and use the interface to expose it. One common pattern is for the interface to be exported and the struct to not be.

type Util interface { } type util struct { }

jefchien · 2025-03-28T19:22:55Z

receiver/awsebsnvmereceiver/internal/nvme/util_stub.go

+import "errors"
+
+func (u *Util) GetAllDevices() ([]DeviceFileAttributes, error) {
+	return nil, errors.New("nvme stub: nvme not supported")


nit: Don't include the "file name" in the error.

Suggested change

return nil, errors.New("nvme stub: nvme not supported")

return nil, errors.New("nvme not supported")

jefchien · 2025-03-28T21:11:54Z

receiver/awsebsnvmereceiver/factory.go

+	)
+}
+
+func arrayToSet(arr []string) map[string]struct{} {


nit: We technically already have this function as part of https://github.com/aws/amazon-cloudwatch-agent/blob/main/internal/util/collections/collections.go#L103

jefchien · 2025-03-28T22:24:03Z

receiver/awsebsnvmereceiver/scraper.go

+		if foundWorkingDevice {
+			s.logger.Debug("emitted metrics for nvme device with controller id", zap.Int("controllerID", id))
+		} else {
+			s.logger.Info("unable to get metrics for nvme device with controller id", zap.Int("controllerID", id))


nit: Why is this an info log?

jefchien · 2025-03-28T22:38:30Z

receiver/awsebsnvmereceiver/internal/nvme/constants.go

+	NvmeDevicePrefix     = "nvme"
+	DevDirectoryPath     = "/dev"
+	NvmeSysDirectoryPath = "/sys/class/nvme"
+
+	EbsNvmeModelName = "Amazon Elastic Block Store"


nit: Most of these are only used within the package and should not be exported.

Suggested change

NvmeDevicePrefix = "nvme"

DevDirectoryPath = "/dev"

NvmeSysDirectoryPath = "/sys/class/nvme"

EbsNvmeModelName = "Amazon Elastic Block Store"

devicePrefix = "nvme"

devDirectoryPath = "/dev"

sysDirectoryPath = "/sys/class/nvme"

ebsModelName = "Amazon Elastic Block Store"

Even DevDirectoryPath could avoid being exported if you had a simple function like

func DevPath(device string) string { return filepath.Join(devDirectoryPath, device) }

jefchien · 2025-03-28T22:39:43Z

receiver/awsebsnvmereceiver/internal/nvme/util_unix.go

+
+	devices := []DeviceFileAttributes{}
+	for _, entry := range entries {
+		if strings.HasPrefix(entry.Name(), NvmeDevicePrefix) {


nit: Likely not an issue in this directory, but you could check entry.IsDir() first and ignore those.

jefchien · 2025-03-28T22:43:36Z

receiver/awsebsnvmereceiver/internal/nvme/ebs_metrics_unix.go

+var (
+	ErrInvalidEBSMagic = errors.New("invalid EBS magic number")
+	ErrParseLogPage    = errors.New("failed to parse log page")
+)


nit: If they aren't used outside of the package, don't export them.

jefchien · 2025-03-28T22:45:05Z

receiver/awsebsnvmereceiver/internal/nvme/ebs_metrics_unix.go

+func nvmeReadLogPage(fd uintptr, logID uint8) ([]byte, error) {
+	data := make([]byte, 4096) // 4096 bytes is the length of the log page.
+	bufferLen := len(data)
+
+	if bufferLen > math.MaxUint32 {
+		return nil, errors.New("nvmeReadLogPage: bufferLen exceeds MaxUint32")
+	}


I don't get this check. If we define the length of the slice, how would this ever return an error?

duhminick force-pushed the dominic-nvme-receiver branch from 66fb288 to 546fb3e Compare March 18, 2025 15:04

duhminick changed the base branch from main to ebs March 18, 2025 15:19

Add new awsebsnvmereceiver

be8e93b

duhminick force-pushed the dominic-nvme-receiver branch from 546fb3e to be8e93b Compare March 18, 2025 15:23

honor resources field

1ba325f

duhminick marked this pull request as ready for review March 18, 2025 19:39

duhminick requested a review from a team as a code owner March 18, 2025 19:39

duhminick changed the base branch from ebs to main March 18, 2025 19:40

duhminick added 2 commits March 18, 2025 19:42

remove stale comment

92aa89a

add resource filtering unit test

4b2fee5

lisguo reviewed Mar 24, 2025

View reviewed changes

use fmt.Sscanf to parse device file name

35b753a

duhminick added 2 commits March 25, 2025 17:38

only enable one metric by default

cf1c59e

rename resources to devices

9d4a053

lisguo reviewed Mar 26, 2025

View reviewed changes

lisguo approved these changes Mar 27, 2025

View reviewed changes

jefchien reviewed Mar 28, 2025

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add new awsebsnvmereceiver #1603

Add new awsebsnvmereceiver #1603

duhminick commented Mar 18, 2025 •

edited

Loading

lisguo commented Mar 25, 2025

duhminick commented Mar 25, 2025

duhminick commented Mar 25, 2025

lisguo left a comment

lisguo Mar 26, 2025

duhminick Mar 26, 2025 •

edited

Loading

jefchien Mar 28, 2025

lisguo Mar 26, 2025

duhminick Mar 26, 2025

jefchien Mar 28, 2025

duhminick Mar 28, 2025

jefchien Mar 28, 2025

jefchien Mar 28, 2025

jefchien Mar 28, 2025

jefchien Mar 28, 2025

jefchien Mar 28, 2025

jefchien Mar 28, 2025

jefchien Mar 28, 2025

jefchien Mar 28, 2025

jefchien Mar 28, 2025

jefchien Mar 28, 2025

	fmt.Sscanf(device, "nvme%dn%dp%d", &controller, &namespace, &partition)
	_, _ = fmt.Sscanf(device, "nvme%dn%dp%d", &controller, &namespace, &partition)

	if _, isAllowed := s.allowedDevices[deviceName]; !isAllowed {
	s.logger.Debug("skipping un-allowed device", zap.String("device", deviceName))
	continue
	}

	return nil, errors.New("nvme stub: nvme not supported")
	return nil, errors.New("nvme not supported")

Add new awsebsnvmereceiver #1603

Are you sure you want to change the base?

Add new awsebsnvmereceiver #1603

Conversation

duhminick commented Mar 18, 2025 • edited Loading

Note to Reviewers

Changes 2:

Description of the issue

Description of changes

License

Tests

Resource is explicitly empty

One device (nvme0) is in resources

* for resources

Sample Config

Requirements

lisguo commented Mar 25, 2025

duhminick commented Mar 25, 2025

duhminick commented Mar 25, 2025

lisguo left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

duhminick Mar 26, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

duhminick commented Mar 18, 2025 •

edited

Loading

`*` for resources

duhminick Mar 26, 2025 •

edited

Loading