[Feature Request]: Supports customization mapping addtional details to HTTP events

[AllanZhengYP]

Please provide your feature request

Our use case: when HTTP error happens, we want to find out the if the error is from requests to AWS service endpoints. If so, we want to collect the request IDs from the error response. So we can file tickets to the service team to investigate the service log if needed.

Gap: The request IDs are normally present in the response headers(e.g. x-amz-request-id). Currently the HTTP events does not have a way to enrich the event with information from headers.

Proposed solution: Provide a configuration to map request/response headers into additional attributes to HTTP events. Maybe we need to same for error events as well. Similarly to enrich the error events with addtional attributes from our Error classes.

Current workaround: Currently we have to replicate the CHR/Fetch/Error plugin implimentation but add addtional shimmings.

[qhanam]

Thanks for the clear and concise request! I think this would be a good feature and is needed. My intuition is that we could add a parameter to the HTTP plugin configuration.

@AllanZhengYP would you be interested in submitting a PR?

@williazz do you have any thoughts? Any risks?

[williazz]

Awesome idea. People have asked similar questions in the past about scraping information from the response.

Security wise, this puts greater strain on the Shared Responsibility Model so we have shied away from these ideas in the past. But I think it's doable because we have a very good reason.

If you are interested in submitting a PR, I would suggest limiting the scope to x-amz-request-id, but keep it open for extension in the future.

[AllanZhengYP]

@williazz Thanks for the quick follow up. We do have a few use cases where we do need to collect the request & response headers beyond just x-amz-request-id. Does CWR team open to a more flexible and open interface to map other headers?

Understandably, the shared responsibility model discourages the interfaces that could potentially be used incorrectly. What do you think a configuration only sets addtional boolean attribute if a given header is set?

[williazz]

Technically, RUM service team is only responsible for the security of the cloud. We are not responsible for deciding what data users want to log.

Why don't you just give it a go with the headers you are interested in? Users are always free to fork this repo and scrape whatever data they want as custom events, so this decision should not block you.

[AllanZhengYP]

There are a couple of problems with the custom events from where I can see:

1. The custom events does not show up many of the RUM dashboard tabs, e.g. HTTP Requests, User journey.
2. We can enble both the native http events(to populate the RUM dashboard), and custom HTTP events with addtional info we care. But it should double to number the events. This may also bring nagative impact to the performance since we are shimming the XHR in 2 layers.