Support Go 1.25 FlightRecorder API

[wshirey]

Package Name

profiler

Package Version(s)

No response

Describe the feature you'd like

Go 1.25 has introduce a FlightRecorder which is similar to Datadog's continuous profiler but provides developers a way to manually trigger profile capture and persistence. I would like to request that Datadog support this interface and provide a io.Writer that we could wire up to the FlightRecorder. Currently I am sending profiles to Google Cloud Storage for offline analysis but it would be great if we could send to Datadog as well.

For example, I would like to to send a profile when a an API requests exceeds a duration to further troubleshoot. This blog article illustrates that example.

Is your feature request related to a problem?

Yes, being able to manually trigger and send profiles from application events.

Describe alternatives you've considered

Uploading profiles to Google Cloud Storage for offline analysis. This lacks integration with other Datadog tooling and observability.

Additional context

No response

[nsrip-dd]

👋 Profiling maintainer here. We're definitely interested in supporting the flight recorder to get more targeted execution traces.

Are you familiar with our existing execution trace support? It's under the "timeline" view in the Datadog continous profile UI, as well as in the "profiles" tab in the APM trace UI when you inspect a span. See this blog post for more info. We use the same data (execution traces) that the flight recorder produces. The big difference is that we take a recording using the old API, once every 15 minutes on average, and up to 5MiB per recording. We do this due to data volume concerns. The hope with the new flight recording API is that we could make the data collection more targeted towards things you care about, and less random.

I bring this up to preface this part of your request: "provide a io.Writer that we could wire up to the FlightRecorder". I had pictured that we would support the flight recorder by using it internally in the profiler library. And then we would provide some interface, like a function you can call to request a flight recording. Would that work for you? Or would you prefer to supply your own flight recordings for our library to upload?

[wshirey]

Wow, @nsrip-dd, I did not expect a response this soon, especially on a Monday morning, so thanks for that!

I did see that blog post but like you mentioned, it's not guaranteed to capture profiling of a specific application event. Is there more information about what criteria needs to be satisfied for the automatic profile to be captured and sent to DD?

While I would prefer the io.Writer approach so that we could profile locally and swap out io.Writer implementations, I understand there might be additional metadata you might need to collect to correlate other observability data with the trace. So I think I would be open to either implementation.

Do you have an estimated timeline when we might see this feature added?

[nsrip-dd]

Wow, @nsrip-dd, I did not expect a response this soon, especially on a Monday morning, so thanks for that!

My pleasure! I'm excited to see somebody interested in the flight recorder.

Is there more information about what criteria needs to be satisfied for the automatic profile to be captured and sent to DD?

Right now, the only requirement is to have this profiler library enabled. We have a few knobs to configure how often an execution trace is captured (DD_PROFILING_EXECUTION_TRACE_PERIOD) and how big it is (DD_PROFILING_EXECUTION_TRACE_LIMIT_BYTES). But the timing is still randomized and at the moment we don't provide a way to control it.

Do you have an estimated timeline when we might see this feature added?

I don't have an estimated timeline right now. Would you be interested in trying out experimental changes for this? i.e. running with a branch of this library, perhaps on a staging/test service, using an API that might not be the final API we settle on? We've floated ideas internally but it would be great to get some hands-on experience with an API.

[wshirey]

Yes if there was a way to opt in we would definitely be interested in being guinea pigs!