[OpAMP] OpAMP Supervisor Integration in Fleet Server for EDOT Collector Compatibility

[Maxime-CLS]

OpAMP Supervisor Integration in Fleet Server for EDOT Collector Compatibility

Summary

This feature request proposes integrating OpAMP Supervisor capabilities into Fleet Server to enable native management of EDOT (Elastic Distribution of OpenTelemetry) collectors through Elastic's Fleet management interface.

Background

Currently, Fleet Server excels at managing Elastic Agents but lacks native support for EDOT collectors. With the growing adoption of EDOT collectors in enterprise environments, there's a clear need to bridge this gap and provide unified observability agent management.

The OpAMP (Open Agent Management Protocol) Supervisor is a separate binary that manages OpenTelemetry Collector instances via an OpAMP client, handling configuration merging (remote + local), additional binary downloads, and OpAMP server implementation for collector communication.

Problem Statement

Current Challenges

• Management Fragmentation: Organizations must use separate tools to manage Elastic Agents and EDOT collectors
• Operational Complexity: Different configuration workflows and deployment processes for different agent types
• Configuration Drift: Inconsistent configuration management approaches lead to operational overhead

User Pain Points

• DevOps teams need to maintain multiple agent management systems
• Lack of centralized policy enforcement for EDOT collectors
• Difficult troubleshooting across heterogeneous agent environments
• Complex rollout procedures for configuration updates

Proposed Solution

Integrate OpAMP Supervisor functionality directly into Fleet Server to enable:

Core Capabilities

• Unified Agent Management: Manage both Elastic Agents and EDOT collectors from a single Fleet interface
• Configuration Management: Centralized configuration distribution for EDOT collectors
• Binary Distribution: Collector binary management and updates through Fleet's existing infrastructure

Technical Approach

┌─────────────────┐    OpAMP Protocol    ┌──────────────────────┐
│  EDOT Collector │ ◄─────────────────── │   Fleet Server       │
│                 │                      │ (with OpAMP Supervisor) │
└─────────────────┘                      └──────────────────────┘
                                                    │
                                                    │ Fleet Protocol
                                                    ▼
                                         ┌──────────────────────┐
                                         │    Kibana Fleet UI   │
                                         └──────────────────────┘

Detailed Requirements

1. OpAMP Server Integration

• Embed OpAMP Supervisor capabilities within Fleet Server

2. Configuration Management

• Remote Configuration: Push collector configurations from Kibana Fleet UI
• Configuration Merging: Support local and remote configuration combination
• Validation: Validate EDOT collector configurations before deployment
• Versioning: Track configuration versions and enable rollback capabilities

3. Binary Management

• Collector Binaries: Download and distribute EDOT collector binaries
• Additional Components: Support for processor, receiver, and exporter plugins
• Version Management: Handle collector version updates and compatibility

4. Fleet UI Integration

• Policy Management: Create and manage policies for OpenTelemetry collectors
• Configuration Editor: YAML/JSON editor for EDOT collector configurations

Benefits

Operational Benefits

• Unified Management: Single pane of glass for all observability agents
• Reduced Complexity: Simplified deployment and management workflows
• Consistent Policies: Centralized policy enforcement across agent types
• Streamlined Updates: Coordinated updates for all observability components

Technical Benefits

• Scalability: Leverage Fleet Server's proven scalability for collector management
• Reliability: Built-in high availability and disaster recovery capabilities
• Integration: Seamless integration with existing Elastic Stack deployments
• Standards Compliance: Native support for OpenTelemetry and OpAMP standards

Business Benefits

• Future-Proofing: Alignment with OpenTelemetry ecosystem evolution

Community Impact

This feature would:

• Demonstrate commitment to open standards (OpenTelemetry/OpAMP)
• Foster community adoption through enhanced flexibility
  Target Release: [TBD based on roadmap prioritization]

[cmacknz]

Currently, Fleet Server excels at managing Elastic Agents but lacks native support for EDOT collectors

You can centrally manage Elastic Agent (which is EDOT collector) with Fleet which supports OpenTelemetry integrations to configure the collector. This happens with our proprietary Fleet protocol and not with OpAMP right now.

Conceptually the Elastic Agent architecture is very close to the OpAMP architecture, where elastic-agent is the supervisor and elastic-otel-collector (EDOT collector) is the collector. We will gradually align more with opamp for collector management (see elastic/elastic-agent#15797 for example which is about how we supervise elastic-otel-collector) but we already have central management of our own collector.

OpAMP at the protocol level between agent and fleet server would bring in compatibility with upstream and 3rd party collectors. This is not currently prioritized but the architecture is capable of implementing it one day.

We are very unlikely to include opampsupervisor directly in it's current form anytime soon as we will always need a superset of it (to for example manage the Elastic Defend service) but as it becomes more extendable we may be able to embed it into elastic-agent.

TL;DR:

• If you are asking for OpAMP to centrally manage EDOT collector, just use Elastic Agent. It already supports this and is functionally equivalent to a supervised EDOT collector.
• If you are asking for OpAMP for compatibility with non-Elastic collectors, this is not yet prioritized.

[Maxime-CLS]

Thank you very much for this detailed and insightful response. I greatly appreciate the clarification regarding the current Elastic Agent architecture.

Acknowledgment of Points Raised

You are absolutely right to highlight that Elastic Agent already functions as a supervisor for the EDOT collector, and I now have a much better understanding of the existing architectural alignment. The reference to issue #15797 clearly shows the direction being taken to improve supervision of the elastic-otel-collector.

Organizational Context and Constraints

To clarify the context of my request: it stems from specific architectural constraints within our enterprise environment. Our architecture teams are seeking to maximize interoperability and alignment with OpenTelemetry standards in our global observability strategy.

The objective is to be able to manage the EDOT collector via standard OpAMP protocols, which would enable us to achieve better integration with our existing multi-vendor ecosystem and unified governance of OpenTelemetry collectors.

Desired Benefits

This approach would provide us with:

• Process Standardization: Unified protocols for all our OpenTelemetry collectors
• Operational Simplification: Homogeneous workflows for DevOps teams
• Standards Compliance: Complete alignment with OpenTelemetry/OpAMP specifications
• Integration Flexibility: Native compatibility with our multi-vendor monitoring stack

Collaborative Perspective

We fully recognize the value of the Elastic ecosystem and wish to continue using EDOT. This request is driven by a complementary approach rather than replacement, aiming to leverage Elastic's strengths while respecting our enterprise architecture constraints.

Would there be a possibility to consider this evolution in the medium/long-term roadmap? This would actually reinforce Elastic's commitment to the OpenTelemetry ecosystem.

Thank you again for this constructive discussion.