[OpAMP] Map fields from AgentToServer message to Agent fields

[juliaElastic]

What is the problem this PR solves?

Map data from OpAMP AgentToServer message to Agent fields in .fleet-agents

How does this PR solve the problem?

Add OpAMP message fields to Agent document:

• convert capabilities to string array
• convert effective config to json object and sanitize
• add health and set last_checkin_status, last_checkin_message, etc.
• add identifying and non-identifying attributes
• add sequence number

How to test this PR locally

Follow instructions in https://github.com/elastic/fleet-server/blob/main/docs/opamp.md

Download and extract otel collector, e.g.:

curl -L -O https://github.com/open-telemetry/opentelemetry-collector-releases/releases/download/v0.144.0/otelcol-contrib_0.144.0_darwin_arm64.tar.gz 

Create otel config to include system fields and internal telemetry, e.g. otel-opamp.yaml:

receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317

processors:
  resourcedetection:
    detectors: ["system","env"]
    system:
      hostname_sources: ["os"]
      resource_attributes:
        host.name:
          enabled: true
        host.arch:
          enabled: true
        os.description:
          enabled: true
        os.type:
          enabled: true

exporters:
  debug:
    verbosity: detailed
  elasticsearch/otel:
    endpoints: [ "http://localhost:9200" ]
    api_key: ${env:ES_API_KEY} 
    mapping:
      mode: otel
  otlp:
    endpoint: "http://localhost:4317"
    tls:
      insecure: true

extensions:
  opamp:
    server:
      http:
        endpoint: http://localhost:8220/v1/opamp
        tls:
          insecure: true
        headers:
          Authorization: ApiKey ${env:FLEET_ENROLLMENT_TOKEN}
    instance_uid: ${env:INSTANCE_UID}
    capabilities:
      reports_effective_config: true

service:
  pipelines:
    logs:
      receivers: [otlp]
      exporters: [debug]
    metrics:
      receivers: [otlp]
      processors: [resourcedetection]
      exporters: [elasticsearch/otel]
  extensions: [opamp]

  # publish collector internal telemetry
  telemetry:
    metrics:
      level: detailed
      readers:
        - periodic:
            interval: 30000
            exporter:
              otlp:
                protocol: grpc
                endpoint: http://localhost:4317

Create API keys and start otel collector:

 cd ~/Downloads/otelcol-contrib_0.144.0_darwin_arm64
 
 export INSTANCE_UID=<uuid> # e.g. "519b8d7a-2da8-7657-b52d-492a9de33313"
 export OTEL_RESOURCE_ATTRIBUTES="service.instance.id=$INSTANCE_UID" # to include instance id in internal telemetry data
 export ES_API_KEY=<api_key> # ES API key from observability onboarding UI
 export FLEET_ENROLLMENT_TOKEN=<enrollment_token> 
 ./otelcol-contrib --config ./otel-opamp.yaml

Design Checklist

• I have ensured my design is stateless and will work when multiple fleet-server instances are behind a load balancer.
• I have or intend to scale test my changes, ensuring it will work reliably with 100K+ agents connected.
• I have included fail safe mechanisms to limit the load on fleet-server: rate limiting, circuit breakers, caching, load shedding, etc.

Checklist

• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation
• I have made corresponding change to the default configuration files
• I have added tests that prove my fix is effective or that my feature works
• I have added an entry in ./changelog/fragments using the changelog tool

Related issues

Closes https://github.com/elastic/ingest-dev/issues/6982

[mergify]

This pull request does not have a backport label. Could you fix it @juliaElastic? 🙏
To fixup this pull request, you need to add the backport labels for the needed
branches, such as:

• backport-./d./d is the label to automatically backport to the 8./d branch. /d is the digit
• backport-active-all is the label that automatically backports to all active branches.
• backport-active-8 is the label that automatically backports to all active minor branches for the 8 major.
• backport-active-9 is the label that automatically backports to all active minor branches for the 9 major.

[cmacknz]

Depending on merge order, the test in #6289 will have to be updated not to expect the updating state anymore.

[ycombinator]

LGTM!

[juliaElastic]

@ycombinator Could you help merge the PR, it seems I no longer have access.

[michel-laterman]

Can we compare to es.ErrIndexNotFound here instead?

[juliaElastic]

done

[michel-laterman]

qq: the policy on fleet will never have a policy id > 1, if it can then agents can appear as outdated

[juliaElastic]

The UI hides the policy name and revision for collectors, as policies are not relevant for them (other than needing an enrollment token to authenticate with).

[juliaElastic]

removed the status field check from the agent doc, it's calculated by a runtime field by Fleet UI