Skip to content

[Kafka]: Custom Kafka log integration with sasl mechanism SCRAM-SHA-256 #15525

@kesia-elastic

Description

@kesia-elastic

Integration Name

Kafka [kafka]

Dataset Name

No response

Integration Version

1.23.1

Agent Version

8.18

OS Version and Architecture

n.a.

User Goal

kafka_log (Custom Kafka Logs)
Would it be possible to Add a var to expose Filebeat kafka input option sasl.mechanism and wire it in the stream template?
Reason: Many Kafka clusters require SCRAM-SHA-256/512; the current package exposes username/password and Kerberos, but not the mechanism selector.

Existing Features

Docs: https://www.elastic.co/docs/reference/beats/filebeat/filebeat-input-kafka#_sasl_mechanism
Current manifest: packages/kafka_log/data_stream/generic/manifest.yml defines username and password, plus a kerberos block and ssl yaml.
Current template: packages/kafka_log/data_stream/generic/agent/stream/kafka.yml.hbs renders username, password, Kerberos, and ssl, but no sasl.mechanism.
EPR latest: kafka_log 1.8.1, kafka 1.23.1 (kafka is for broker logs/metrics; kafka_log is the generic topic reader).
Impact

Without sasl.mechanism, users connecting to brokers that enforce SCRAM must run Elastic Agent in standalone, fork the package, or request an internal override. This blocks common managed Kafka deployments.

What did you see?

Would it be possible to add a new var in packages/kafka_log/data_stream/generic/manifest.yml:
name: sasl.mechanism type: text title: SASL Mechanism description: "SASL mechanism to use. One of PLAIN, SCRAM-SHA-256, SCRAM-SHA-512. Leave empty to disable or when using Kerberos." required: false show_user: true
Wire the var in packages/kafka_log/data_stream/generic/agent/stream/kafka.yml.hbs:
{{#if sasl.mechanism}} sasl.mechanism: {{sasl.mechanism}} {{/if}}

Anything else?

No response

Metadata

Metadata

Labels

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions