klag-exporter

A high-performance Apache Kafka® consumer group lag exporter written in Rust. Calculates both offset lag and time lag (latency in seconds) with accurate timestamp-based measurements.

Features

Accurate Time Lag Calculation — Directly reads message timestamps from Kafka® partitions instead of interpolating from lookup tables
Dual Export Support — Native Prometheus HTTP endpoint (/metrics) and OpenTelemetry OTLP export
Non-blocking Scrapes — Continuous background collection with instant metric reads (no Kafka® calls during Prometheus scrapes)
Multi-cluster Support — Monitor multiple Kafka® clusters with independent collection loops and failure isolation
Flexible Filtering — Regex-based whitelist/blacklist for consumer groups and topics
Configurable Granularity — Topic-level (reduced cardinality) or partition-level metrics
Custom Labels — Add environment, datacenter, or any custom labels to all metrics
Full Authentication Support — SASL/PLAIN, SASL/SCRAM, SSL/TLS, and Kerberos via librdkafka
Production Ready — Health (/health) and readiness (/ready) endpoints for Kubernetes deployments
Resource Efficient — Written in Rust with async/await, minimal memory footprint, and bounded concurrency

Why klag-exporter?

Comparison with Existing Solutions

Feature	klag-exporter	kafka-lag-exporter	KMinion
Language	Rust	Scala (JVM)	Go
Time Lag	Direct timestamp reading	Interpolation from lookup table	Offset-only (requires PromQL)
Idle Producer Handling	Shows actual message age	Shows 0 (interpolation fails)	N/A
Memory Usage	~20-50 MB	~200-500 MB (JVM)	~50-100 MB
Startup Time	< 1 second	5-15 seconds (JVM warmup)	< 1 second
OpenTelemetry	Native OTLP	No	No
Blocking Scrapes	No	No	Yes

Time Lag Accuracy

Problem with interpolation (kafka-lag-exporter):

Builds a lookup table of (offset, timestamp) pairs over time
Interpolates to estimate lag — breaks when producers stop sending
Shows 0 lag incorrectly for idle topics
Requires many poll cycles to build accurate tables

Our approach (targeted timestamp sampling):

Seeks directly to the consumer group's committed offset
Reads actual message timestamp — always accurate
Handles idle producers correctly (shows true message age)
TTL-cached to prevent excessive broker load

Quick Start

Using Docker

docker run -d \
  -p 8000:8000 \
  -v $(pwd)/config.toml:/etc/klag-exporter/config.toml \
  klag-exporter:latest \
  --config /etc/klag-exporter/config.toml

Using Binary

# Build from source
cargo build --release

# Run with config file
./target/release/klag-exporter --config config.toml

# Run with debug logging
./target/release/klag-exporter -c config.toml -l debug

Using Docker Compose (Demo Stack)

A complete demo environment with Kafka®, Prometheus, and Grafana is available in the test-stack/ directory:

cd test-stack
docker-compose up -d --build

# Access points:
# - Grafana:       http://localhost:3000 (admin/admin)
# - Prometheus:    http://localhost:9090
# - klag-exporter: http://localhost:8000/metrics
# - Kafka UI:      http://localhost:8080

Configuration

Create a config.toml file:

[exporter]
poll_interval = "30s"
http_port = 8000
http_host = "0.0.0.0"
granularity = "partition"  # "topic" or "partition"

[exporter.timestamp_sampling]
enabled = true
cache_ttl = "60s"
max_concurrent_fetches = 10

[exporter.otel]
enabled = false
endpoint = "http://localhost:4317"
export_interval = "60s"

[[clusters]]
name = "production"
bootstrap_servers = "kafka1:9092,kafka2:9092"
group_whitelist = [".*"]
group_blacklist = []
topic_whitelist = [".*"]
topic_blacklist = ["__.*"]  # Exclude internal topics

[clusters.consumer_properties]
"security.protocol" = "SASL_SSL"
"sasl.mechanism" = "PLAIN"
"sasl.username" = "${KAFKA_USER}"
"sasl.password" = "${KAFKA_PASSWORD}"

[clusters.labels]
environment = "production"
datacenter = "us-east-1"

Environment Variable Substitution

Use ${VAR_NAME} syntax in config values. The exporter will substitute with environment variable values at startup.

Metrics

Partition Offset Metrics

Metric	Labels	Description
`kafka_partition_latest_offset`	cluster_name, topic, partition	High watermark offset
`kafka_partition_earliest_offset`	cluster_name, topic, partition	Low watermark offset

Consumer Group Metrics (Partition Level)

Metric	Labels	Description
`kafka_consumergroup_group_offset`	cluster_name, group, topic, partition, member_host, consumer_id, client_id	Committed offset
`kafka_consumergroup_group_lag`	cluster_name, group, topic, partition, member_host, consumer_id, client_id	Offset lag
`kafka_consumergroup_group_lag_seconds`	cluster_name, group, topic, partition, member_host, consumer_id, client_id	Time lag in seconds

Consumer Group Aggregate Metrics

Metric	Labels	Description
`kafka_consumergroup_group_max_lag`	cluster_name, group	Max offset lag across partitions
`kafka_consumergroup_group_max_lag_seconds`	cluster_name, group	Max time lag across partitions
`kafka_consumergroup_group_sum_lag`	cluster_name, group	Sum of offset lag
`kafka_consumergroup_group_topic_sum_lag`	cluster_name, group, topic	Sum of offset lag per topic

Operational Metrics

Metric	Labels	Description
`kafka_consumergroup_poll_time_ms`	cluster_name	Time to poll all offsets
`kafka_lag_exporter_scrape_duration_seconds`	cluster_name	Collection cycle duration
`kafka_lag_exporter_up`	—	1 if healthy, 0 otherwise

HTTP Endpoints

Endpoint	Description
`GET /metrics`	Prometheus metrics
`GET /health`	Liveness probe (always 200 if running)
`GET /ready`	Readiness probe (200 when metrics available)
`GET /`	Basic info page

Architecture

┌─────────────────────────────────────────────────────────────────────────────┐
│                              Main Application                                │
│  ┌──────────────┐  ┌──────────────────┐  ┌──────────────────────────────┐  │
│  │   Config     │  │   HTTP Server    │  │      Metrics Registry        │  │
│  │   Loader     │  │  (Prometheus +   │  │   (In-memory Gauge Store)    │  │
│  │              │  │   Health)        │  │                              │  │
│  └──────────────┘  └──────────────────┘  └──────────────────────────────┘  │
│                                                         ▲                   │
│  ┌──────────────────────────────────────────────────────┴────────────────┐  │
│  │                      Cluster Manager (per cluster)                    │  │
│  │  ┌─────────────────┐  ┌─────────────────┐  ┌─────────────────────┐   │  │
│  │  │  Offset         │  │  Timestamp      │  │  Metrics            │   │  │
│  │  │  Collector      │  │  Sampler        │  │  Calculator         │   │  │
│  │  │  (Admin API)    │  │  (Consumer)     │  │                     │   │  │
│  │  └─────────────────┘  └─────────────────┘  └─────────────────────┘   │  │
│  └───────────────────────────────────────────────────────────────────────┘  │
│                                                                             │
│  ┌───────────────────────────────────────────────────────────────────────┐  │
│  │                      Export Layer                                     │  │
│  │  ┌─────────────────────────┐  ┌─────────────────────────────────┐    │  │
│  │  │  Prometheus Exporter    │  │  OpenTelemetry Exporter         │    │  │
│  │  │  (HTTP /metrics)        │  │  (OTLP gRPC)                    │    │  │
│  │  └─────────────────────────┘  └─────────────────────────────────┘    │  │
│  └───────────────────────────────────────────────────────────────────────┘  │
└─────────────────────────────────────────────────────────────────────────────┘

Building from Source

Prerequisites

Rust 1.78 or later
CMake (for librdkafka)
OpenSSL development libraries
SASL development libraries (optional, for Kerberos)

Build

# Debug build
cargo build

# Release build (optimized)
cargo build --release

# Run tests
cargo test

# Run linter
cargo clippy

Docker Build

docker build -t klag-exporter:latest .

Kubernetes Deployment

apiVersion: apps/v1
kind: Deployment
metadata:
  name: klag-exporter
spec:
  replicas: 1
  selector:
    matchLabels:
      app: klag-exporter
  template:
    metadata:
      labels:
        app: klag-exporter
      annotations:
        prometheus.io/scrape: "true"
        prometheus.io/port: "8000"
        prometheus.io/path: "/metrics"
    spec:
      containers:
      - name: klag-exporter
        image: klag-exporter:latest
        ports:
        - containerPort: 8000
        livenessProbe:
          httpGet:
            path: /health
            port: 8000
          initialDelaySeconds: 5
          periodSeconds: 10
        readinessProbe:
          httpGet:
            path: /ready
            port: 8000
          initialDelaySeconds: 5
          periodSeconds: 10
        volumeMounts:
        - name: config
          mountPath: /etc/klag-exporter
        env:
        - name: KAFKA_USER
          valueFrom:
            secretKeyRef:
              name: kafka-credentials
              key: username
        - name: KAFKA_PASSWORD
          valueFrom:
            secretKeyRef:
              name: kafka-credentials
              key: password
      volumes:
      - name: config
        configMap:
          name: klag-exporter-config
---
apiVersion: v1
kind: Service
metadata:
  name: klag-exporter
  labels:
    app: klag-exporter
spec:
  ports:
  - port: 8000
    targetPort: 8000
    name: metrics
  selector:
    app: klag-exporter

Grafana Dashboard

A pre-built Grafana dashboard is included in test-stack/grafana/provisioning/dashboards/kafka-lag.json with:

Total consumer lag overview
Max time lag per consumer group
Per-partition lag breakdown
Offset progression over time
Message rate calculations
Exporter health status

Security Considerations

Timestamp Sampling and Message Content

To calculate accurate time lag, klag-exporter fetches messages from Kafka® at the consumer group's committed offset to read the message timestamp. Only the timestamp metadata is extracted — the message payload (key and value) is never read, logged, or stored.

Risk	Level	Notes
Data exposure in logs	None	Only topic/partition/offset/timestamp logged, never payload
Data in memory	Low	Payload briefly in process memory (~ms), then dropped
Data exfiltration	None	Payload never sent, stored, or exposed via API
Network exposure	Same as any consumer	Use TLS (`security.protocol=SSL`) for encryption

For sensitive environments:

Disable timestamp sampling with timestamp_sampling.enabled = false — you'll still get offset lag metrics
Fetch size is limited to 256KB per partition to minimize data transfer
Run klag-exporter with minimal privileges and restricted network access

Troubleshooting

Time Lag Shows Gaps in Grafana

This is expected when:

Consumer catches up completely (lag = 0)
Timestamp cache expires and refetch is in progress
Kafka® fetch times out

Solutions:

Increase cache_ttl in config
Use Grafana's "Connect null values" option
For alerting, use avg_over_time() or last_over_time()

High Memory Usage

Reduce max_concurrent_fetches
Use granularity = "topic" instead of "partition"
Add more restrictive group_blacklist / topic_blacklist patterns

Connection Errors

Verify bootstrap_servers are reachable
Check authentication configuration in consumer_properties
Ensure network policies allow connections to Kafka® brokers

License

MIT License — see LICENSE for details.

Contributing

Contributions are welcome! Please open an issue or submit a pull request.

Trademark Notice

Apache®, Apache Kafka®, and Kafka® are either registered trademarks or trademarks of the Apache Software Foundation in the United States and/or other countries. This project is not affiliated with, endorsed by, or sponsored by the Apache Software Foundation. For more information about Apache trademarks, please see the Apache Trademark Policy.

Name		Name	Last commit message	Last commit date
Latest commit History 24 Commits
docs/images		docs/images
src		src
test-stack		test-stack
.gitignore		.gitignore
Cargo.lock		Cargo.lock
Cargo.toml		Cargo.toml
Dockerfile		Dockerfile
LICENSE		LICENSE
README.md		README.md
config.example.toml		config.example.toml

License

softwaremill/klag-exporter

Folders and files

Latest commit

History

Repository files navigation

klag-exporter

Features

Why klag-exporter?

Comparison with Existing Solutions

Time Lag Accuracy

Quick Start

Using Docker

Using Binary

Using Docker Compose (Demo Stack)

Configuration

Environment Variable Substitution

Metrics

Partition Offset Metrics

Consumer Group Metrics (Partition Level)

Consumer Group Aggregate Metrics

Operational Metrics

HTTP Endpoints

Architecture

Building from Source

Prerequisites

Build

Docker Build

Kubernetes Deployment

Grafana Dashboard

Security Considerations

Timestamp Sampling and Message Content

Troubleshooting

Time Lag Shows Gaps in Grafana

High Memory Usage

Connection Errors

License

Contributing

Trademark Notice

About

Resources

License

Uh oh!

Stars

Watchers

Forks

Releases

Packages 0

Uh oh!

Languages

Packages