Streaming Choices: Kafka, NATS, Pulsar, and RabbitMQ Examples

A single unified Maven project demonstrating message streaming patterns across four popular brokers: Kafka, NATS, Pulsar, and RabbitMQ.

All examples compile to a single JAR and run via a convenient Makefile interface, making it easy to switch between technologies and compare their APIs side-by-side.

---

Prerequisites

• Java 17+ (compiler release configured in pom.xml)
• Maven 3.8+ (mvn command available)
• Docker and Docker Compose (for running brokers)
• make (for the Makefile)

Verify your setup:

java -version
mvn -version
docker compose version

---

Quick Start

1. Build the project:

   make build

   Compiles all Java source files from kafka/src, nats/src, pulsar/src, and rabbitmq/src.

2. Start a broker (e.g., Kafka):

   make up TECHNOLOGY=kafka

3. Initialize the broker (create topics/streams as needed):

   make init TECHNOLOGY=kafka

4. Run a producer:

   make run TECHNOLOGY=kafka ROLE=producer

5. Run a consumer (in another terminal):

   make run TECHNOLOGY=kafka ROLE=consumer

6. Stop the broker:

   make down TECHNOLOGY=kafka

---

Makefile Targets

Core Commands

Target	Usage	Purpose
build	make build	Compile all source files (no test execution)
clean	make clean	Remove build artifacts
help	make help	Show this command reference

Broker Lifecycle

Target	Usage	Purpose
up	make up TECHNOLOGY=<tech>	Start Docker containers for the broker
down	make down TECHNOLOGY=<tech>	Stop and remove containers
init	make init TECHNOLOGY=<tech>	Create topics/streams in the broker
logs	make logs TECHNOLOGY=<tech>	Stream logs from running containers

Running Examples

Target	Usage	Purpose
run	make run TECHNOLOGY=<tech> ROLE=<role>	Compile and run the specified example

Note: run automatically calls build first, so you don't need to compile separately.

---

Technologies and Roles

Kafka

• Roles: producer, consumer
• Example commands:

  make run TECHNOLOGY=kafka ROLE=producer
  make run TECHNOLOGY=kafka ROLE=consumer

• Broker URL: localhost:9092
• Features in example:
  • Idempotent producer with async sends
  • Manual offset management in consumer
  • 3-partition topic creation

NATS

• Roles: core, producer, consumer
• Example commands:

  make run TECHNOLOGY=nats ROLE=core       # Core NATS pub/sub
  make run TECHNOLOGY=nats ROLE=producer   # JetStream producer
  make run TECHNOLOGY=nats ROLE=consumer   # JetStream push consumer

• Client URL: localhost:4222
• Monitoring UI: http://localhost:8222/jsz
• Features in example:
  • Core wildcard subscriptions and request/reply
  • JetStream (persistent) producer and consumer
  • Durable push consumer with explicit ACKing

Pulsar

• Roles: producer, consumer
• Example commands:

  make run TECHNOLOGY=pulsar ROLE=producer
  make run TECHNOLOGY=pulsar ROLE=consumer

• Binary Protocol: localhost:6650
• Admin UI: http://localhost:8081
• Features in example:
  • Message batching and LZ4 compression
  • Rich metadata (properties, keys)
  • Key-shared subscription type (per-key ordering)

RabbitMQ

• Roles: producer, consumer
• Example commands:

  make run TECHNOLOGY=rabbitmq ROLE=producer
  make run TECHNOLOGY=rabbitmq ROLE=consumer

• AMQP Port: localhost:5672
• Management UI: http://localhost:15672 (admin / secret)
• Features in example:
  • Topic exchange with routing key patterns (wildcard)
  • Quorum queues (Raft-replicated, exactly-once)
  • Publisher confirms and explicit message ACKing

---

Example Workflows

Test a Single Technology End-to-End

For Kafka:

# Terminal 1: Start broker and initialize
make up        TECHNOLOGY=kafka
make init      TECHNOLOGY=kafka

# Terminal 2: Run producer
make run       TECHNOLOGY=kafka ROLE=producer

# Terminal 3: Run consumer
make run       TECHNOLOGY=kafka ROLE=consumer

# When done:
make down      TECHNOLOGY=kafka

Switch Between Technologies

# Stop Kafka
make down TECHNOLOGY=kafka

# Start NATS
make up   TECHNOLOGY=nats
make init TECHNOLOGY=nats

# Run NATS JetStream producer
make run  TECHNOLOGY=nats ROLE=producer

# Run NATS JetStream consumer
make run  TECHNOLOGY=nats ROLE=consumer

Monitor Logs

# In one terminal, follow logs while broker is running
make logs TECHNOLOGY=pulsar

# In another terminal, run examples
make run  TECHNOLOGY=pulsar ROLE=producer

---

Project Structure

streaming-choices/
├── pom.xml                          # Root Maven build (all libs included)
├── Makefile                         # Unified interface for all examples
├── kafka/
│   ├── pom.xml                      # (deprecated; use root pom.xml)
│   ├── scripts/
│   │   ├── docker-compose-kafka.yaml
│   │   └── run.sh
│   └── src/
│       ├── OrderProducer.java
│       └── OrderConsumer.java
├── nats/
│   ├── pom.xml                      # (deprecated; use root pom.xml)
│   ├── scripts/
│   │   ├── docker-compose-nats.yaml
│   │   └── run.sh
│   └── src/
│       ├── NatsCoreExample.java
│       ├── NatsJetStreamProducer.java
│       └── NatsJetStreamPushConsumer.java
├── pulsar/
│   ├── pom.xml                      # (deprecated; use root pom.xml)
│   ├── scripts/
│   │   ├── docker-compose-pulsar.yaml
│   │   └── run.sh
│   └── src/
│       ├── PulsarOrderProducer.java
│       └── PulsarOrderConsumer.java
├── rabbitmq/
│   ├── pom.xml                      # (deprecated; use root pom.xml)
│   ├── scripts/
│   │   ├── docker-compose-rabbitmq.yaml
│   │   └── run.sh
│   └── src/
│       ├── RabbitMQTopology.java
│       ├── RabbitMQOrderProducer.java
│       └── RabbitMQOrderConsumer.java
└── README.md                        # This file

---

Building

The root pom.xml consolidates all dependencies and source paths. To build:

mvn clean compile

Or via Makefile:

make build

This produces compiled classes in target/classes/, which are then used by make run.

---

Troubleshooting

"Connection refused" when running examples

The broker is not yet ready. Wait a moment after make up:

make up TECHNOLOGY=kafka
sleep 5
make run TECHNOLOGY=kafka ROLE=producer

Broker crashes on startup

For RabbitMQ specifically, the image must be rabbitmq:3.12-management-alpine or later. The docker-compose-rabbitmq.yaml is already configured with this version.

"Unsupported TECHNOLOGY" error

Check that you are using a valid technology name:

make help

Valid options: kafka, nats, pulsar, rabbitmq

Port already in use

Stop the conflicting container:

docker ps                    # Find container names
docker stop <container>      # Stop it
make up TECHNOLOGY=kafka     # Try again

---

Per-Folder run.sh Scripts

Each technology folder also has a scripts/run.sh convenience wrapper that calls make up and make init. For example:

./kafka/scripts/run.sh      # Equivalent to: make up TECHNOLOGY=kafka && make init TECHNOLOGY=kafka

These wrappers work from any directory because they resolve the project root path.

---

Performance Notes

• Kafka: Batching is configured (5ms linger, 32KB batch size) for higher throughput. Idempotent producer for exactly-once semantics.
• NATS: JetStream push consumer for low-latency delivery. Async publishes with explicit acks.
• Pulsar: LZ4 compression enabled. Batching windows set to 5ms. Per-key ordering via key-shared subscription.
• RabbitMQ: Quorum queues (3-replica Raft) for high availability. Publisher confirms for delivery guarantees.

---

License

These examples are provided as-is for educational purposes.