README.md

Module 6: Stream Processing

Practical part: PyFlink

In this video, we show how to use PyFlink to consume data from a Kafka stream

• Code: PyFlink examples
• 2025 Homework

Theoretical part: Kafka + Java (optional)

In this set of videos, we cover Kafka and give examples in Java

Code: Java examples

Stream processing

• 🎥 6.0.1 Introduction
• 🎥 6.0.2 What is stream processing
• 🎥 6.3 What is kafka?
• 🎥 6.4 Confluent cloud
• 🎥 6.5 Kafka producer consumer
• 🎥 6.6 Kafka configuration

Links:

• Slides
• Kafka Configuration Reference
• Confluent cloud trial

Kafka Streams

• 🎥 6.7 Kafka stream basics
• 🎥 6.8 Kafka stream join
• 🎥 6.9 Kafka stream testing
• 🎥 6.10 Kafka stream windowing
• 🎥 6.11 Kafka ksqldb & Connect
• 🎥 6.12 Kafka Schema registry

Links:

• Slides
• Streams Concepts

PySpark - Structured Streaming (optional)

Code: Python examples

Please follow the steps described under pyspark-streaming

• 🎥 6.13 Kafka Streaming with Python
• 🎥 6.14 Pyspark Structured Streaming

Other streaming resources

Kafka Streams with JVM library

• Confluent Kafka Streams
• Scala Example

KSQL and ksqlDB

• Introducing KSQL: Streaming SQL for Apache Kafka
• ksqlDB

Kafka Connect

• Making Sense of Stream Data

Community notes

Did you take notes? You can share them here.

• Notes by Alvaro Navas
• Marcos Torregrosa's blog (spanish)
• Notes by Oscar Garcia
• 2024 videos transcript by Maria Fisher
• Notes by Shayan Shafiee Moghadam
• Add your notes here (above this line)