Initial

.gitignore

@@ -38,3 +38,6 @@ classes/
 # MacOS
 .DS_Store
 
+# config files
+vertx-proxy/src/main/resources/config.json
+

README.md

@@ -6,26 +6,17 @@
 
 
 The goal of this project is to provide proxy-based, topic-level encryption-at-rest for [Apache Kafka®](https://kafka.apache.org/).  
-To learn more about the background and architecture of topic encryption, see our [overview document](doc/README.md).
 
-The next planned milestones in the project are:
+### Documentation
+To learn more about the background and architecture of topic encryption, see our [overview document](doc/README.md). 
 
-## M1, May 14: Foundation
-- Technical specification of the project
-- Assessment of viable proxy 
-  - Envoy vs. a custom-developed proxy (in golang or Java)
+The [getting started guide](doc/getting-started.md) explains how to compile and run the encrypting proxy for testing and evaluation.
 
-## M2, June 04: Alpha proxy
-- Initial implementation of selected proxy architecture
-  - stand-alone, not yet integrated
+### Project structure
+The project consists of two nested projects:
+- [encmod](encmod/),  the topic encryption module
+- [vertx-proxy](vertx-proxy/), an experimental Kafka proxy for developing and testing the encryption module. 
 
-## M3, June 18: Proxy integration evaluation
-- First version of the software encryption module
-- Integration of encryption module with proxy
-- Evaluation of proxy integration into Strimzi and build environment
 
-## M4, July 02: Alpha Strimzi integration
-- Integrate proxy with the Strimzi project
-- Integrate encryption module
 
 

common/pom.xml

@@ -0,0 +1,6 @@
+<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 https://maven.apache.org/xsd/maven-4.0.0.xsd">
+  <modelVersion>4.0.0</modelVersion>
+  <groupId>strimzi.io</groupId>
+  <artifactId>common</artifactId>
+  <version>0.0.1-SNAPSHOT</version>
+</project>

doc/README.md

@@ -4,7 +4,7 @@
 
 # Proxy-Based Topic-level Encryption at Rest for Kakfka
 
-The goal of this project is to provide proxy-based, topic-level encryption-at-rest for [Apache Kafka](https://kafka.apache.org/). 
+The goal of this project is to provide proxy-based, topic-level encryption-at-rest for [Apache Kafka](https://kafka.apache.org/).  This document provides an overview of the motivation and architecture of the encrypting proxy. For more details, see the [references below](#references) for links to our research paper [1] and project proposal [2].
 
 Although Kafka provides multiple authentication methods and encrypted communication over [TLS](https://en.wikipedia.org/wiki/Transport_Layer_Security), it does not encrypt data at rest.
 Yet Kafka is increasingly used as a store of data, not just as a means of transferring data from one location to another. 
@@ -28,12 +28,13 @@ The diagram below depicts the main components of the proposal, illustrating clie
 One core component, the _Encryption Module_, provides the encryption functionality. 
 A second core component, the _Proxy_, intercepts Kafka connections and delegates message processing to the Encryption Module.
 
-Topic can be encrypted by different keys, allowing brokers to store a mix of encrypted and unencrypted data, with data owners managing the keys to their topics. 
+Topics can be encrypted by different keys, allowing brokers to store a mix of encrypted and unencrypted data, with data owners managing the keys to their topics. 
 Keys will be stored in an external key management system with access policies and logging.
 
-In the coming weeks we will be providing the specification for the core components along with a roadmap. 
 We look forward to engaging with the Community in developing this exciting extension to Strimzi and Kafka!
 
-P.S. The original [Strimzi proposal #17](https://github.com/strimzi/proposals/blob/master/017-kafka-topic-encryption.md) provides additional background.
+## References
+1. [Securing Kafka with Encryption-at-Rest](https://ieeexplore.ieee.org/abstract/document/9671388/), Chris Giblin, Sean Rooney, Pascal Vetsch, and Adrian Preston, 2021 IEEE International Conference on Big Data (Big Data)
+2. The original [Strimzi proposal #17](https://github.com/strimzi/proposals/blob/master/017-kafka-topic-encryption.md) provides additional background.
 
 

doc/getting-started.md

@@ -0,0 +1,125 @@
+# Getting started
+
+Requirements:
+- a Kafka instance, version 2.8.0 or older, which you can configure
+- Java 17
+- Apache maven installed in your command line environment
+- git command
+
+
+The steps for getting started with this initial version of topic encryption are outlined below:
+1. Clone the repository and set your working path.
+2. Compile
+3. Configure the Kafka advertised address
+4. Configure the proxy
+5. Run the proxy
+6. Start kafka
+7. Run kafka clients
+
+Each of these steps is described in detail below with an example.
+
+## Scenario
+
+In the scenario to get started, all components run on the same system, `localhost`.  The Kafka broker can also run remotely. The minimum requirement is that one can update the broker configuration file and restart the broker. In this example, however, we run the broker locally.
+
+The proxy will listen on port 1234 and the broker listens on its standard port 9092 as depicted below:
+
+```
+ Kafka client        Proxy         Kafka broker
+      o------------o 1234 o------------o 9092
+```
+
+The clients are reconfigured to use port 1234 (details below).
+
+A policy to encrypt all topics with the same key, along with a test key management system (KMS) which returns a hard-coded AES key, is used.
+
+The following sections provide details for each step in running the encrypting proxy.
+
+### 1. Clone the repository and set your working path
+```
+git clone [email protected]:strimzi/topic-encryption.git
+cd topic-encryption
+```
+
+### 2. Compile
+
+```
+mvn install
+```
+
+### 3. Configure the Kafka broker's listeners
+The address advertised by Kafka must be that of the proxy, not the broker itself.
+
+Modify the `advertised.listeners` property in `$KAFKA_HOME/config/server.properties` to point to the proxy host and port, as shown in the snippet below:
+
+```
+# The address the socket server listens on. It will get the value returned from 
+# java.net.InetAddress.getCanonicalHostName() if not configured.
+#   FORMAT:
+#     listeners = listener_name://host_name:port
+#   EXAMPLE:
+#     listeners = PLAINTEXT://your.host.name:9092
+listeners=PLAINTEXT://:9092
+
+# Hostname and port the broker will advertise to producers and consumers. If not set, 
+# it uses the value for "listeners" if configured.  Otherwise, it will use the value
+# returned from java.net.InetAddress.getCanonicalHostName().
+advertised.listeners=PLAINTEXT://127.0.0.1:1234
+```
+Stop the Kafka broker and start it after the proxy is running.
+
+### 4. Configure the proxy
+Set the working directory to the proxy's target folder:
+```
+$ cd vertx-proxy/target/
+```
+
+Create a configuration file, `config.json` and add the following JSON contents:
+
+```
+{
+  "listening_port" : 1234,
+  "kafka_broker"   : "localhost:9092",
+  "policy_repo"    : "test"
+}
+```
+### 5. Run the proxy
+With the current path set to the target directory, run the proxy with the following Java invocation:
+
+```
+$ java -cp vertx-proxy-0.0.1-SNAPSHOT-fat.jar io.strimzi.kafka.proxy.vertx.VertRunner
+```
+
+If successfully started, the following output appears:
+```
+$ java -cp vertx-proxy-0.0.1-SNAPSHOT-fat.jar io.strimzi.kafka.proxy.vertx.VertRunner
+WARNING: sun.reflect.Reflection.getCallerClass is not supported. This will impact performance.
+2022-04-13 10:30:12 INFO  KafkaProxyVerticle:46 35 - Kafka version: 2.8.0
+2022-04-13 10:30:12 INFO  KafkaProxyVerticle:75 35 - Listening on port 1234
+```
+
+### 6. Start Kafka broker
+
+Now start the Kafka broker, for example:
+```
+$KAFKA_HOME/bin/kafka-server-start.sh config/server.properties 
+```
+
+### 7. Run Kafka clients
+Start the Kafka console producer (note the proxy address in the broker list):
+
+```
+$KAFKA_HOME/bin/kafka-console-producer.sh  --broker-list localhost:1234 --topic enctest --producer.config config/producer.properties 
+```
+
+Start the Kafka console consumer, like the producer, specifying the proxy host and port:
+```
+$KAFKA_HOME/bin/kafka-console-consumer.sh --bootstrap-server localhost:1234 --consumer.config config/consumer.properties --topic enctest  --from-beginning
+```
+
+Enter arbitry data in the producer and verify that it appears in consumer. 
+
+Inspect the topic segment files and verify they indeed are encrypted.
+```
+$KAFKA_HOME/kafka-dump-log.sh --files /tmp/kafka-logs/enctest-0/00000000000000000000.log --value-decoder-class kafka.serializer.StringDecoder
+```

encmod/README.md

@@ -0,0 +1,5 @@
+# Topic Encryption Module
+
+This component is concerned strictly with the encryption and decryption of Kafka records.
+
+

encmod/pom.xml

@@ -0,0 +1,41 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 https://maven.apache.org/xsd/maven-4.0.0.xsd">
+  <parent>
+        <groupId>io.strimzi</groupId>
+        <artifactId>topic-encryption</artifactId>
+        <version>0.0.1-SNAPSHOT</version>
+  </parent>
+  <modelVersion>4.0.0</modelVersion>
+  <artifactId>encmod</artifactId>
+  <name>encryption module</name>
+  <description>desc</description>
+
+  <dependencies>
+  	<dependency>
+  		<groupId>org.apache.kafka</groupId>
+  		<artifactId>kafka-clients</artifactId>
+  	</dependency>
+    <dependency>
+        <groupId>org.apache.logging.log4j</groupId>
+        <artifactId>log4j-api</artifactId>
+    </dependency>
+    <dependency>
+        <groupId>org.apache.logging.log4j</groupId>
+        <artifactId>log4j-core</artifactId>
+    </dependency>
+    <dependency>
+        <groupId>org.apache.logging.log4j</groupId>
+        <artifactId>log4j-slf4j-impl</artifactId>
+    </dependency>   
+    <dependency>
+        <groupId>org.slf4j</groupId>
+        <artifactId>slf4j-api</artifactId>
+    </dependency>   
+    <dependency>
+        <groupId>junit</groupId>
+        <artifactId>junit</artifactId>
+        <scope>test</scope>
+    </dependency>    
+  </dependencies>
+
+</project>

encmod/src/main/java/io/strimzi/kafka/topicenc/EncModControl.java

@@ -0,0 +1,28 @@
+/*
+ * Copyright Strimzi authors.
+ * License: Apache License 2.0 (see the file LICENSE or http://apache.org/licenses/LICENSE-2.0.html).
+ */
+package io.strimzi.kafka.topicenc;
+
+/**
+ * This defines the interface to the Encryption Module to functions
+ * controlling its internal state. So, for example, can an implementation
+ * receiving events from a key management system (KMS), notify the module
+ * to purge a key because it has expired. If we consider the
+ * Encryption Module's encrypt() and decrypt() functions to comprise
+ * the data path, this interface describes its control path. 
+ * 
+ * Currently this interface is a placeholder but will be continually 
+ * extended as the implementation matures.
+ */
+public interface EncModControl {
+
+    /**
+     * Purge the key, indicated by the keyRef argument, from any 
+     * internal state such that the key in question is now longer used.
+     * This supports key revokation.
+     * 
+     * @param keyref A key reference, understood by the Encryption Module and its KMS, identifying the key to purge.
+     */
+	void purgeKey(String keyref);
+}