Kafka retention policy

[slifty]

Task

Description

Very little time has been spent on the sample kafka configuration in this repository, and I suspect that kafka retention is very long (maybe forever, maybe several days.)

At any rate, my docker filled up after processing a single channel for about 20 minutes, so we should set the retention policy to something short (e.g. 30 seconds). The intent is that things are processed as they are streamed so I don't think retention is actually ever desirable.

Relevant Resources / Research

• https://stackoverflow.com/a/49982114/159522

[slifty]

Some updates:

1. default retention time is seven days
2. Here's a good article: https://www.baeldung.com/kafka-message-retention

[slifty]

Another update -- it looks like the docker container we are using doesn't make this configurable, though there is an open PR to add that functionality

That said, the Countertop can specify topic level configurations using the configEntries attribute when creating a topic.

With that in mind, this becomes a Countertop issue.

[slifty]

This is now resolved -- hooray!