Some functions can be made cleaner, and some can be extracted to separate files in order to better organize the different parts of the ingester. A cleaner code will help move further with additions and modifications to it.