Plan for Improving Kosh's Resiliency

[dennyabrain]

My current assessment of the status quo is that, Kosh is largely feature complete for what we need for a v1.0.0 release but lacks on the front of reliability, resilience and security. I will lay down a rough scope in this email. First a glossary of key elements of what constitutes kosh :

1. Feluda : a multimodal mutlilingual search engine built by Tattle. Uses Elasticsearch as the data store and implements full text search (for text posts) and vector search for images and videos. The current iteration of feluda has highly decoupled modules that can potentially be scaled up and down independent of each other. Major components are
   1. REST API Server : Its main job is servicing API requests. Since some indexing tasks can be time consuming, the server often will just take a client request and add it to a message queue which is then operated upon by what we call workers. Code entrypoint : https://github.com/tattle-made/feluda/blob/master/src/api/server.py
   2. Indexer (a type of Worker) - This takes image,video, text and converts them into appopriate representation and saves in Elasticsearch. Code entrypoint - https://github.com/tattle-made/feluda/blob/master/src/api/indexer.py
   3. Reporter - This notifies the client service when the indexing process is done. Code entrypoint - https://github.com/tattle-made/feluda/blob/master/src/api/reporter.py
2. Kosh : Its components are
   1. Web Frontend : the UI to access all storage and search related features. Code entrypoint - https://github.com/tattle-made/kosh-v2/tree/main/src/frontend
   2. API Server : coordinates between Web Frontend, SQL Storage, Mongo Storage and Feluda. Also responsible for authentication and User Management Etc. Code entrypoint - https://github.com/tattle-made/kosh-v2/tree/main/src/backend
3. File Server : This is an abuse prevention mechanism. All links to media items (text, images and videos) is proxied via this server so that one could not potentially bankrupt us by downloading media items in an infinite loop. It does a basic rate limiting as of now.
4. Infrastructure :
   1. SQL : Primary data store for kosh
   2. Mongo : Stores unstructured metadata associated with each post within a dataset
   3. ElasticSearch : For Fulltext and vector search
   4. RabbitMQ : a message queue to coordinate operations of various feluda workers
5. Data Upload Scripts : These are scripts that upload a dataset to Kosh. These basically use Kosh's rest API so as such can be written in any language. Right now some js scripts exist here - https://github.com/tattle-made/kosh-v2/tree/main/src/scripts-upload-media
6. Scrapers : These are related to kosh in that without good scrapers scraping data and creating datasets there won't be anything to upload to Kosh. Their roles is basically to get data from somewhere and store it in a database and file storage like s3. The database and file storage chosen is irrelevant for us because once the data exists, its the responsibility of the Data Upload Scripts to push it to kosh in a consistent format.
   1. Tattle's own scrapers push to a mongo database and aws s3
      1. whatsapp scraper - https://github.com/tattle-made/whatsapp-scraper
      2. factcheck article scraper - https://github.com/tattle-made/factchecking-sites-scraper

Repositories of Interest :
Kosh : https://github.com/tattle-made/kosh-v2/
Feluda : https://github.com/tattle-made/feluda

With this background, to build a more stable, resilient and secure kosh, the following tasks need to be done

1. Improve Security
   1. Add basic authentication to Feluda's /search endpoint. feluda#21
   2. As Tattle, I want to be sure that unauthorized access to the data is not possible #31
2. Improve Resilience and Scaling
   1. Improve Resilience feluda#22
3. Single Node Elasticsearch
   1. Run elasticsearch on k8 services-infrastructure#11
4. Discuss strategy to automate manual chores. A few of our processes to ensure everything works end to end is still manual, lets see if some of this can be automated.