Feature : Index Media

[dennyabrain]

The feature includes

1. Periodically checking if there is new data in Kosh that hasn't been indexed in the configured indexers and making API calls to index them
2. Monitor the status of Index jobs per Media
3. Get an overview of index job status per dataset4. Retry failed index jobs of a Media
4. If we ever encounter a Media that the indexers can't process, we would like to blacklist it so that any further attempts to index this Media is not made till the issue is resolved.
5. See history of index attempts per Media

Motivation

There is no one solution to getting great search results from a search engine. The process of finding tradeoffs and acceptable solutions is highly iterative. This can only be done at a good pace if the following tasks are cheap to perform :

1. Tweak indexer's algorithm
2. Reset Indices
3. Index Posts
   This feature addresses 2 and 3.

[dennyabrain]

I had worked a bit on thinking about the database structure that would be needed to enable this. So I am documenting it here to start a conversation and critique of what is missing.

I think we should have 2 tables - IndexStatus and IndexHistory. Index status would store the current status of indexing of a media item into the multiple possible indexers. Index history would store all previous values of index status

Both tables will have identical structure

column name	description	notes
indexerId	int	unique IDs to determine which indexer was the media sent to
mediaIid	uuid
status	enum	enqueued, processing, indexed, failed, blacklisted
statusCode	int	a code like http status codes that can uniquely identify issues with indexing
createdAt	timestamp
updatedAt	timestamp

The association between Media, IndexStatus and IndexHistoryy would be :

• Post has One IndexStatus
• Post has Many IndexHistory
• IndexStatus belongs to one Media
• IndexHistory belongs to one Media

[gpstrange]

We can add indexStatus to posts table, rather than maintaining a separate table for IndexStatus. IndexHistory can be a separate table. Association could be,

• Post has Many IndexHistory
• IndexHistory belongs to Post

[dennyabrain]

I am not posting any UI mocks for now. If Guru or any developer want help on that, I am happy to upload some mocks here.

[dennyabrain]

Possible User Stories to implement :

1. As an admin, I want to see the aggregate index status of a dataset.
   I am imagining that for an Dataset, it might look like this

indexer	success	failure
feluda_resnet_vec	80%	20%
phash	100%	0 %

2. As an admin, I want to see the index status and history of a Media

indexer	status	actions
feluda_resnet_vec	success
phash	fail	retry or blacklist

3. As an admin I want to see all Media which hasn't indexed successfully. I will like to group this result by media type and indexer type. I would like to select multiple Media items and issue an action. Action could be index, blacklist, retry etc

[dennyabrain]

This diagram explains the Data Indexing mechanism of Feluda and the APIs that Feluda and Kosh must provide for this to work.

1. Kosh makes a POST request to /index endpoint of Feluda. It passes along the necessary post data, metadata and a config object
2. Feluda's request handler parses the request object and add the necessary fields to a RabbitMQ queue named index-queue
3. One or more workers called indexers can be enabled to start processing the items in the index-queue. These workers are responsible for processing the media item and generating their simplified representation and store it in elasticsearch. They are also responsible for adding the result to a report-queue
4. Another set of workers called reporters can start processing the item in the report-queue whenever they are enabled. They will make a POST request to Kosh's /index/report API endpoint. Kosh is then responsible to update the index status and index history tables in its database.

[dennyabrain]

We should have a way to trigger indexing from kosh to feluda for every dataset or post. And we should also have a script that we can setup to run periodically that gets all unindexed posts and attempts indexing it.