Ability to map external API objects to search documents during index rebuild

[MilanLund]

We have an external REST API endpoint that responds with a JSON array containing hundreds of objects. Each object includes a property that is a URL (each object actually represents a web page that has no DB representation in XbyK).

I’m looking for a way to index each of these objects using this package so that users of our site can find them as search records, alongside other records that represent web page items.

I reviewed the documentation but couldn’t find a suitable solution. It seems that the current limitation lies in MapToLuceneDocumentOrNull, which maps web page items to Documents one by one.

It would be great to have an option that allows custom logic during an index rebuild (for example, requesting the REST API and creating as many Documents as needed).

Is there a way to achieve this with the current functionality, or would support for this need to be added?

[liparova]

Hi @MilanLund,

Thanks, we’ll look into this and let you know how to proceed.

[bkapustik]

Hello, we currently do not have support for the requested functionality. This would need to be added.

[cdumange]

Hi

Would use that too. We have a API that we integrate into a kentico module and we have no clear way of indexing those values as it's only based around reusable content and pages.

[DGibbsCrafted]

@bkapustik - We're also facing a similar problem, we would like to index custom module class data but the package only supports pages/reusable content.

Is this on your development roadmap at all? We would like a way of providing a custom index implementation which isn't based on a predefined list of page / content types but can be built from any data source.

The K13 ICustomSearchIndex is what we'd like:
https://docs.kentico.com/13/configuring-xperience/setting-up-search-on-your-website/using-locally-stored-search-indexes/creating-local-search-indexes/creating-custom-smart-search-indexes

[bluemodus-mwills]

Hi @bkapustik & @DavidSlavik ,

We have customers who need external content added to an index too (e.g., courses from an LMS, pages from a separate Foundation site).

Would this work for getting external content into the index?

• Let a custom indexing strategy expose a GetItemsForRebuild method that returns a list with a new model, IndexEventExternalItemModel.
• RebuildInternal appends whatever it returns before publishing, so external items go through the pipeline.
• ProcessLuceneTasksWebFarmTask has a new property LuceneExternalQueueItems.
• LuceneQueueWorker queues IndexEventExternalItemModel in EnqueueLuceneQueueItem and populates the tasks new property in ProcessItems.
• A customer's custom indexing strategy could then use pattern matching in the MapToLuceneDocumentOrNull method, and call whatever external APIs needed to get the details of the external item.

Thank you!