How to implement multiple RAG / Text Searches?

[roldengarm]

We currently have one plugin that retrieves data from a backend system if needed, i.e. RAG
We now also want to add a Google Search. However, no matter what I do, it never executes both. I've tried tweaking the prompt, tried changing the settings, etc.
I've initially implemented the Google Search as a custom plugin, but then I discovered the new Text Search (in preview).
Still, the same problem.

I've even tried converting our existing backend system plugin to an implementation of ITextSearch, but then it still only ever picks one.

Isn't it possible to have RAG over multiple plugins? We're using o3-mini as our model.

[moonbox3]

Adding @westey-m for help.

[westey-m]

Hi @roldengarm,
Getting the LLM to invoke both when they are explicitly optional could be tricky. Having multiple plugins is certainly supported, but if you leave it to the LLM to decide which to invoke, it's important to help it understand which to invoke in which situation.
E.g. when using textsearch.CreateWithGetTextSearchResults, you can provide a description as an optional parameter which allows you to describe what that plugin does in more detail, allowing the LLM to pick the most appropriate one.

If ultimately you want both to always be invoked whenever a search is required, it might make sense to create an aggregate ITextSearch implementation, so that the LLM doesn't have to figure out that it always needs to invoke both, if it decides that it needs to invoke any at all. It would be a simple class that implements the ITextSearch interface, and take two or more ITextSearch implementations as input, and just calls all of them and aggregates the results before returning.

There is of course also the option of invoking the plugin manually in the prompt template, which will allow you to always invoke both, but then the LLM would not be the one deciding. See https://learn.microsoft.com/en-us/semantic-kernel/concepts/text-search/text-search-plugins?pivots=programming-language-csharp#search-plugin-with-citations

[roldengarm]

Thanks @westey-m for understanding the issue, I can work with this!