Allow specifying preferred / suggested models for a specific prompt / task

[felixarntz]

This has been raised quite a bit in the past few weeks/months, but we never actually got to implementing it: We need to allow developers to (optionally!) specify a model preference list. I believe this is a critical feature for the following reasons:

• We want to encourage developers to use the SDK in a model agnostic way, i.e. not force usage of a specific model.
• The reality though is that many experienced developers will have preferences for which model(s) to use for a given process in their code, either simply as a personal preference or, ideally, due to having run evals and identified the strongest model(s) for that process.
• Today, these developers can only accomplish their goal by force-choosing their one preferred model, which is an understandable choice from their end, but actively goes against what we want to encourage.

Allowing to express model preferences/suggestions without requiring that one of these models to be used addresses these problems.

Users still benefit from the best model(s) for the specific task when they're available, but if on a site that model's provider is not configured, they're not locked out from using the feature.

Technical design

High-level suggestion:

• We need to have a method on PromptBuilder that allows to provide a list of model IDs, or tuples of provider ID plus model ID.

More concretely:

• Naming is a hard question, as always:
  • usingPreferredModels seems wrong, as that sounds too strong in that always one of these models will be used, which is not the case.
  • usingModelPreferences sounds better, but "preferences" is somehow ambiguous as it's often used as a synonym for "options" or "config", which is not what this is.
  • usingModelSuggestions might be a good compromise. Maybe there's another name that's even more suitable though :)
• I think we should allow passing an array/list, where each entry is:
  • either a model ID string (e.g. 'gemini-flash-2.5')
  • or a tuple of provider ID string plus model ID string (e.g. ['google', 'gemini-flash-2.5'])
  • or a ModelInterface instance, used simply to extract provider ID and model ID (e.g. Google::model('gemini-flash-2.5'))
• Under the hood we could introduce an SDK internal class like ModelPreferences or ModelSuggestions (pending naming decisions above) that receives the array and then has methods to filter a given list<ModelMetadata> or list<ProviderModelsMetadata>, or extract the specific "most preferred/suggested" model out of such a list.

[JasonTheAdams]

Yup! This is pretty much aligned with what I had in mind. Two thoughts:

1. I like usingModelPreference (somehow singular sounds better to me)
2. Make it variadic such that it supports either a model ID string or ModelInterface instance — so it can actually be a mix (an probably normalized)

But yeah! I agree and think this will be useful to some folks.

[felixarntz]

1. I like usingModelPreference (somehow singular sounds better to me)

Love this! So simple, and it avoids the confusion that people may associate "preferences" as a common term for "settings" or "options", which is not what this is.

But yeah! I agree and think this will be useful to some folks.

FWIW I'd even say, at least in the current stage, it should be crucial to many folks. Right now, our SDK has no model preference whatsoever, which means the first alphabetical model found that satisfies the criteria is used. That's not great, because that means e.g. someone that configures Google often gets a gemini-1.5 model which is long out of date and far from the SOTA models. That's probably something we'll also want to improve internally in the SDK, e.g. by including a default model preferences list in each model provider itself - let's do that in a separate issue, but it could potentially be built on top of the mechanism we introduce here.