Problem
#306 handles files and images attachments to messages.
These attachments are currently added to the context, in the history.
However, some models doesn't have the capability to read them, which results to an error. Therefore, the attachment has to be removed from the user message when the error occurs.
When switching from a model (with vision capability) to another (without vision capability), the error can come from a previous message. This is currently handled by going through the entire history to find attachments and remove them (see #306 (comment)).
Proposed Solution
After #262, we have a way to fill capabilities per model
|
export interface IProviderModelInfo { |
Currently, this is used to get the context window available (https://github.com/jupyterlite/ai/blob/main/src/providers/generated-context-windows.ts) per model, computed with https://github.com/jupyterlite/ai/blob/main/scripts/sync-model-context-windows.mjs.
The information comes from https://models.dev/, which also provides an input field, that could probably be useful to build the list of input capabilities per model.
Using that, we could rebuild the history from the chat message when switching of model, removing the attachments if not supported.
Problem
#306 handles files and images attachments to messages.
These attachments are currently added to the context, in the history.
However, some models doesn't have the capability to read them, which results to an error. Therefore, the attachment has to be removed from the user message when the error occurs.
When switching from a model (with vision capability) to another (without vision capability), the error can come from a previous message. This is currently handled by going through the entire history to find attachments and remove them (see #306 (comment)).
Proposed Solution
After #262, we have a way to fill capabilities per model
ai/src/tokens.ts
Line 207 in 0f8ccdc
Currently, this is used to get the context window available (https://github.com/jupyterlite/ai/blob/main/src/providers/generated-context-windows.ts) per model, computed with https://github.com/jupyterlite/ai/blob/main/scripts/sync-model-context-windows.mjs.
The information comes from https://models.dev/, which also provides an input field, that could probably be useful to build the list of input capabilities per model.
Using that, we could rebuild the history from the chat message when switching of model, removing the attachments if not supported.