Let built-in tools fallback on custom tools for models that don't support them

[DouweM]

Description

Built-in tools are great, but specifying them on an agent currently limits you to using it only with models that support all of them, which makes it hard to use FallbackModel or try different models in evals.

It'd be great if you could specify a function tool (or toolset) to be used as a fallback when a built-in tool isn't available:

• WebSearchTool -> Tavily or DuckDuckGo from https://ai.pydantic.dev/common-tools/
• CodeExecutionTool -> https://github.com/pydantic/mcp-run-python
• ImageGenerationTool -> custom tool that calls an image model
• MemoryTool -> e.g. Add mem0 as a memory provider #3129 (This already does something somewhat similar in that it requires a memory tool to be defined)
• UrlContextTool -> custom tool that runs requests.get and renders HTML to Markdown or something
• MCPServerTool from Support OpenAI and Anthropic native MCP support via MCPServerTool builtin tool #3101 -> MCPServer from https://ai.pydantic.dev/mcp/client/ or FastMCPToolset from Add FastMCPToolset #2784

I imagine something like @agent.tool(fallback_for_builtin=WebSearchTool) (and similar on the Tool class). That would then be stored on the ToolDefinition, so that Model.prepare_request can look at the ModelRequestParameters.function_tools and .builtin_tools, check which are supported by the model in question (via new properties on self.profile: ModelProfile most likely), and then keep only one or the other. We'd only raise an error for unsupported built-in tools if no fallback was provided.

To support falling back from a built-in tool to a toolset (like with MemoryTool or MCPServerTool), we'd need a way to set this on a toolset as well, most likely when registering it on an agent rather than hard-coded on the toolset itself. With MCPServerTool, we'd also need to consider that a given MCPServer/FastMCPToolset would be a fallback for a specific MCPServerTool instance, not all MCPServerTools in general.

References

No response