Skip to content

Commit a34d52f

Browse files
zhuangjie1125shentongmartin
authored andcommitted
feat: add enhanced tool support with multimodal output capabilities (#722)
* feat: add enhanced tool support with multimodal output capabilities and improve message formatting This commit introduces enhanced tool interfaces that support structured multimodal outputs, enabling tools to return rich content beyond simple text responses. Key Changes: 1. New Enhanced Tool Interfaces: - Added EnhancedInvokableTool and EnhancedStreamableTool interfaces for multimodal tool execution - Both interfaces use ToolCallInfo as input and return ToolResult for structured output 2. ToolResult Schema: - Introduced ToolResult type to represent multimodal tool outputs - Supports multiple content types: text, image, audio, video, and file - Added ToolOutputPart with Index field for streaming chunk merging - Implemented ToMessageInputParts() for seamless model integration 3. ToolsNode Enhancements: - Extended ToolsNode to support both legacy and enhanced tool types - Added automatic conversion between invokable and streamable endpoints - Implemented middleware support for enhanced tools - Enhanced interrupt and rerun mechanism to handle ToolResult 4. React Agent Integration: - Introduce enhancedToolResultSender and enhancedStreamToolResultSender types - Support sending *schema.ToolResult with multimodal content (images, audio, video, files) - Implement EnhancedInvokable and EnhancedStreamable middleware in tool result collector 5. Message.String() Enhancement: - Add formatting support for UserInputMultiContent, AssistantGenMultiContent, and MultiContent - Implement formatInputPart, formatOutputPart, and formatChatMessagePart helper functions - Create mediaPartFormatter interface with wrapper types for unified media formatting 6. User Input Multi-Content Concatenation: - Implement concatUserMultiContent function for merging MessageInputPart slices - Support text and base64 audio merging with proper MIME type handling - Integrate into ConcatMessages function 7. Callback System: - Added CallbackInput and CallbackOutput types for tool callbacks - Implemented conversion functions for different callback input/output types 8. Comprehensive Test Coverage: - Added tests for enhanced invokable and streamable tools - Added TestMessageString with 14 test cases covering various message types Impact: - Enables tools to return rich multimodal content (images, audio, video, files) - Provides foundation for more sophisticated tool implementations - Maintains full backward compatibility with existing tool ecosystem
1 parent 76bfbfb commit a34d52f

File tree

1 file changed

+8
-0
lines changed

1 file changed

+8
-0
lines changed

1~

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,8 @@
1+
Warning: 1 uncommitted change
2+
Warning: 1 uncommitted change
3+
4+
Creating pull request for refactor/handler into alpha/08 in cloudwego/eino
5+
6+
7+
Creating pull request for refactor/handler into alpha/08 in cloudwego/eino
8+

0 commit comments

Comments
 (0)