computer use for google

[narengogi]

Google computer use is a beta tool supported in the gemini computer use model
https://docs.cloud.google.com/vertex-ai/generative-ai/docs/computer-use

The supported parameters for the tool are

environment (can either be "ENVIRONMENT_BROWSER" and "")
excluded_predefined_functions (list of strings like ["navigate"]

example payloads:
single turn

{
    "model": "gemini-2.5-computer-use-preview-10-2025",
    "stream": false,
    "messages": [
        {
            "role": "system",
            "content": "Say Hi"
        },
        {
            "role": "user",
            "content": "Go to google.com and search for 'weather in New York'"
        }
    ],
    "tools": [
        {
            "type": "function",
            "function": {
                "name": "computer_use",
                "parameters": {
                    "environment": "ENVIRONMENT_BROWSER"
                }
            }
        }
    ]
}

{
    "model": "gemini-2.5-computer-use-preview-10-2025",
    "stream": false,
    "messages": [
        {
            "role": "system",
            "content": "Say Hi"
        },
        {
            "role": "user",
            "content": "Go to google.com and search for 'weather in New York'"
        },
        {
            "role": "assistant",
            "tool_calls": [
                {
                    "id": "portkey-50925c03-b8cc-4057-948b-13a9d9de19e0",
                    "type": "function",
                    "function": {
                        "name": "open_web_browser",
                        "arguments": "{}"
                    }
                }
            ]
        },
        {
            "role": "user",
            "content": "I've opened the browser"
        }
    ],
    "tools": [
        {
            "type": "function",
            "function": {
                "name": "computerUse",
                "parameters": {
                    "environment": "ENVIRONMENT_BROWSER"
                }
            }
        }
    ]
}

[matter-code-review]

Description

Summary By MatterAI

🔄 What Changed

Added support for Google's computerUse tool in Vertex AI integration by introducing new tool name mappings and transformation logic. Extended tool handling in both google-vertex-ai and google providers to recognize and transform computerUse and related tools like googleSearchRetrieval.

🔍 Impact of the Change

Enables AI models to use Google's computer interaction capabilities (e.g., screen reading, desktop automation) via computerUse tool. Improves tool extensibility by centralizing supported tools in googleTools array and routing transformations through transformGoogleTools. Reduces code duplication by reusing tool builders across providers.

📁 Total Files Changed

File	ChangeLog
Tool Config src/providers/google-vertex-ai/chatComplete.ts	Added googleTools and transformGoogleTools imports; updated tool processing logic to handle special Google tools
Utils Additions src/providers/google-vertex-ai/utils.ts	Introduced googleTools list and transformGoogleTools function to handle computerUse, googleSearch, and googleSearchRetrieval tools
Provider Sync src/providers/google/chatComplete.ts	Aligned with Vertex AI changes by importing and using shared Google tool transformation logic

🧪 Test Added/Recommended

Recommended

• Unit test for transformGoogleTools covering all tool types (computerUse, googleSearch, googleSearchRetrieval)
• Integration test validating computerUse tool payload structure in Vertex AI request
• Negative test for unsupported tool names falling back to default function declaration

🔒 Security Vulnerabilities

N/A

⏳ Estimated code review effort

MEDIUM (~15 minutes)

Tip

Quality Recommendations

1. Add unit tests for transformGoogleTools to validate all tool type transformations

2. Include input validation for computerUse environment and excludedPredefinedFunctions parameters

3. Add JSDoc comments to transformGoogleTools explaining each supported tool and transformation logic

4. Ensure googleTools array is frozen (Object.freeze) to prevent runtime mutations

♫ Tanka Poem

Code now sees the screen 🖥️
Computer use, bold and keen—
AI takes the wheel.
Tools align, paths merge clean,
Gemini's new dream.

Sequence Diagram

sequenceDiagram
    participant Client
    participant GoogleChatComplete
    participant transformGoogleTools
    participant VertexAPI

    Client->>GoogleChatComplete: send tool request (e.g., computerUse)
    GoogleChatComplete->>GoogleChatComplete: check if tool.function.name in googleTools
    alt Supported Google Tool
        GoogleChatComplete->>transformGoogleTools: transformGoogleTools(tool)
        transformGoogleTools->>transformGoogleTools: route by tool name
        transformGoogleTools-->>GoogleChatComplete: return structured tool config
        GoogleChatComplete->>VertexAPI: include in request payload
    else Custom Function
        GoogleChatComplete->>GoogleChatComplete: push to functionDeclarations
        GoogleChatComplete->>VertexAPI: send as standard function tool
    end
    VertexAPI-->>Client: execute tool and return response

Loading

[matter-code-review]

✅ Reviewed the changes: Review of Google-specific tool support implementation