Optimize skills for MCP + Skill combo (96.3% eval pass rate)#24
Conversation
- Add 'Critical Rules' section at top with SDK/model deprecation warnings - Add 'Documentation Lookup' section with MCP-first instructions - Add 'Behavior Guidelines' section to prevent over-research - Restructure for clearer visual hierarchy with dividers Evaluation results (gemini-3-flash-preview): - Vanilla: 7.7% pass rate - MCP Only: 72.6% pass rate - Skill Only: 82.9% pass rate - Skill+MCP (optimized): 96.3% pass rate The key insight: when MCP is available, instruct the model to use it as the ONLY documentation source and generate code immediately.
Summary of ChangesHello, I'm Gemini Code Assist1! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed! This pull request significantly refines the Gemini API development skills by introducing explicit guidelines and structural improvements. The changes aim to enhance the model's efficiency and accuracy when interacting with Gemini APIs, particularly by enforcing the use of current models and SDKs, streamlining documentation lookup processes, and guiding immediate code generation. This optimization is designed to improve the overall evaluation pass rate of the skills. Highlights
Using Gemini Code AssistThe full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips. Invoking Gemini You can request assistance from Gemini at any point by creating a comment using either
Customization To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a Limitations & Feedback Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for GitHub and other Google products, sign up here. Footnotes
|
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Code Review
This pull request refactors two skill documents, gemini-api-dev/SKILL.md and gemini-interactions-api/SKILL.md, to optimize them for an AI model, particularly when an MCP (Model-in-the-Loop Computing Platform) is available. The changes introduce a standardized structure with 'Critical Rules', 'Documentation Lookup', and 'Behavior Guidelines' sections, which significantly improves clarity and provides more direct instructions. My review focuses on ensuring consistency and actionability of these instructions for the model. I've identified a couple of areas where the instructions could be made more explicit and consistent across the skills, particularly in fallback scenarios and for external dependencies like SDK versions.
| - Latest version: https://central.sonatype.com/artifact/com.google.genai/google-genai/versions | ||
| - Gradle: `implementation("com.google.genai:google-genai:${LAST_VERSION}")` |
There was a problem hiding this comment.
The instruction for finding the latest Java SDK version requires visiting a webpage to find the value for ${LAST_VERSION}. This is ambiguous for a model and not easily actionable. To make this more robust, please provide explicit instructions on how to retrieve this version (e.g., by using a specific tool to fetch and parse the URL). An even better alternative would be to provide the version number directly in this document and keep it updated periodically.
|
|
||
| ### When MCP is NOT Installed (Fallback Only) | ||
|
|
||
| If no MCP documentation tools are available, fetch from the official docs: |
There was a problem hiding this comment.
The fallback documentation lookup instructions are inconsistent with gemini-api-dev/SKILL.md. That skill explicitly mentions the fetch_url tool. To improve consistency and clarity for the model, please explicitly mention the tool to be used for fetching these documentation URLs.
| If no MCP documentation tools are available, fetch from the official docs: | |
| If no MCP documentation tools are available, use the `fetch_url` tool to read from the official docs: |
|
@markmcd i addressed feedback in latest commit - and also added mcp usage to live api skill |
| <version>${LAST_VERSION}</version> | ||
| </dependency> | ||
| ``` | ||
| - `deep-research-pro-preview-12-2025`: Deep Research agent |
There was a problem hiding this comment.
Does this work on the regular API? It makes sense to leave in for the Interactions Skill but if this doesn't work on GC, then we should remove it.
There was a problem hiding this comment.
good point - its not supported by GC, i'm removing
markmcd
left a comment
markmcd
left a comment
There was a problem hiding this comment.
LGTM! can we hold off merging until the other blog goes out? just so they match for a few minutes :)
3 participants
Evaluation results (gemini-3.1-flash-lite):
The key insight: when MCP is available, instruct the model to use it as the ONLY documentation source and generate code immediately.