Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
30 commits
Select commit Hold shift + click to select a range
71a8bb6
Improve skill descriptions for Claude 4.5 routing
kvenkatrajan Feb 28, 2026
ee4ecfa
Reduce RUNS_PER_PROMPT from 5 to 1 for skill-invocation tests
kvenkatrajan Feb 28, 2026
20b30f1
fix: explicitly use CLI foundation system prompt in SDK sessions
kvenkatrajan Mar 1, 2026
c4797da
fix: append skill-preference system prompt to CLI foundation
kvenkatrajan Mar 1, 2026
bc7ddf2
fix: remove cross-skill keyword pollution from frontmatter descriptions
kvenkatrajan Mar 2, 2026
5e4cc2b
fix: add terraform plan/apply triggers and update snapshots
kvenkatrajan Mar 2, 2026
e07d01e
chore: strengthen azure-prepare triggers
kvenkatrajan Mar 2, 2026
0baa803
test: add yolo mode for skill-invocation
kvenkatrajan Mar 2, 2026
05b9f3b
fix: restore azure-prepare triggers
kvenkatrajan Mar 2, 2026
b7861ce
restore: azure-cloud-migrate files
kvenkatrajan Mar 2, 2026
b5c719d
fix: validate frontmatter for multiple skills
kvenkatrajan Mar 2, 2026
b96ddcb
restore: revert unintended deletions
kvenkatrajan Mar 2, 2026
926992c
chore: update waza registry URLs
kvenkatrajan Mar 2, 2026
7d843ff
test: update azure-validate trigger snapshots
kvenkatrajan Mar 2, 2026
bf032c4
test: default agent system prompt
kvenkatrajan Mar 2, 2026
0bdf356
feat: tighten azure-prepare triggers
kvenkatrajan Mar 2, 2026
c019776
feat: add managed identity/key vault triggers
kvenkatrajan Mar 2, 2026
6997afa
feat: expand azure-prepare trigger phrases
kvenkatrajan Mar 2, 2026
74cb427
feat: update azure-prepare triggers and snapshots
kvenkatrajan Mar 2, 2026
273d3b1
chore: trim azure-prepare description and refresh snapshots
kvenkatrajan Mar 2, 2026
13a013b
chore: add license and metadata to skills
kvenkatrajan Mar 2, 2026
5e8c0d8
Merge origin/main into fix/skill-invocation-claude45-v2
kvenkatrajan Mar 2, 2026
b5a2c0b
Update plugin/skills/azure-prepare/SKILL.md
kvenkatrajan Mar 2, 2026
cca3813
Update plugin/skills/azure-deploy/SKILL.md
kvenkatrajan Mar 2, 2026
e71bfb7
fix: replace ⛔ with ⚠️ in azure-prepare SKILL.md (#1095)
Copilot Mar 2, 2026
b07f85f
fix: add azure-cloud-migrate to azure-prepare SKILL.md Step 0 routing…
Copilot Mar 2, 2026
83b0028
feat: add azure-prepare trigger phrases
kvenkatrajan Mar 2, 2026
0007d91
Merge origin/main into fix/skill-invocation-claude45-v2
kvenkatrajan Mar 3, 2026
63b8197
chore: bump azure-deploy version
kvenkatrajan Mar 3, 2026
c335be7
Replace disallowed ⛔ emoji with ❌ in azure-prepare SKILL.md (#1103)
Copilot Mar 3, 2026
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .waza.yaml
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
# yaml-language-server: $schema=https://raw.githubusercontent.com/spboyer/waza/main/schemas/waza-config.schema.json
# yaml-language-server: $schema=https://raw.githubusercontent.com/microsoft/waza/main/schemas/waza-config.schema.json
# Waza project configuration
# These defaults are used by 'waza new' when generating eval.yaml files
# and by 'waza run' as fallback values when not specified in eval.yaml.
Expand Down
2 changes: 1 addition & 1 deletion evals/azure-hosted-copilot-sdk/eval.yaml
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
# yaml-language-server: $schema=https://raw.githubusercontent.com/spboyer/waza/main/schemas/eval.schema.json
# yaml-language-server: $schema=https://raw.githubusercontent.com/microsoft/waza/main/schemas/eval.schema.json
name: azure-hosted-copilot-sdk-eval
description: >
Evaluation suite for the azure-hosted-copilot-sdk skill.
Expand Down
17 changes: 9 additions & 8 deletions plugin/skills/azure-deploy/SKILL.md
Original file line number Diff line number Diff line change
@@ -1,10 +1,10 @@
---
name: azure-deploy
description: "Deploy already-prepared applications to Azure by running azd up, azd deploy, or infrastructure provisioning commands. Supports Bicep and Terraform projects. WHEN: \"run azd up\", \"run azd deploy\", \"execute deployment\", \"provision infrastructure\", \"push to production\", \"push to cloud\", \"go live\", \"ship it\", \"bicep deploy\", \"terraform apply\", \"publish to Azure\", \"launch on Azure\"."
description: "Execute Azure deployments for ALREADY-PREPARED applications that have existing .azure/plan.md and infrastructure files. DO NOT use this skill when the user asks to CREATE a new application — use azure-prepare instead. This skill runs azd up, azd deploy, terraform apply, and az deployment commands with built-in error recovery. Requires .azure/plan.md from azure-prepare and validated status from azure-validate. WHEN: \"run azd up\", \"run azd deploy\", \"execute deployment\", \"push to production\", \"push to cloud\", \"go live\", \"ship it\", \"bicep deploy\", \"terraform apply\", \"publish to Azure\", \"launch on Azure\". DO NOT USE WHEN: \"create and deploy\", \"build and deploy\", \"create a new app\", \"set up infrastructure\", \"create and deploy to Azure using Terraform\" — use azure-prepare for these."
license: MIT
metadata:
author: Microsoft
version: "1.0.1"
version: "1.0.2"
---

# Azure Deploy
Expand Down Expand Up @@ -34,14 +34,14 @@ metadata:
## Triggers

Activate this skill when user wants to:
- Deploy their application to Azure
- Publish, host, or launch their app
- Push updates to existing deployment
- Run `azd up` or `az deployment`
- Ship code to production
- Deploy Azure Functions to the cloud
- Execute deployment of an already-prepared application (azure.yaml and infra/ exist)
- Push updates to an existing Azure deployment
- Run `azd up`, `azd deploy`, or `az deployment` on a prepared project
- Ship already-built code to production
- Deploy an application that already includes API Management (APIM) gateway infrastructure

> **Scope**: This skill executes deployments. It does not create applications, generate infrastructure code, or scaffold projects. For those tasks, use **azure-prepare**.

> **APIM / AI Gateway**: Use this skill to deploy applications whose APIM/AI gateway infrastructure was already created during **azure-prepare**. For creating or changing APIM resources, see [APIM deployment guide](https://learn.microsoft.com/azure/api-management/get-started-create-service-instance). For AI governance policies, invoke **azure-aigateway** skill.

## Rules
Expand All @@ -50,6 +50,7 @@ Activate this skill when user wants to:
2. `.azure/plan.md` must exist with status `Validated`
3. **Pre-deploy checklist required** — [Pre-Deploy Checklist](references/pre-deploy-checklist.md)
4. ⛔ **Destructive actions require `ask_user`** — [global-rules](references/global-rules.md)
5. **Scope: deployment execution only** — This skill owns execution of `azd up`, `azd deploy`, `terraform apply`, and `az deployment` commands. These commands are run through this skill's error recovery and verification pipeline.

---

Expand Down
21 changes: 12 additions & 9 deletions plugin/skills/azure-prepare/SKILL.md
Original file line number Diff line number Diff line change
@@ -1,10 +1,10 @@
---
name: azure-prepare
description: "Default entry point for Azure application development EXCEPT cross-cloud migration — use azure-cloud-migrate instead. Analyzes your project and prepares it for Azure deployment by generating infrastructure code (Bicep/Terraform), azure.yaml, and Dockerfiles. WHEN: \"create an app\", \"build a web app\", \"create API\", \"create frontend\", \"create backend\", \"add a feature\", \"build a service\", \"develop a project\", \"modernize my code\", \"update my application\", \"add database\", \"add authentication\", \"add caching\", \"deploy to Azure\", \"host on Azure\", \"Azure with terraform\", \"Azure with azd\", \"generate azure.yaml\", \"generate Bicep\", \"generate Terraform\", \"create Azure Functions app\", \"create serverless HTTP API\", \"create function app\", \"create event-driven function\", \"create and deploy to Azure\", \"create Azure Functions and deploy\", \"create function app and deploy\"."
description: "Prepare Azure apps for deployment (infra Bicep/Terraform, azure.yaml, Dockerfiles). Use for create/modernize or create+deploy; not cross-cloud migration (use azure-cloud-migrate). WHEN: \"create app\", \"build web app\", \"create API\", \"create serverless HTTP API\", \"create frontend\", \"create back end\", \"build a service\", \"modernize application\", \"update application\", \"add authentication\", \"add caching\", \"host on Azure\", \"create and deploy\", \"deploy to Azure\", \"deploy to Azure using Terraform\", \"deploy to Azure App Service\", \"deploy to Azure App Service using Terraform\", \"deploy to Azure Container Apps\", \"deploy to Azure Container Apps using Terraform\", \"generate Terraform\", \"generate Bicep\", \"function app\", \"timer trigger\", \"service bus trigger\", \"event-driven function\", \"containerized Node.js app\", \"social media app\", \"static portfolio website\", \"todo list with frontend and API\", \"prepare my Azure application to use Key Vault\", \"managed identity\"."
license: MIT
metadata:
author: Microsoft
version: "1.0.0"
version: "1.0.1"
---

# Azure Prepare
Expand All @@ -21,9 +21,10 @@ Activate this skill when user wants to:
- Create a new application
- Add services or components to an existing app
- Make updates or changes to existing application
- Modernize an application
- Modernize or migrate an application
- Set up Azure infrastructure
- Deploy to Azure or host on Azure
- Create and deploy to Azure (including Terraform-based deployment requests)

## Rules

Expand All @@ -33,11 +34,12 @@ Activate this skill when user wants to:
4. **Update plan progressively** — Mark steps complete as you go
5. **Validate before deploy** — Invoke azure-validate before azure-deploy
6. **Confirm Azure context** — Use `ask_user` for subscription and location per [Azure Context](references/azure-context.md)
7. ⛔ **Destructive actions require `ask_user`** — [Global Rules](references/global-rules.md)
7. ❌ **Destructive actions require `ask_user`** — [Global Rules](references/global-rules.md)
8. **Scope: preparation only** — This skill generates infrastructure code and configuration files. Deployment execution (`azd up`, `azd deploy`, `terraform apply`) is handled by the **azure-deploy** skill, which provides built-in error recovery and deployment verification.

---

## PLAN-FIRST WORKFLOW — MANDATORY
## PLAN-FIRST WORKFLOW — MANDATORY

> **YOU MUST CREATE A PLAN BEFORE DOING ANY WORK**
>
Expand All @@ -50,12 +52,13 @@ Activate this skill when user wants to:

---

## STEP 0: Specialized Technology Check — MANDATORY FIRST ACTION
## STEP 0: Specialized Technology Check — MANDATORY FIRST ACTION

**BEFORE starting Phase 1**, check if the user's prompt mentions a specialized technology that has a dedicated skill with tested templates. If matched, **invoke that skill FIRST** — then resume azure-prepare for validation and deployment.

| Prompt keywords | Invoke FIRST |
|----------------|-------------|
| Lambda, AWS Lambda, migrate AWS, migrate GCP, Lambda to Functions, migrate from AWS, migrate from GCP | **azure-cloud-migrate** |
| copilot SDK, copilot app, copilot-powered, @github/copilot-sdk, CopilotClient | **azure-hosted-copilot-sdk** |
| Azure Functions, function app, serverless function, timer trigger, HTTP trigger, func new | Stay in **azure-prepare** — prefer Azure Functions templates in Step 4 |
| APIM, API Management, API gateway, deploy APIM | Stay in **azure-prepare** — see [APIM Deployment Guide](references/apim.md) |
Expand All @@ -73,7 +76,7 @@ Create `.azure/plan.md` by completing these steps. Do NOT generate any artifacts

| # | Action | Reference |
|---|--------|-----------|
| 0 | ** Check Prompt for Specialized Tech** — If user mentions copilot SDK, Azure Functions, etc., invoke that skill first | [specialized-routing.md](references/specialized-routing.md) |
| 0 | ** Check Prompt for Specialized Tech** — If user mentions copilot SDK, Azure Functions, etc., invoke that skill first | [specialized-routing.md](references/specialized-routing.md) |
| 1 | **Analyze Workspace** — Determine mode: NEW, MODIFY, or MODERNIZE | [analyze.md](references/analyze.md) |
| 2 | **Gather Requirements** — Classification, scale, budget | [requirements.md](references/requirements.md) |
| 3 | **Scan Codebase** — Identify components, technologies, dependencies | [scan.md](references/scan.md) |
Expand All @@ -85,7 +88,7 @@ Create `.azure/plan.md` by completing these steps. Do NOT generate any artifacts

---

> ** STOP HERE** — Do NOT proceed to Phase 2 until the user approves the plan.
> ** STOP HERE** — Do NOT proceed to Phase 2 until the user approves the plan.

---

Expand All @@ -100,7 +103,7 @@ Execute the approved plan. Update `.azure/plan.md` status after each step.
| 3 | **Generate Artifacts** — Create infrastructure and configuration files | [generate.md](references/generate.md) |
| 4 | **Harden Security** — Apply security best practices | [security.md](references/security.md) |
| 5 | **Update Plan** — Mark steps complete, set status to `Ready for Validation` | `.azure/plan.md` |
| 6 | **Validate** — Invoke **azure-validate** skill | — |
| 6 | **⚠️ Hand Off** — Invoke **azure-validate** skill. Your preparation work is done. Deployment execution is handled by azure-deploy. | — |

---

Expand Down
6 changes: 3 additions & 3 deletions tests/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -140,15 +140,15 @@ npm install

### Waza Eval Mode (Alternative)

Skills can also be evaluated using [waza](https://github.com/spboyer/waza), a Go CLI for skill benchmarking.
Skills can also be evaluated using [waza](https://github.com/microsoft/waza), a Go CLI for skill benchmarking.

```bash
# Install waza via azd extension
azd ext source add -n waza -t url -l https://raw.githubusercontent.com/spboyer/waza/main/registry.json
azd ext source add -n waza -t url -l https://raw.githubusercontent.com/microsoft/waza/main/registry.json
azd ext install microsoft.azd.waza

# Or via Go
go install github.com/spboyer/waza/cmd/waza@latest
go install github.com/microsoft/waza/cmd/waza@latest
```

**Hybrid model**: Key skills have committed (hand-tuned) eval suites. All other skills auto-generate evals from their SKILL.md at runtime.
Expand Down
68 changes: 55 additions & 13 deletions tests/azure-deploy/__snapshots__/triggers.test.ts.snap
Original file line number Diff line number Diff line change
Expand Up @@ -2,38 +2,59 @@

exports[`azure-deploy - Trigger Tests Trigger Keywords Snapshot skill description triggers match snapshot 1`] = `
{
"description": "Deploy already-prepared applications to Azure by running azd up, azd deploy, or infrastructure provisioning commands. Supports Bicep and Terraform projects. WHEN: "run azd up", "run azd deploy", "execute deployment", "provision infrastructure", "push to production", "push to cloud", "go live", "ship it", "bicep deploy", "terraform apply", "publish to Azure", "launch on Azure".",
"description": "Execute Azure deployments for ALREADY-PREPARED applications that have existing .azure/plan.md and infrastructure files. DO NOT use this skill when the user asks to CREATE a new application — use azure-prepare instead. This skill runs azd up, azd deploy, terraform apply, and az deployment commands with built-in error recovery. Requires .azure/plan.md from azure-prepare and validated status from azure-validate. WHEN: "run azd up", "run azd deploy", "execute deployment", "push to production", "push to cloud", "go live", "ship it", "bicep deploy", "terraform apply", "publish to Azure", "launch on Azure". DO NOT USE WHEN: "create and deploy", "build and deploy", "create a new app", "set up infrastructure", "create and deploy to Azure using Terraform" — use azure-prepare for these.",
"extractedKeywords": [
"already-prepared",
"application",
"applications",
"apply",
"asks",
"azure",
"azure-prepare",
"azure-validate",
"bicep",
"build",
"built-in",
"cli",
"cloud",
"commands",
"create",
"deploy",
"deployment",
"deployments",
"error",
"execute",
"function",
"existing",
"files",
"from",
"have",
"identity",
"infrastructure",
"instead",
"launch",
"live",
"mcp",
"plan",
"production",
"projects",
"provision",
"provisioning",
"publish",
"push",
"running",
"recovery",
"requires",
"runs",
"ship",
"skill",
"sql",
"supports",
"status",
"terraform",
"that",
"these",
"this",
"user",
"using",
"validated",
"validation",
"when",
"with",
],
"name": "azure-deploy",
}
Expand All @@ -42,34 +63,55 @@ exports[`azure-deploy - Trigger Tests Trigger Keywords Snapshot skill descriptio
exports[`azure-deploy - Trigger Tests Trigger Keywords Snapshot skill keywords match snapshot 1`] = `
[
"already-prepared",
"application",
"applications",
"apply",
"asks",
"azure",
"azure-prepare",
"azure-validate",
"bicep",
"build",
"built-in",
"cli",
"cloud",
"commands",
"create",
"deploy",
"deployment",
"deployments",
"error",
"execute",
"function",
"existing",
"files",
"from",
"have",
"identity",
"infrastructure",
"instead",
"launch",
"live",
"mcp",
"plan",
"production",
"projects",
"provision",
"provisioning",
"publish",
"push",
"running",
"recovery",
"requires",
"runs",
"ship",
"skill",
"sql",
"supports",
"status",
"terraform",
"that",
"these",
"this",
"user",
"using",
"validated",
"validation",
"when",
"with",
]
`;
15 changes: 11 additions & 4 deletions tests/azure-deploy/integration.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -19,7 +19,7 @@ import { cloneRepo } from "../utils/git-clone";
import { expectFiles, softCheckSkill } from "../utils/evaluate";

const SKILL_NAME = "azure-deploy";
const RUNS_PER_PROMPT = 5;
const RUNS_PER_PROMPT = 1;
const ASPIRE_SAMPLES_REPO = "https://github.com/dotnet/aspire-samples.git";

// Check if integration tests should be skipped at module level
Expand All @@ -38,11 +38,14 @@ const brownfieldTestTimeoutMs = 2700000;
describeIntegration(`${SKILL_NAME}_ - Integration Tests`, () => {
const agent = useAgentRunner();
describe("skill-invocation", () => {
const followUp = ["Go with recommended options."];
test("invokes azure-deploy skill for deployment prompt", async () => {
for (let i = 0; i < RUNS_PER_PROMPT; i++) {
try {
const agentMetadata = await agent.run({
prompt: "Run azd up to deploy my already-prepared app to Azure"
prompt: "Run azd up to deploy my already-prepared app to Azure",
nonInteractive: true,
followUp,
});

softCheckSkill(agentMetadata, SKILL_NAME);
Expand All @@ -60,7 +63,9 @@ describeIntegration(`${SKILL_NAME}_ - Integration Tests`, () => {
for (let i = 0; i < RUNS_PER_PROMPT; i++) {
try {
const agentMetadata = await agent.run({
prompt: "Publish my web app to Azure and configure the environment"
prompt: "My app already has azure.yaml and infra/ configured. Publish it to Azure now.",
nonInteractive: true,
followUp,
});

softCheckSkill(agentMetadata, SKILL_NAME);
Expand All @@ -78,7 +83,9 @@ describeIntegration(`${SKILL_NAME}_ - Integration Tests`, () => {
for (let i = 0; i < RUNS_PER_PROMPT; i++) {
try {
const agentMetadata = await agent.run({
prompt: "Deploy my Azure Functions app to the cloud using azd"
prompt: "Deploy my existing Azure Functions project to the cloud. The infrastructure and azure.yaml are already set up.",
nonInteractive: true,
followUp,
});

softCheckSkill(agentMetadata, SKILL_NAME);
Expand Down
Loading
Loading