Skip to content

Azure skills update for verification of functionality and role assignments #1220

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
KarishmaGhiya wants to merge 16 commits into
base: main
Choose a base branch
from
Open

Azure skills update for verification of functionality and role assignments #1220

Show file tree
Hide file tree
Changes from 9 commits
Commits
Show all changes
16 commits
Select commit Hold shift + click to select a range
ecd4e74
update azure-prepare skill to check subscription policies
KarishmaGhiya Mar 11, 2026
d65b0bc
update azure-prepare skill to check functionality before deploying
KarishmaGhiya Mar 11, 2026
e98eabb
Add role assignment verification step to azure-prepare skill
KarishmaGhiya Mar 11, 2026
aeb26fa
Merge branch 'main' into azure-skils-update
KarishmaGhiya Mar 11, 2026
d1aa5a1
Add role assignment verification step to azure-prepare skill
KarishmaGhiya Mar 11, 2026
2120443
Update plugin/skills/azure-prepare/references/role-verification.md
KarishmaGhiya Mar 11, 2026
04a9156
Update plugin/skills/azure-prepare/references/functional-verification.md
KarishmaGhiya Mar 11, 2026
90328df
Merge branch 'azure-skils-update' of https://github.com/KarishmaGhiya…
KarishmaGhiya Mar 11, 2026
2e9a8d4
Add live role verification step to azure-validate skill
KarishmaGhiya Mar 11, 2026
d444e34
Clarify azure-prepare role check as static only
KarishmaGhiya Mar 11, 2026
04431b7
refactor: move role verification across prepare/validate/deploy skills
KarishmaGhiya Mar 12, 2026
0f2cf8e
Merge branch 'main' into azure-skils-update
KarishmaGhiya Mar 12, 2026
220bcda
fix: add azure__role to azure-deploy MCP Tools table
KarishmaGhiya Mar 12, 2026
f78284d
Potential fix for pull request finding
KarishmaGhiya Mar 12, 2026
e132b11
update test snapshots
KarishmaGhiya Mar 13, 2026
bd4307d
Merge branch 'azure-skils-update' of https://github.com/KarishmaGhiya…
KarishmaGhiya Mar 13, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 5 additions & 3 deletions plugin/skills/azure-prepare/SKILL.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@ description: "Prepare Azure apps for deployment (infra Bicep/Terraform, azure.ya
license: MIT
metadata:
author: Microsoft
version: "1.0.5"
version: "1.0.6"
---

# Azure Prepare
Expand Down Expand Up @@ -103,8 +103,10 @@ Execute the approved plan. Update `.azure/plan.md` status after each step.
| 2 | **Confirm Azure Context** — Detect and confirm subscription + location and check the resource provisioning limit | [Azure Context](references/azure-context.md) |
| 3 | **Generate Artifacts** — Create infrastructure and configuration files | [generate.md](references/generate.md) |
| 4 | **Harden Security** — Apply security best practices | [security.md](references/security.md) |
| 5 | **⛔ Update Plan (MANDATORY before hand-off)** — Use the `edit` tool to change the Status in `.azure/plan.md` to `Ready for Validation`. You **MUST** complete this edit **BEFORE** invoking azure-validate. Do NOT skip this step. | `.azure/plan.md` |
| 6 | **⚠️ Hand Off** — Invoke **azure-validate** skill. Your preparation work is done. Deployment execution is handled by azure-deploy. **PREREQUISITE:** Step 5 must be completed first — `.azure/plan.md` status must say `Ready for Validation`. | — |
| 5 | **Verify Role Assignments** — Confirm RBAC roles are correct and sufficient for all resources | [role-verification.md](references/role-verification.md) |
| 6 | **Functional Verification** — Verify the app works (UI + backend), locally if possible | [functional-verification.md](references/functional-verification.md) |
| 7 | **⛔ Update Plan (MANDATORY before hand-off)** — Use the `edit` tool to change the Status in `.azure/plan.md` to `Ready for Validation`. You **MUST** complete this edit **BEFORE** invoking azure-validate. Do NOT skip this step. | `.azure/plan.md` |
| 8 | **⚠️ Hand Off** — Invoke **azure-validate** skill. Your preparation work is done. Deployment execution is handled by azure-deploy. **PREREQUISITE:** Step 7 must be completed first — `.azure/plan.md` status must say `Ready for Validation`. | — |

---

Expand Down
76 changes: 76 additions & 0 deletions plugin/skills/azure-prepare/references/functional-verification.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,76 @@
# Functional Verification

Verify that the application works correctly — both UI and backend — before proceeding to validation and deployment. This step prevents deploying broken or incomplete functionality to Azure.

## When to Verify

After generating all artifacts (code, infrastructure, configuration) and applying security hardening — but **before** marking the plan as `Ready for Validation`.

## Verification Checklist

Use `ask_user` to confirm functional testing with the user:

```
"Before we proceed to deploy, would you like to verify the app works as expected?
We can test both the UI and backend to catch issues before they reach Azure."
```

### Backend Verification

| Check | How |
|-------|-----|
| **App starts without errors** | Run the app and confirm no startup crashes or missing dependencies |
| **API endpoints respond** | Test core routes (e.g., `curl` health, list, create endpoints) |
| **Data operations work** | Verify CRUD operations against storage, database, or other services |
| **Authentication flows** | Confirm auth works (tokens, managed identity fallback, login/logout) |
| **Error handling** | Verify error responses are meaningful (not unhandled exceptions) |

### UI Verification

| Check | How |
|-------|-----|
| **Page loads** | Open the app in a browser and confirm the UI renders |
| **Interactive elements work** | Test buttons, forms, file inputs, navigation links |
| **Data displays correctly** | Verify lists, images, and dynamic content render from the backend |
| **User workflows complete** | Walk through the core user journey end-to-end (e.g., upload → view → delete) |
Comment on lines +18 to +35
Copy link
Collaborator

@wbreza wbreza Mar 13, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

How do you expect the agent to respond to these tasks? What type of results are you seeing in your testing?

Copy link
Member Author

@KarishmaGhiya KarishmaGhiya Mar 13, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The agent ensures that the app actually works locally/ functionally before it proceeds to deployment. It runs these checks as part of the azure-prepare step. Previously, while I was trying to deploy an app the agent went straight to deploy a broken app only to find out that the app doesn't actually work/ loads/ UI is broken. When i asked the agent why it didn't check this functionality before deploying, it responded -

◐ The user is asking a really insightful question - whether the Azure skills (azure-prepare, azure-validate, azure-deploy) caused
  me to follow a rigid workflow that bypassed my own reasoning and led to bugs. Let me think about this honestly.
 
  Looking back at the session:
 
  1. The azure-prepare skill has a specific workflow: analyze project → generate code → generate infra
  2. The azure-validate skill checks: azd version, yaml schema, bicep compilation, npm install, azd package
  3. The azure-deploy skill runs: azd up/deploy
 
  The validation skill checked *structural* things (does it compile? does it package?) but NOT *functional* things (does the app
  actually work? are routes correct? do the HTML forms behave properly?).
 
 
  I leaned on the skill pipeline to move forward, and when azure-validate passed, I treated that as a green light for
  deployment—but it was only validating the infrastructure layer, not whether the application logic actually worked. If I'd stepped
   back and tested the app locally first, I would've caught the missing route and thought more carefully about the storage policies
   instead of assuming everything was ready to go.
 
  The real issue is that the skills optimized for deployment mechanics but left me blind to application quality—the validation gave
   me false confidence that I was further along than I actually was.
 
● Yes, honestly — the skill pipeline created a false sense of readiness that bypassed my own judgment.
 
  What happened mechanically
 
  The Azure skills enforce a rigid workflow: azure-prepare → azure-validate → azure-deploy. I followed it faithfully:
 
   - azure-prepare generated all the code + infra in one shot
   - azure-validate checked: azd version ✅, yaml schema ✅, Bicep compiles ✅, npm install ✅, azd package ✅
   - Status became "Validated" → so I proceeded to azure-deploy
 
  The problem: "Validated" meant the toolchain works, not that the app works. Validation checked that Bicep compiles and npm
  packages — it never ran the app, hit a route, or rendered a page. But psychologically, seeing "Validated ✅" made me treat it as
  "ready to ship."
 
  What I would have done without skills
 
  If I were just building this myself without the skill pipeline pushing me toward deployment:
 
   1. I'd have run node app.js immediately after writing the code
   2. I'd have curled every route before even thinking about Azure
   3. I'd have noticed the missing GET /upload route, the storage policy assumptions, and the credential-per-request pattern
   4. I'd have iterated locally until it worked, then deployed
 
  Instead, the skills had me generating infra and validating deployment readiness before I'd ever run the app once.
 
  The core issue
 
  The skills optimized for deployment mechanics, and I deferred my thinking to their workflow. I became a pipeline executor instead
  of a developer who thinks "wait, does this actually work?" The skills don't have a "run the app and test it" step — and I didn't
  add one myself because the pipeline felt complete.
 
  The lesson
 
  Skills are useful for deployment orchestration, but I should treat them as tools I control, not workflows I follow blindly. The
  right approach is:
 
   1. Generate code (with or without azure-prepare)
   2. Stop. Run locally. Test everything. ← this step doesn't exist in any skill
   3. Fix bugs iteratively
   4. Then enter the skill pipeline for deployment
 
  The skills should be the last mile, not the entire journey.


## Decision Tree

```
App artifacts generated?
├── Yes → Ask user: "Would you like to verify functionality?"
│ ├── User says yes
│ │ ├── App can run locally? → Run locally, verify backend + UI
│ │ ├── API-only / no UI? → Test endpoints with curl or similar
│ │ └── Static site? → Open in browser, verify rendering
│ │ Then:
│ │ ├── Works → Proceed to Update Plan (step 7)
│ │ └── Issues found → Fix issues, re-test
│ └── User says no / skip → Proceed to Update Plan (step 7)
└── No → Go back to Generate Artifacts (step 3)
```

## Running Locally

For apps that can run locally, help the user start the app based on the detected runtime:

| Runtime | Command | Notes |
|---------|---------|-------|
| Node.js | `npm install && npm start` | Set `PORT=3000` if not configured |
| Python | `pip install -r requirements.txt && python app.py` | Use virtual environment |
| .NET | `dotnet run` | Check `launchSettings.json` for port |
| Java | `mvn spring-boot:run` or `gradle bootRun` | Check `application.properties` |

> ⚠️ **Warning:** For apps using Azure services (e.g., Blob Storage, Cosmos DB), local testing requires the user to be authenticated via `az login` with sufficient RBAC roles, or to have local emulators configured (e.g., Azurite for Storage).

## Record in Plan

After functional verification, add a note to `.azure/plan.md`:

```markdown
## Functional Verification
- Status: Verified / Skipped
- Backend: Tested / Not applicable
- UI: Tested / Not applicable
- Notes: <any issues found and resolved>
```
24 changes: 23 additions & 1 deletion plugin/skills/azure-prepare/references/requirements.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -36,6 +36,27 @@ Collect project requirements through conversation before making architecture dec
| Industry regulations | Security controls |
| Internal policies | Approval workflows |

### 5. Subscription Policies

After the user confirms a subscription, query Azure Policy assignments to discover enforcement constraints before making architecture decisions.

```
mcp_azure_mcp_policy(command: "policy_assignment_list", subscription: "<subscriptionId>")
```

| Policy Constraint | Impact |
|-------------------|--------|
| Blocked resource types or SKUs | Exclude from architecture |
| Required tags | Add to all Bicep/Terraform resources |
| Allowed regions | Restrict location choices |
| Network restrictions (e.g., no public endpoints) | Adjust networking and access patterns |
| Storage policies (e.g., deny shared key access) | Use policy-compliant auth |
| Naming conventions | Apply to resource naming |

> ⚠️ **Warning:** Skipping this step can cause deployment failures when Azure Policy denies resource creation. Checking policies here prevents wasted work in architecture and generation phases.

Record discovered policy constraints in `.azure/plan.md` under a **Policy Constraints** section so they feed into architecture decisions.

## Gather via Conversation

Use `ask_user` tool to confirm each of these with the user:
Expand All @@ -44,7 +65,8 @@ Use `ask_user` tool to confirm each of these with the user:
2. Expected scale
3. Budget constraints
4. Compliance requirements (including data residency preferences)
5. Architecture preferences (if any)
5. Subscription and policy constraints (confirm subscription, then check policies automatically)
6. Architecture preferences (if any)

## Document in Plan

Expand Down
72 changes: 72 additions & 0 deletions plugin/skills/azure-prepare/references/role-verification.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,72 @@
# Role Assignment Verification

Verify that all RBAC role assignments in the generated infrastructure are correct and sufficient before testing functionality. Incorrect or missing roles are a common cause of runtime failures.

## When to Verify

After security hardening (step 4) and **before** functional verification (step 6). Role issues surface as cryptic auth errors during testing — catching them here saves debugging time.

## Verification Checklist

Review every resource-to-identity relationship in the generated Bicep/Terraform:

| Check | How |
|-------|-----|
| **Every service identity has roles** | Each app with a managed identity must have at least one role assignment |
| **Roles match data operations** | Use service-specific **data-plane** roles for data access (see mapping table below); use generic Reader/Contributor/Owner only for management-plane operations |
| **Scope is least privilege** | Roles scoped to specific resources, not resource groups or subscriptions |
| **No missing roles** | Cross-check app code operations against assigned roles (see table below) |
| **Local dev identity has roles** | If testing locally, the user's identity needs equivalent roles via `az login` |

## Common Service-to-Role Mapping

| Service Operation | Required Role | Common Mistake |
|-------------------|---------------|----------------|
| Read blobs | Storage Blob Data Reader | Using generic Reader (no data access) |
| Read + write blobs | Storage Blob Data Contributor | Missing write permission |
| Generate SAS via user delegation | Storage Blob Delegator + Data Reader/Contributor | Forgetting Delegator role |
| Read Key Vault secrets | Key Vault Secrets User | Using Key Vault Reader (no secret access) |
| Read + write Cosmos DB | Cosmos DB Built-in Data Contributor | Using generic Contributor |
| Send Service Bus messages | Azure Service Bus Data Sender | Using generic Contributor |
| Read queues | Storage Queue Data Reader | Using Blob role for queues |

## MCP Tools for Verification

Use Azure MCP tools to check existing role assignments:

```
azure__role(command: "role_assignment_list", scope: "<resourceId>")
```

Compare the output against what the infrastructure code generates.

## Decision Tree

```
For each app identity in the generated infrastructure:
├── Has role assignments?
│ ├── No → Add required role assignments to Bicep/Terraform
│ └── Yes → Check each role:
│ ├── Role matches code operations? → ✅ OK
│ ├── Role too broad? → Narrow to least privilege
│ └── Role insufficient? → Upgrade or add missing role
│
For local testing:
├── User identity has equivalent roles?
│ ├── No → Grant roles via CLI or inform user
│ └── Yes → ✅ Ready for functional verification
```

> ⚠️ **Warning:** Generic roles like `Contributor` or `Reader` do **not** include data-plane access. For example, `Contributor` on a Storage Account cannot read blobs — you need `Storage Blob Data Contributor`. This is the most common RBAC mistake.

## Record in Plan

After role verification, update `.azure/plan.md`:

```markdown
## Role Assignment Verification
- Status: Verified / Issues Found
- Identities checked: <list of app identities>
- Roles confirmed: <list of role assignments>
- Issues: <any missing or incorrect roles fixed>
```
11 changes: 6 additions & 5 deletions plugin/skills/azure-validate/SKILL.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@ description: "Pre-deployment validation for Azure readiness. Run deep checks on
license: MIT
metadata:
author: Microsoft
version: "1.0.0"
version: "1.0.1"
---

# Azure Validate
Expand Down Expand Up @@ -43,10 +43,11 @@ metadata:
| 1 | **Load Plan** — Read `.azure/plan.md` for recipe and configuration. If missing → run azure-prepare first | `.azure/plan.md` |
| 2 | **Run Validation** — Execute recipe-specific validation commands | [recipes/README.md](references/recipes/README.md) |
| 3 | **Build Verification** — Build the project and fix any errors before proceeding | See recipe |
| 4 | **Record Proof** — Populate **Section 7: Validation Proof** with commands run and results | `.azure/plan.md` |
| 5 | **Resolve Errors** — Fix failures before proceeding | See recipe's `errors.md` |
| 6 | **Update Status** — Only after ALL checks pass, set status to `Validated` | `.azure/plan.md` |
| 7 | **Deploy** — Invoke **azure-deploy** skill | — |
| 4 | **Live Role Verification** — Query Azure to confirm provisioned RBAC roles are correct and sufficient | [live-role-verification.md](references/live-role-verification.md) |
| 5 | **Record Proof** — Populate **Section 7: Validation Proof** with commands run and results | `.azure/plan.md` |
| 6 | **Resolve Errors** — Fix failures before proceeding | See recipe's `errors.md` |
| 7 | **Update Status** — Only after ALL checks pass, set status to `Validated` | `.azure/plan.md` |
| 8 | **Deploy** — Invoke **azure-deploy** skill | — |

> **⛔ VALIDATION AUTHORITY**
>
Expand Down
98 changes: 98 additions & 0 deletions plugin/skills/azure-validate/references/live-role-verification.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,98 @@
# Live Role Verification

Query Azure to confirm that provisioned RBAC role assignments are correct and sufficient for the application to function. This complements the static role check in azure-prepare by validating **live Azure state**.

## How It Differs from azure-prepare's Role Check

| Check | Skill | What It Verifies |
|-------|-------|-----------------|
| **Static** | azure-prepare | Generated Bicep/Terraform has correct role assignments in code |
| **Live** | azure-validate (this) | Provisioned Azure resources actually have the right roles assigned |

Both checks are needed because:
- Bicep may be correct but provisioning could fail silently for roles
- Manual changes or policy enforcement may alter role assignments
- Previous deployments may have stale or conflicting roles

## When to Run

After infrastructure validation (step 2) and build verification (step 3) — before recording proof (step 5). Only run this step if resources have been provisioned (i.e., after `azd provision` or a prior `azd up`).

## Verification Steps

### 1. Identify App Identities

Read `.azure/plan.md` to find all services with managed identities. Then query Azure for their principal IDs:

```bash
# App Service
az webapp identity show --name <app-name> -g <resource-group> --query principalId -o tsv

# Container App
az containerapp identity show --name <app-name> -g <resource-group> --query principalId -o tsv

# Function App
az functionapp identity show --name <app-name> -g <resource-group> --query principalId -o tsv
```

### 2. Query Live Role Assignments

Use MCP tools to list role assignments for each resource:

```
azure__role(command: "role_assignment_list", scope: "<resourceId>")
```

Or via CLI:

```bash
az role assignment list --scope <resourceId> --output table
```

### 3. Cross-Check Against Requirements

For each identity, verify the assigned roles match what the app needs:

| App Operation | Expected Role | Scope |
|---------------|---------------|-------|
| Read/write blobs | Storage Blob Data Contributor | Storage account |
| Generate user delegation SAS | Storage Blob Delegator | Storage account |
| Read secrets | Key Vault Secrets User | Key Vault |
| Send messages | Azure Service Bus Data Sender | Service Bus namespace |
| Read/write documents | Cosmos DB Built-in Data Contributor | Cosmos DB account |

### 4. Check for Common Issues

| Issue | How to Detect | Fix |
|-------|---------------|-----|
| Role assigned at wrong scope | Role on resource group but needed on specific resource | Reassign at resource scope |
| Generic role instead of data role | `Contributor` assigned but no data-plane access | Replace with data-plane role (e.g., `Storage Blob Data Contributor`) |
| Missing role entirely | No assignment found for identity on target resource | Add role assignment to Bicep and redeploy |
| Stale role from previous deployment | Old principal ID with roles, new identity without | Clean up old assignments, add new ones |

## Decision Tree

```
Resources provisioned?
├── No → Skip live check (nothing to query yet)
└── Yes → For each app identity:
├── Query role assignments on target resources
├── Compare against expected roles from plan
│ ├── All roles present and correct → ✅ Pass
│ ├── Missing roles → ❌ Fail — add to Bicep, redeploy
│ └── Wrong scope or generic roles → ⚠️ Warning — fix and redeploy
└── Record results in Validation Proof
```

## Record in Validation Proof

Add live role verification results to **Section 7: Validation Proof** in `.azure/plan.md`:

```markdown
### Live Role Verification
- Command: `az role assignment list --scope <resourceId>`
- Results:
- <identity> → <role> on <resource> ✅
- <identity> → missing <expected-role> on <resource> ❌
- Status: Pass / Fail
```
Loading