Skip to content

Eval-driven content fixes: #31, #23, #24, #29 (vally 0.8.0) - #36

Open
David Pine (IEvangelist) wants to merge 7 commits into
mainfrom
dapine/skill-evals-content
Open

Eval-driven content fixes: #31, #23, #24, #29 (vally 0.8.0)#36
David Pine (IEvangelist) wants to merge 7 commits into
mainfrom
dapine/skill-evals-content

Conversation

@IEvangelist

@IEvangelist David Pine (IEvangelist) commented Jun 29, 2026

Copy link
Copy Markdown
Member

Content fixes driven by the vally 0.8.0 eval loop

Stacked on #35 (base branch dapine/skill-evals-vally). Rebased onto the updated
#35 tip, so this branch now inherits the vally 0.8.0 migration and the
scoring: threshold: 0.7 gate blocks on every spec (the missing scoring blocks
were the earlier eval-CI failure cause).

Each fix follows the eval-first red→green loop: add a capability stimulus that
captures the desired behavior, fix the skill content, and verify the stimulus
passes before committing. Rebase onto main once #35 merges.

Issues addressed

Commit Issue Summary
b028315 #31 aspireify: reduce await noise in TypeScript AppHost examples — de-await chainable add*/with* calls across the TS/JS references, add a "when you actually need await" section + a Hard Rules anti-pattern row, and a fluent-style stimulus.
915bbf1 #23 aspireify: document the JS monorepo shared-package build pattern — a one-shot shared-build resource consumers waitForCompletion on, waitForCompletion vs waitFor, and the tsc --incremental skip-emit pitfall + prebuild clean fix.
6172885 #24 aspire: add an Aspire 13.4 breaking-changes reference (adopting the distilled content from #25) plus the PostgreSQL 17→18 data-volume incompatibility, wired into the router's References.
d080ff3 #29 aspireify: add Bicep / ARM IaC → AppHost conversion guidance — resource→AddAzure* map, AddBicepTemplate fallback, and the provision-new vs reference-existing decision, with a grounded infra/main.bicep fixture.
6be4a93 #31 (follow-up) aspireify: explain why un-awaited TS resource vars are still type-correct.

Verification

vally lint skills → 6/6 passing; every spec carries scoring: threshold: 0.7.
Capability stimuli are grounded with real workspace fixtures so the executor
activates the skill and reads the relevant reference rather than answering from
memory. Model-backed eval gating runs in CI on vally 0.8.0.

Fixes #31
Fixes #23
Fixes #24
Fixes #29

@IEvangelist David Pine (IEvangelist) changed the title Eval-driven content fixes: #31, #23, #24, #29 Eval-driven content fixes: #31, #23, #24, #29 (vally 0.8.0) Jul 13, 2026
@IEvangelist
David Pine (IEvangelist) force-pushed the dapine/skill-evals-content branch 2 times, most recently from 235d5a2 to 46a0293 Compare July 16, 2026 18:54
@IEvangelist
David Pine (IEvangelist) force-pushed the dapine/skill-evals-content branch 3 times, most recently from ba6e748 to ef6c30c Compare July 29, 2026 17:53
David Pine (IEvangelist) and others added 6 commits July 30, 2026 06:44
TypeScript `add*`/`with*` builder methods return a fluent `*ResourcePromise`
that is both a `PromiseLike` and a chainable builder. Only `createBuilder()`
and `build().run()` actually need `await`; intermediate calls should be
assigned to a `const` and chained directly. The examples over-used `await`,
teaching an awkward orchestration-flavored style.

- typescript-authoring.md: de-await skeleton + 10 example snippets; add a new
  "Promise-fluent return types — when you actually need `await`" section and a
  Hard Rules anti-pattern row.
- apphost-wiring.md: de-await WithReference, cross-service env wiring,
  container lifetime, and data-volume TS snippets.
- javascript-apps.md: de-await dev-script and workspace-monorepo snippets.
- SKILL.md: de-await the TS column of the C#-vs-TS quick reference.
- evals/eval.yaml: add capability stimulus `aspireify-fluent-ts-001`
  (tag issue=31) codifying the fluent no-over-await style via
  output-not-contains (`await builder.add`, `(await builder`) plus a prompt
  grader; verified green locally (5/5 graders, 2 skills loaded).

Fixes #31

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Brownfield JS/TS monorepos commonly have a source-linked shared package
(`packages/shared` compiled to `dist/`) consumed by sibling services/apps via
npm/yarn/pnpm workspaces. The naive wiring "works" until `tsc --incremental`
silently skips emit, leaving `dist/` empty — consumers then fail at first
request with a runtime error that looks like an Aspire bug.

- javascript-apps.md: new "Shared library packages in monorepos" section —
  model the build as a one-shot `addJavaScriptApp` resource that consumers
  `.waitForCompletion()` on; explain `waitForCompletion` vs `waitFor`; a
  prominent up-front warning plus a dedicated "tsc incremental emit pitfall"
  subsection with the `prebuild` clean-step fix (and `tsc --build --clean`
  alternative); and a "when NOT to use this pattern" note for registry-published
  packages.
- SKILL.md: add a Scan-table heuristic row (root `workspaces` + a member whose
  `main`/`exports` points at `dist/` depended on by another member) and a
  Propose callout, both pointing at the new section.
- evals/eval.yaml: add capability stimulus `aspireify-monorepo-shared-001`
  (tag issue=23) requiring the shared-build + waitForCompletion wiring AND the
  tsc-incremental pitfall warning; verified green locally (2/2 runs, 4/4
  graders).

Fixes #23

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
`aspire agent init` on the 13.4 CLI only shipped the 13.3 breaking-changes
scrub list, so agents had no 13.4 reference to scrub generated code, scripts,
CI, or deployment snippets against.

- references/aspire-13-4-breaking-changes.md: new agent-facing scrub list
  sourced from the 13.4 release notes — `aspire exec` removal, `aspire ps`
  flag removals, TypeScript `.aspire/modules/` consolidation, persistent
  executable/project lifetimes, Kubernetes `WithRoute`→`WithPath` + path-type
  split + external-endpoint requirement, `WithHelm` consolidation, Azure Front
  Door naming, Foundry `PublishAsHostedAgent`→`WithComputeEnvironment` and the
  `AddPromptAgent` parameter reorder, `PublishAsNpmPackageScript`→
  `PublishAsPackageScript`, Keycloak HTTPS, named resource-command options,
  pre-startup TS validation, `Aspire.Hosting.Testing` HTTPS default, RabbitMQ
  4.3, `AddNatsClient`, the **PostgreSQL 17→18 data-volume incompatibility**,
  and a 13.3→13.4 migration checklist.
- references/aspire-breaking-changes.md: new version-aware index that points at
  the right per-version scrub list with upstream release-notes fallbacks.
- SKILL.md: wire both new references into the References section.
- evals/eval.yaml: add capability stimulus `router-upgrade-13-4-001`
  (tag issue=24) requiring specific 13.4 breaking changes, the Postgres 18
  data-volume warning, and the `aspire update --self` → `aspire update` flow;
  verified green locally (2/2 runs, 3/3 graders).

Fixes #24

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Add a reference for converting exported Azure infrastructure-as-code
(hand-written or `az group export` Bicep / ARM) into an Aspire AppHost:

- New references/bicep-to-apphost.md: conversion workflow, an authoritative
  Bicep `type` → `AddAzure*` integration map, `ConfigureInfrastructure`
  tweaks, the custom `AddBicepTemplate` fallback for resources Aspire does
  not model natively, threading Bicep `outputs` as references, "what not to
  port", a decision tree, and validation via `aspire publish` diffing.
- Lead the reference with an IMPORTANT callout covering the two steps agents
  most often miss: map every natively-supported resource to its `AddAzure*`
  API, and always raise provision-new vs reference-existing for production
  infrastructure (`AsExisting` / `PublishAsExisting` / `AddConnectionString`).
- Wire the reference into SKILL.md: a Scan-table row for Azure IaC, an
  imperative "you MUST open bicep-to-apphost.md before proposing" callout, a
  Propose clarifying question, and References-list entries.
- Add capability stimulus `aspireify-bicep-convert-001` plus an
  `evals/azure-bicep/infra/main.bicep` fixture (storage + Service Bus +
  Azure Maps) so the agent converts real IaC rather than answering from
  memory. Verified green: 3/3 trials, all five graders pass.

Fixes #29

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…rect

Issue #31 reduced `await` noise in the TypeScript AppHost examples, but the
reference didn't explain *why* passing an un-awaited resource variable to a
fluent API (e.g. `withReference(db)`, `waitFor(api)`) is type-correct — only
that it works at runtime. That left room to second-guess whether assigned
variables still need `await` for their types to line up.

Spell out the type-system reason, grounded in the generated SDK: every
resource-accepting parameter is generated as `Awaitable<T> = T | PromiseLike<T>`
and each `*ResourcePromise` is a `PromiseLike<TheResource>`, so a plain
`const db = builder.addPostgres('pg')` is the intended, fully-typed form. Also
clarify that the two remaining awaits (`createBuilder()` and
`builder.build().run()`) are about execution, not types.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The stimulus asked to wire ServiceDefaults into 'each of my projects' but its environment shipped only MyApp.AppHost/Program.cs. With no service Program.cs to edit, the agent hunted for the Api/Worker projects and wandered past the 600s trial timeout (~1/3 trials), tripping evalHadExecutionErrors and hard-failing the whole eval.

- Add minimal MyApp.ApiService/Program.cs and MyApp.Web/Program.cs fixtures (the 'before' state, no ServiceDefaults wired) plus MyApp.Web.csproj so the canonical csharp-apphost fixture matches what the AppHost already references (apiservice + webfrontend).

- Mount the two service Program.cs files and the ServiceDefaults .csproj into the stimulus so the agent has concrete files to edit instead of hunting.

- Align the adds_service_defaults grader to the AppHost's real projects (ApiService and Web, not a phantom Worker).

- Add 'scoring: binary' to the four yes/no prompt graders; under the default scale_1_5 an adequate answer normalised to 0.5, below the 0.7 threshold. Mirrors the router/orchestration hardening in 5ca3eaf.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Copilot-Session: 788994fa-ffbf-4722-a7b8-0dcc3b174dbf
@IEvangelist
David Pine (IEvangelist) changed the base branch from dapine/skill-evals-vally to main July 30, 2026 11:45
@IEvangelist
David Pine (IEvangelist) marked this pull request as ready for review July 30, 2026 11:45
Copilot AI review requested due to automatic review settings July 30, 2026 11:45
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR updates the aspireify and aspire skill content to address eval-driven documentation gaps and correctness issues introduced/validated via the vally 0.8.0 evaluation loop, adding new references and new stimuli/fixtures to gate the intended behaviors.

Changes:

  • Updates TypeScript AppHost guidance to prefer fluent *ResourcePromise chaining (minimizing unnecessary await) and adds new eval stimuli to enforce the pattern.
  • Adds JavaScript/TypeScript monorepo guidance for source-linked shared packages (one-shot build resource + waitForCompletion, with tsc --incremental caveat) and corresponding eval stimulus.
  • Adds new references for Bicep/ARM IaC → AppHost conversion and Aspire 13.4 breaking-change scrubs, plus eval/fixture updates to validate routing and upgrade guidance.

Reviewed changes

Copilot reviewed 14 out of 14 changed files in this pull request and generated 2 comments.

Show a summary per file
File Description
skills/aspireify/SKILL.md Adds Bicep conversion reference + scan/propose heuristics; updates TS quick reference to de-await addProject.
skills/aspireify/references/typescript-authoring.md Rewrites TS examples to avoid over-await; adds an explanation section about fluent promise types.
skills/aspireify/references/javascript-apps.md De-awaits examples; adds “Shared library packages in monorepos” guidance + pitfall callouts.
skills/aspireify/references/bicep-to-apphost.md New reference for mapping Bicep/ARM resources to native Aspire APIs with custom-template fallbacks.
skills/aspireify/references/apphost-wiring.md Updates TS wiring snippets to remove unnecessary await.
skills/aspireify/evals/eval.yaml Adds/updates stimuli and binary-scored graders for fluent TS style, monorepo shared builds, and IaC conversion.
skills/aspire/SKILL.md Adds references to the breaking-changes index + new 13.4 scrub list.
skills/aspire/references/aspire-breaking-changes.md New version-index reference for Aspire breaking changes.
skills/aspire/references/aspire-13-4-breaking-changes.md New Aspire 13.4 breaking-change scrub list (agent-facing).
skills/aspire/evals/eval.yaml Adds a 13.4 upgrade stimulus to validate specific breaking-change guidance (incl. PG data volumes).
evals/csharp-apphost/MyApp.Web/Program.cs New fixture content used by updated aspireify ServiceDefaults stimulus.
evals/csharp-apphost/MyApp.Web/MyApp.Web.csproj New web project fixture for eval workspace context.
evals/csharp-apphost/MyApp.ApiService/Program.cs New API service fixture for eval workspace context.
evals/azure-bicep/infra/main.bicep New Bicep fixture for IaC→AppHost conversion stimulus.
Comments suppressed due to low confidence (1)

skills/aspireify/references/typescript-authoring.md:232

  • The YARP Routing snippet uses web.getEndpoint('http') but web is not declared in the example, making the snippet incomplete.
const api = builder.addProject('api', '../Api/Api.csproj');
const yarp = builder.addYarp('gateway')
    .addRoute('/api/{**catch-all}', api.getEndpoint('http'))
    .addCatchAllRoute(web.getEndpoint('http'));

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

| `Microsoft.EventHub/namespaces` | `AddAzureEventHubs("eh")` | |
| `Microsoft.SignalRService/signalR` | `AddAzureSignalR("signalr")` | |
| `Microsoft.SignalRService/webPubSub` | `AddAzureWebPubSub("wps")` | |
| `Microsoft.CognitiveServices/accounts` (OpenAI/Foundry) | `AddAzureOpenAI("openai")` / `AddAzureAIFoundry(...)` | confirm name per release |
Comment on lines +92 to +103
```ts
const api = builder.addProject('api', '../Api/Api.csproj')
.withReference(db)
.waitFor(db)
.withExternalHttpEndpoints();

builder.addViteApp('web', '../web')
.withEnvironment('VITE_API_URL', api.getEndpoint('http'))
.waitFor(api)
.withExternalHttpEndpoints()
.withBrowserLogs();
```
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

2 participants