docs: document eval matrix

Author: montymxb

packages/langium-ai-tools/README.md

@@ -151,6 +151,298 @@ You can also define custom evaluators that are more tuned to the needs of your D
 
 In general we stick to focusing on what Langium can do to help with evaluation, but leave the opportunity open for you to extend, supplement, or modify evaluation logic as you see fit.
 
+### Evaluation Matrix
+
+The Evaluation Matrix provides a framework for testing multiple model configurations against a set of test cases using Langium AI evaluators. This is particularly helpful when comparing across models, prompt strategies, RAG setups, or other variations in your AI stack.
+
+In practice an evaluation matrix can be helpful when deciding between which models or services to use up front, but this can also be done externally by levering the evaluator directly yourself.
+
+#### Overview
+
+The evaluation matrix orchestrates three key components:
+- **Runners**: Functions that execute prompts against models, services, or other response generators
+- **Cases**: Test scenarios with expected outputs, which can be defined in code or loaded from YAML files
+- **Evaluators**: Metrics that score actual responses against expected responses
+
+The matrix fires off each runner against each test case, evaluates the results with all configured evaluators, and produces reports with aggregated metrics.
+
+#### Runners
+
+A runner is an interface that takes a prompt and message history, returning a response string. This serves as an abstraction so you can test your own setup as it stands
+
+Runners can wrap:
+- Direct model calls (Ollama, OpenAI, Anthropic, etc.)
+- RAG pipelines with vector database lookups
+- Multi-step agent workflows
+- Really any system that produces some text output
+
+```ts
+import { type Runner, type Message } from 'langium-ai-tools/evaluator';
+
+const myRunner: Runner = {
+    name: 'my-model',
+    runner: async (prompt: string, messages: Message[]) => {
+        // call your model/service here
+        const response = await myModel.generate(prompt, messages);
+        return response;
+    }
+};
+```
+
+#### Cases
+
+Test cases define the input and expected output for evaluation:
+
+```ts
+import { type EvalCase } from 'langium-ai-tools/evaluator';
+
+const testCase: EvalCase = {
+    name: 'hello-world-grammar',
+    prompt: 'Generate a Langium grammar for a simple hello world DSL',
+    expected_response: `grammar HelloWorld
+    entry Greeting: 'hello' name=ID;
+    terminal ID: /[_a-zA-Z][\\w_]*/;`,
+    // optional fields
+    history: [{
+        role: 'system',
+        content: 'You are an expert in Langium grammars.'
+    }],
+    tags: ['grammar', 'beginner'],
+    only_check_codeblocks: false
+};
+```
+
+Cases can also be loaded from YAML files for easier management:
+
+```yaml
+# eval-cases.yaml
+eval_cases:
+  - name: "hello-world-grammar"
+    prompt: "Generate a Langium grammar for a simple hello world DSL"
+    expected_response: |
+      grammar HelloWorld
+      entry Greeting: 'hello' name=ID;
+      terminal ID: /[_a-zA-Z][\w_]*/;
+    tags:
+      - "grammar"
+      - "beginner"
+```
+
+And can be loaded like so:
+
+```ts
+import { loadFromYaml } from 'langium-ai-tools/evaluator';
+import { readFileSync } from 'fs';
+
+const yamlContent = readFileSync('eval-cases.yaml', 'utf-8');
+const cases = loadFromYaml(yamlContent);
+```
+
+##### Evaluators
+
+Evaluators score responses against expected outputs. You can use built-in evaluators or create custom ones:
+
+```ts
+import { LangiumEvaluator, mergeEvaluators } from 'langium-ai-tools/evaluator';
+import { createMyDSLServices } from './my-dsl';
+
+const services = createMyDSLServices(EmptyFileSystem).MyDSL;
+
+// use the built-in Langium evaluator
+const langiumEval = new LangiumEvaluator(services);
+
+// or create a custom evaluator
+class CustomEvaluator extends Evaluator {
+    async evaluate(response: string, expected: string): Promise<Partial<EvaluatorResult>> {
+        return {
+            data: {
+                custom_metric: calculateMetric(response, expected)
+            }
+        };
+    }
+}
+
+// merge multiple evaluators
+const combinedEval = mergeEvaluators(
+    langiumEval,
+    new CustomEvaluator()
+);
+```
+
+#### Setting Up an Evaluation Matrix
+
+Create and run an evaluation matrix by combining your runners, cases, and evaluators:
+
+```ts
+import { EvalMatrix } from 'langium-ai-tools/evaluator';
+
+const matrix = new EvalMatrix({
+    config: {
+        name: 'Model Comparison',
+        description: 'Comparing different models for DSL generation',
+        history_folder: '.eval-history',
+        num_runs: 3  // number of times to run each combination
+    },
+    runners: [
+        myModelRunner,
+        myModelWithRAGRunner,
+        competitorModelRunner
+    ],
+    evaluators: [
+        {
+            name: 'Langium Parser + Validation',
+            eval: new LangiumEvaluator(services)
+        },
+        {
+            name: 'Edit Distance',
+            eval: new EditDistanceEvaluator()
+        }
+    ],
+    cases: testCases
+});
+
+// run the matrix and get results
+const results = await matrix.run();
+```
+
+#### Configuration Options
+
+The `EvalMatrixConfig` provides several options:
+
+- **name**: Descriptive name for this evaluation run
+- **description**: Longer description of what's being evaluated
+- **history_folder**: Directory where results will be saved as timestamped JSON files
+- **num_runs**: Number of times to run each runner-case-evaluator combination (for averaging)
+
+#### Working with Results
+
+The matrix produces detailed results that include:
+
+```ts
+type EvaluatorResult = {
+    // combined runner-case-evaluator name
+    name: string;
+    metadata: {
+        runner: string;
+        evaluator: string;
+        testCase: EvalCase;
+        actual_response: string;
+        // runtime in seconds
+        duration: number;
+        run_count: number;
+    };
+    data: {
+        // runtime in seconds
+        runtime: number;
+        // evaluator-specific metrics (errors, warnings, edit_distance, etc.)
+        ...
+    };
+};
+```
+
+Results are automatically saved to the history folder with timestamps. You can then process these results using utility functions:
+
+```ts
+import {
+    averageAcrossCases,
+    averageAcrossRunners,
+    loadLastResults
+} from 'langium-ai-tools/evaluator';
+
+// average results across multiple runs of the same case
+const avgResults = averageAcrossCases(results);
+
+// average across all cases for each runner
+const runnerAvgs = averageAcrossRunners(results);
+
+// load results from previous runs
+const lastResults = loadLastResults('.eval-history');
+const last3Runs = loadLastResults('.eval-history', 3);
+
+// display results
+console.table(avgResults.map(r => ({
+    name: r.name,
+    ...r.data
+})));
+```
+
+#### Complete Example
+
+And here's a full example combining all the parts listed above:
+
+```ts
+import {
+    EvalMatrix,
+    LangiumEvaluator,
+    averageAcrossCases,
+    type Runner,
+    type EvalCase
+} from 'langium-ai-tools/evaluator';
+import { createMyDSLServices } from './my-dsl';
+import { EmptyFileSystem } from 'langium';
+import ollama from 'ollama';
+
+// setup services
+const services = createMyDSLServices(EmptyFileSystem).MyDSL;
+
+// define runners
+const baseRunner: Runner = {
+    name: 'llama3.2-base',
+    runner: async (prompt: string, messages: Message[]) => {
+        const response = await ollama.chat({
+            model: 'llama3.2',
+            messages: [...messages, { role: 'user', content: prompt }]
+        });
+        return response.message.content;
+    }
+};
+
+// define test cases
+const cases: EvalCase[] = [
+    {
+        name: 'simple-grammar',
+        prompt: 'Create a grammar for arithmetic expressions',
+        expected_response: `grammar Arithmetic
+        entry Expression: Addition;
+        Addition: Multiplication ({infer BinaryExpression.left=current} operator=('+' | '-') right=Multiplication)*;
+        Multiplication: Primary ({infer BinaryExpression.left=current} operator=('*' | '/') right=Primary)*;
+        Primary: '(' Expression ')' | value=NUMBER;
+        terminal NUMBER returns number: /[0-9]+/;`
+    },
+    // more cases...
+];
+
+// create and run matrix
+const matrix = new EvalMatrix({
+    config: {
+        name: 'DSL Generation Test',
+        description: 'Testing model performance on grammar generation',
+        history_folder: '.eval-results',
+        num_runs: 5
+    },
+    runners: [baseRunner],
+    evaluators: [{
+        name: 'Langium Evaluator',
+        eval: new LangiumEvaluator(services)
+    }],
+    cases
+});
+
+const results = await matrix.run();
+
+// process and display results
+const averaged = averageAcrossCases(results);
+console.log('\nAveraged Results:');
+console.table(averaged.map(r => ({
+    name: r.name,
+    errors: r.data.errors,
+    warnings: r.data.warnings,
+    runtime: r.data.runtime
+})));
+```
+
+For more complete examples, see the [example-dsl-evaluator](../examples/example-dsl-evaluator) project.
+
 ## Contributing
 
 If you want to help feel free to open an issue or a PR. As a general note we're open to accept changes that focus on improving how we can support AI application development for Langium DSLs. But we don't want to provide explicit bindings to actual services/providers at this time, such as LLamaIndex, Ollama, LangChain, or others. Similarly this package doesn't provide direct bindings for AI providers such as OpenAI and Anthropic here. Instead these changes will go into a separate package under Langium AI that is intended for this purpose.