diff --git a/docs/memory-system.md b/docs/memory-system.md new file mode 100644 index 00000000..808ed7a2 --- /dev/null +++ b/docs/memory-system.md @@ -0,0 +1,330 @@ +# Memory System + +Antfarm's memory system prevents the three failure modes of long-running agent workflows: missed writes, missed retrieval, and compaction loss. It provides explicit memory contracts, working set control, checkpoint persistence, and semantic search. + +--- + +## Overview + +When agents run in fresh sessions (the Ralph pattern), they lose access to previous context. The memory system solves this by: + +1. **Explicit Memory Contracts** - Structured persistence for constraints, decisions, context, and checkpoints +2. **Working Set Control** - LRU eviction with priority boosting prevents unbounded growth +3. **Checkpoint System** - Automatic save/restore of workflow state +4. **QMD Query Layer** - Semantic search with ranked results + +--- + +## Memory Contract Types + +### Constraint +Hard rules that must be respected by all future steps. + +```typescript +MemoryContract.setContract( + runId, + 'constraint', + 'database_schema', + { tables: ['users', 'posts'], orm: 'prisma' }, + 10 // high priority +); +``` + +### Decision +Recorded decisions that future steps should know about. + +```typescript +MemoryContract.setContract( + runId, + 'decision', + 'auth_strategy', + { method: 'jwt', library: 'auth.js' }, + 8 +); +``` + +### Context +General information about the codebase or task. + +```typescript +MemoryContract.setContract( + runId, + 'context', + 'tech_stack', + { frontend: 'react', backend: 'express', db: 'postgresql' }, + 5 +); +``` + +### Checkpoint +Workflow state for resumption after interruption. + +```typescript +MemoryContract.createCheckpoint(runId, stepId, { + completedStories: ['story-1', 'story-2'], + currentStory: 'story-3', + branch: 'feature/auth' +}); +``` + +--- + +## Working Set Control + +Each run has a maximum working set size (default: 100 contracts). When exceeded, lowest-priority oldest contracts are evicted. + +### Priority Levels + +| Priority | Use Case | +|----------|----------| +| 10 | Critical constraints (schema, API contracts) | +| 8-9 | Important decisions (architecture, libraries) | +| 5-7 | Context (tech stack, conventions) | +| 1-4 | Temporary data, low-priority context | + +### Access Tracking + +Contracts track access count and last accessed time. Frequently accessed contracts get priority boosts in the working set. + +```typescript +// Get working set with LRU + priority boosting +const workingSet = MemoryContract.getWorkingSet(runId, 20); +``` + +--- + +## Checkpoint System + +Checkpoints automatically save on step completion and restore on step claim. + +### Auto-Save + +When a step completes, the workflow runner creates a checkpoint with: +- Current step ID +- Completed stories +- In-progress story +- Branch name +- Any step-specific state + +### Auto-Restore + +When claiming a step, the runner checks for checkpoints: + +```typescript +const checkpoint = MemoryContract.getLatestCheckpoint(runId); +if (checkpoint) { + // Resume from checkpoint +} +``` + +### Retention + +Only the last 10 checkpoints per run are kept. Older checkpoints are pruned automatically. + +--- + +## QMD Query Layer + +The Query-Model-Data layer provides semantic search over memory contracts. + +### Search Strategies + +1. **Exact Key Match** - Score 1.0 +2. **Key Contains Query** - Score 0.9 +3. **Value Contains Query** - Score 0.7 +4. **N-gram Semantic Similarity** - Score 0.5-0.8 + +### Usage + +```typescript +import { QMD } from './lib/qmd.js'; + +// Search with semantic ranking +const results = QMD.query(runId, 'authentication', { + type: 'constraint', + minPriority: 5, + limit: 10, + minScore: 0.3 +}); + +// Find related contracts +const related = QMD.findRelated(runId, 'auth_strategy', 2); + +// Get suggestions based on context +const suggestions = QMD.suggestRelevant(runId, 'implementing user login', 5); +``` + +--- + +## CLI Commands + +The memory CLI provides debugging and management tools: + +```bash +# List all contracts for a run +antfarm memory list + +# Get a specific contract +antfarm memory get + +# Search by keyword +antfarm memory search + +# QMD semantic search +antfarm memory query + +# Show session operation log +antfarm memory log + +# Show latest checkpoint +antfarm memory checkpoint + +# Clear all memory for a run +antfarm memory clear +``` + +--- + +## Workflow Integration + +### Template Variables + +Workflow steps automatically receive memory context: + +```yaml +input: | + REQUIRED_MEMORY_SEARCH: + Working Set: {{memory_working_set}} + Constraints: {{memory_constraints}} + Decisions: {{memory_decisions}} + Checkpoint: {{checkpoint_step}} +``` + +### Agent Acknowledgment + +Agents must acknowledge constraints before proceeding: + +``` +MEMORY_ACKNOWLEDGED: database_schema (prisma), auth_strategy (jwt) +``` + +This forced acknowledgment prevents agents from ignoring critical constraints. + +--- + +## Migration Guide + +### Existing Workflows + +To add memory support to an existing workflow: + +1. **Add REQUIRED_MEMORY_SEARCH section** to each step template: + +```yaml +input: | + REQUIRED_MEMORY_SEARCH: + Working Set: {{memory_working_set}} + Constraints: {{memory_constraints}} + Decisions: {{memory_decisions}} + Checkpoint: {{checkpoint_step}} + + MEMORY_ACKNOWLEDGED: (list which constraints/decisions you considered) +``` + +2. **Add memory acknowledgment requirement** to instructions: + +```yaml +Instructions: +1. Review memory constraints and decisions above +2. Acknowledge key constraints in MEMORY_ACKNOWLEDGED +3. ... rest of instructions +``` + +3. **Use key_decision and key_constraint outputs**: + +Agents should output decisions/constraints for future steps: + +``` +STATUS: done +key_decision: Using zod for validation (affects all future API contracts) +key_constraint: Database schema uses camelCase columns +``` + +### Database Migration + +The memory tables are created automatically on first use. No manual migration needed. + +--- + +## Best Practices + +1. **Set high priorities for constraints** - Schema changes, API contracts +2. **Record decisions immediately** - Don't wait, write while context is fresh +3. **Use descriptive keys** - `auth_strategy` not `decision_1` +4. **Keep values structured** - JSON objects, not prose +5. **Acknowledge in outputs** - Always list what constraints were considered +6. **Query before assuming** - Use QMD to find related context + +--- + +## Troubleshooting + +### Contracts Not Found + +Check the session log: + +```bash +antfarm memory log +``` + +Look for `read` operations with `✗` status. + +### Working Set Too Small + +Increase the limit: + +```typescript +const contracts = MemoryContract.getAllForRun(runId, 200); +``` + +### Checkpoints Not Restoring + +Verify checkpoint exists: + +```bash +antfarm memory checkpoint +``` + +Check that the workflow runner is calling `getLatestCheckpoint` on step claim. + +--- + +## API Reference + +### MemoryContract + +| Method | Description | +|--------|-------------| +| `setContract(runId, type, key, value, priority?)` | Create or update a contract | +| `getContract(runId, key)` | Retrieve a contract by key | +| `searchContracts(runId, type)` | Get all contracts of a type | +| `getAllForRun(runId, limit?)` | Get all contracts for a run | +| `getByPriority(runId, minPriority)` | Filter by priority threshold | +| `searchByKeyword(runId, keyword)` | Substring search | +| `getWorkingSet(runId, maxSize?)` | LRU + priority ranked contracts | +| `createCheckpoint(runId, stepId, data)` | Save checkpoint | +| `getLatestCheckpoint(runId)` | Get most recent checkpoint | +| `clearRun(runId)` | Delete all data for a run | +| `getSessionLog(runId, limit?)` | Get operation audit log | + +### QMD + +| Method | Description | +|--------|-------------| +| `query(runId, query, options?)` | Semantic search with ranking | +| `findRelated(runId, key, depth?)` | Graph traversal for related contracts | +| `suggestRelevant(runId, context, limit?)` | Context-based suggestions | + +--- + +*Part of the Antfarm workflow system. See [creating-workflows.md](./creating-workflows.md) for workflow authoring.* diff --git a/package-lock.json b/package-lock.json index 3fa58666..a1ad386e 100644 --- a/package-lock.json +++ b/package-lock.json @@ -1,12 +1,12 @@ { "name": "antfarm", - "version": "0.4.1", + "version": "0.5.1", "lockfileVersion": 3, "requires": true, "packages": { "": { "name": "antfarm", - "version": "0.4.1", + "version": "0.5.1", "dependencies": { "json5": "^2.2.3", "yaml": "^2.4.5" diff --git a/src/cli/cli.ts b/src/cli/cli.ts index b69ee8fa..c2b815cc 100755 --- a/src/cli/cli.ts +++ b/src/cli/cli.ts @@ -89,6 +89,13 @@ function printUsage() { "antfarm install Install all bundled workflows", "antfarm uninstall [--force] Full uninstall (workflows, agents, crons, DB)", "", + "antfarm memory list List memory contracts for a run", + "antfarm memory get Get specific memory contract", + "antfarm memory search Search contracts by keyword", + "antfarm memory log Show session operation log", + "antfarm memory checkpoint Show latest checkpoint", + "antfarm memory clear Clear all memory for a run", + "", "antfarm workflow list List available workflows", "antfarm workflow install Install a workflow", "antfarm workflow uninstall Uninstall a workflow (blocked if runs active)", @@ -449,6 +456,12 @@ async function main() { return; } + if (group === "memory") { + const { memoryCommand } = await import("./memory-commands.js"); + const exitCode = memoryCommand(args.slice(1)); + process.exit(exitCode); + } + if (args.length < 2) { printUsage(); process.exit(1); } if (group !== "workflow") { printUsage(); process.exit(1); } diff --git a/src/cli/memory-commands.ts b/src/cli/memory-commands.ts new file mode 100644 index 00000000..be1c0afe --- /dev/null +++ b/src/cli/memory-commands.ts @@ -0,0 +1,163 @@ +import { MemoryContract } from "../lib/memory.js"; +import { QMD } from "../lib/qmd.js"; +import { getDb } from "../db.js"; + +/** + * Memory CLI commands for debugging and management + */ +export function memoryCommand(args: string[]): number { + const subcommand = args[0]; + const runId = args[1]; + + switch (subcommand) { + case "list": + return listContracts(runId); + case "get": + return getContract(runId, args[2]); + case "search": + return searchMemory(runId, args[2]); + case "query": + return queryQMD(runId, args[2]); + case "log": + return showLog(runId); + case "checkpoint": + return showCheckpoint(runId); + case "clear": + return clearRun(runId); + default: + console.log(`Usage: antfarm memory [options] +Commands: + list List all contracts for a run + get Get specific contract by key + search Search contracts by keyword + query QMD semantic search with ranking + log Show session operation log + checkpoint Show latest checkpoint + clear Clear all memory for a run +`); + return 1; + } +} + +function listContracts(runId: string | undefined): number { + if (!runId) { + console.error("Error: run-id required"); + return 1; + } + const contracts = MemoryContract.getAllForRun(runId, 100); + console.log(`\nMemory contracts for run ${runId}:`); + console.log("-".repeat(80)); + console.log(`${"Type".padEnd(12)} ${"Key".padEnd(30)} ${"Priority".padEnd(10)} ${"Accessed"}`); + console.log("-".repeat(80)); + for (const c of contracts) { + const accessed = c.access_count > 0 ? `${c.access_count}x` : "-"; + console.log(`${c.contract_type.padEnd(12)} ${c.key.slice(0, 30).padEnd(30)} ${String(c.priority).padEnd(10)} ${accessed}`); + } + console.log(`\nTotal: ${contracts.length} contracts`); + return 0; +} + +function getContract(runId: string | undefined, key: string | undefined): number { + if (!runId || !key) { + console.error("Error: run-id and key required"); + return 1; + } + const contract = MemoryContract.getContract(runId, key); + if (!contract) { + console.log(`No contract found for key: ${key}`); + return 1; + } + console.log(`\nContract: ${key}`); + console.log("-".repeat(50)); + console.log(`Type: ${contract.contract_type}`); + console.log(`Priority: ${contract.priority}`); + console.log(`Created: ${contract.created_at}`); + console.log(`Accessed: ${contract.access_count || 0} times`); + console.log(`\nValue:`); + console.log(JSON.stringify(contract.value, null, 2)); + return 0; +} + +function searchMemory(runId: string | undefined, keyword: string | undefined): number { + if (!runId || !keyword) { + console.error("Error: run-id and keyword required"); + return 1; + } + const results = MemoryContract.searchByKeyword(runId, keyword); + console.log(`\nSearch results for "${keyword}" in run ${runId}:`); + console.log("-".repeat(80)); + for (const c of results) { + console.log(`[${c.contract_type}] ${c.key}`); + } + console.log(`\nTotal: ${results.length} results`); + return 0; +} + +function showLog(runId: string | undefined): number { + if (!runId) { + console.error("Error: run-id required"); + return 1; + } + const logs = MemoryContract.getSessionLog(runId, 50); + console.log(`\nSession log for run ${runId}:`); + console.log("-".repeat(80)); + console.log(`${"Time".padEnd(20)} ${"Operation".padEnd(15)} ${"Key".padEnd(25)} ${"Status"}`); + console.log("-".repeat(80)); + for (const log of logs) { + const time = new Date(log.timestamp).toLocaleTimeString(); + const status = log.success ? "✓" : "✗"; + console.log(`${time.padEnd(20)} ${log.operation.padEnd(15)} ${(log.key || "").slice(0, 25).padEnd(25)} ${status}`); + } + return 0; +} + +function showCheckpoint(runId: string | undefined): number { + if (!runId) { + console.error("Error: run-id required"); + return 1; + } + const checkpoint = MemoryContract.getLatestCheckpoint(runId); + if (!checkpoint) { + console.log("No checkpoint found"); + return 1; + } + console.log(`\nLatest checkpoint for run ${runId}:`); + console.log("-".repeat(50)); + console.log(`Step: ${checkpoint.step_id}`); + console.log(`Created: ${checkpoint.created_at}`); + console.log(`\nData:`); + console.log(JSON.stringify(checkpoint.checkpoint_data, null, 2)); + return 0; +} + +function clearRun(runId: string | undefined): number { + if (!runId) { + console.error("Error: run-id required"); + return 1; + } + console.log(`Clearing all memory for run ${runId}...`); + MemoryContract.clearRun(runId); + console.log("Done."); + return 0; +} + +function queryQMD(runId: string | undefined, query: string | undefined): number { + if (!runId || !query) { + console.error("Error: run-id and query required"); + return 1; + } + const results = QMD.query(runId, query, { limit: 10, minScore: 0.2 }); + console.log(`\nQMD query results for "${query}" in run ${runId}:`); + console.log("-".repeat(80)); + console.log(`${"Score".padEnd(8)} ${"Strategy".padEnd(15)} ${"Type".padEnd(12)} ${"Key"}`); + console.log("-".repeat(80)); + for (const r of results) { + const score = (r.score * 100).toFixed(0) + "%"; + console.log(`${score.padEnd(8)} ${r.strategy.padEnd(15)} ${r.contract.contract_type.padEnd(12)} ${r.contract.key}`); + } + if (results.length === 0) { + console.log("(no results above threshold)"); + } + console.log(`\nTotal: ${results.length} results`); + return 0; +} diff --git a/src/db.ts b/src/db.ts index 2e9c5552..aa000e09 100644 --- a/src/db.ts +++ b/src/db.ts @@ -101,6 +101,53 @@ function migrate(db: DatabaseSync): void { ) WHERE run_number IS NULL `); } + + // Memory system tables (MEM-001) + db.exec(` + CREATE TABLE IF NOT EXISTS memory_contracts ( + id TEXT PRIMARY KEY, + run_id TEXT NOT NULL REFERENCES runs(id) ON DELETE CASCADE, + contract_type TEXT NOT NULL CHECK(contract_type IN ('constraint', 'decision', 'context', 'checkpoint')), + key TEXT NOT NULL, + value TEXT NOT NULL, + priority INTEGER DEFAULT 5, + created_at TEXT NOT NULL, + accessed_at TEXT, + access_count INTEGER DEFAULT 0, + UNIQUE(run_id, key) + ); + CREATE INDEX IF NOT EXISTS idx_memory_contracts_run ON memory_contracts(run_id); + CREATE INDEX IF NOT EXISTS idx_memory_contracts_key ON memory_contracts(key); + CREATE INDEX IF NOT EXISTS idx_memory_contracts_type ON memory_contracts(contract_type); + + CREATE TABLE IF NOT EXISTS memory_checkpoints ( + id TEXT PRIMARY KEY, + run_id TEXT NOT NULL REFERENCES runs(id) ON DELETE CASCADE, + step_id TEXT NOT NULL, + checkpoint_data TEXT NOT NULL, + created_at TEXT NOT NULL + ); + CREATE INDEX IF NOT EXISTS idx_checkpoints_run ON memory_checkpoints(run_id); + + CREATE TABLE IF NOT EXISTS session_index ( + id TEXT PRIMARY KEY, + run_id TEXT NOT NULL REFERENCES runs(id) ON DELETE CASCADE, + operation TEXT NOT NULL, + key TEXT, + success INTEGER DEFAULT 1, + details TEXT, + timestamp TEXT NOT NULL + ); + CREATE INDEX IF NOT EXISTS idx_session_run ON session_index(run_id); + CREATE INDEX IF NOT EXISTS idx_session_operation ON session_index(operation); + `); + + // Add UNIQUE constraint if not exists (for existing databases) + try { + db.exec(`CREATE UNIQUE INDEX IF NOT EXISTS idx_memory_contracts_run_key ON memory_contracts(run_id, key)`); + } catch { + // Constraint may already exist + } } export function nextRunNumber(): number { diff --git a/src/installer/step-ops.ts b/src/installer/step-ops.ts index bf47b057..5dd2df36 100644 --- a/src/installer/step-ops.ts +++ b/src/installer/step-ops.ts @@ -1,4 +1,5 @@ import { getDb } from "../db.js"; +import { MemoryContract } from "../lib/memory.js"; import type { LoopConfig, Story } from "./types.js"; import fs from "node:fs"; import path from "node:path"; @@ -531,6 +532,41 @@ export function claimStep(agentId: string): ClaimResult { context["has_frontend_changes"] = "false"; } + // MEMORY INTEGRATION (MEM-008): Load working set and checkpoint + const workingSet = MemoryContract.getWorkingSet(step.run_id, 20); + const checkpoint = MemoryContract.getLatestCheckpoint(step.run_id); + + // Inject memory context into template + if (workingSet.length > 0) { + const constraints = workingSet + .filter(c => c.contract_type === 'constraint') + .map(c => `- ${c.key}: ${JSON.stringify(c.value)}`) + .join('\n'); + const decisions = workingSet + .filter(c => c.contract_type === 'decision') + .map(c => `- ${c.key}: ${JSON.stringify(c.value)}`) + .join('\n'); + + context["memory_constraints"] = constraints || "(none)"; + context["memory_decisions"] = decisions || "(none)"; + context["memory_working_set"] = workingSet.map(c => c.key).join(', '); + } else { + context["memory_constraints"] = "(none)"; + context["memory_decisions"] = "(none)"; + context["memory_working_set"] = "(empty)"; + } + + if (checkpoint) { + context["checkpoint_step"] = checkpoint.step_id; + context["checkpoint_data"] = JSON.stringify(checkpoint.checkpoint_data); + } else { + context["checkpoint_step"] = "(none)"; + context["checkpoint_data"] = "{}"; + } + + // Required memory acknowledgment + context["required_memory_search"] = "TRUE"; + // T6: Loop step claim logic if (step.type === "loop") { const loopConfig: LoopConfig | null = step.loop_config ? JSON.parse(step.loop_config) : null; @@ -705,6 +741,23 @@ export function completeStep(stepId: string, output: string): { advanced: boolea "UPDATE runs SET context = ?, updated_at = datetime('now') WHERE id = ?" ).run(JSON.stringify(context), step.run_id); + // MEMORY INTEGRATION (MEM-008): Create checkpoint with key state + const checkpointData = { + step_id: step.step_id, + context_keys: Object.keys(context), + stories_completed: db.prepare("SELECT COUNT(*) as cnt FROM stories WHERE run_id = ? AND status = 'done'").get(step.run_id) as { cnt: number }, + timestamp: new Date().toISOString() + }; + MemoryContract.createCheckpoint(step.run_id, step.step_id, checkpointData); + + // Persist key decisions/constraints from output + if (parsed["key_decision"]) { + MemoryContract.setContract(step.run_id, 'decision', `decision_${step.step_id}`, parsed["key_decision"], 8); + } + if (parsed["key_constraint"]) { + MemoryContract.setContract(step.run_id, 'constraint', `constraint_${step.step_id}`, parsed["key_constraint"], 10); + } + // T5: Parse STORIES_JSON from output (any step, typically the planner) parseAndInsertStories(output, step.run_id); diff --git a/src/lib/memory.ts b/src/lib/memory.ts new file mode 100644 index 00000000..169c3e12 --- /dev/null +++ b/src/lib/memory.ts @@ -0,0 +1,305 @@ +import { DatabaseSync } from "node:sqlite"; +import crypto from "node:crypto"; +import { getDb } from "../db.js"; + +interface Contract { + id: string; + run_id: string; + contract_type: 'constraint' | 'decision' | 'context' | 'checkpoint'; + key: string; + value: any; + priority: number; + created_at: string; + accessed_at?: string; + access_count: number; +} + +/** + * MemoryContract API - Explicit memory persistence for antfarm workflows + * Prevents missed writes, missed retrieval, and compaction loss + */ +export class MemoryContract { + /** + * Create a new memory contract + */ + static setContract( + runId: string, + type: 'constraint' | 'decision' | 'context' | 'checkpoint', + key: string, + value: any, + priority: number = 5 + ): string { + const db = getDb(); + const id = crypto.randomUUID(); + const now = new Date().toISOString(); + + const stmt = db.prepare(` + INSERT INTO memory_contracts (id, run_id, contract_type, key, value, priority, created_at, access_count) + VALUES (?, ?, ?, ?, ?, ?, ?, 0) + ON CONFLICT(run_id, key) DO UPDATE SET + value = excluded.value, + contract_type = excluded.contract_type, + priority = excluded.priority, + accessed_at = NULL, + access_count = 0 + `); + stmt.run(id, runId, type, key, JSON.stringify(value), priority, now); + this._logOperation(runId, "write", key, true); + return id; + } + + /** + * Retrieve a contract by key + */ + static getContract(runId: string, key: string): Contract | null { + const db = getDb(); + const stmt = db.prepare(` + SELECT * FROM memory_contracts + WHERE run_id = ? AND key = ? + `); + const row = stmt.get(runId, key) as Contract | undefined; + + if (row) { + this._markAccessed(row.id); + this._logOperation(runId, "read", key, true); + return { + ...row, + value: JSON.parse(row.value as string) + }; + } + this._logOperation(runId, "read", key, false); + return null; + } + + /** + * Search contracts by type + */ + static searchContracts(runId: string, type: string): Contract[] { + const db = getDb(); + const stmt = db.prepare(` + SELECT * FROM memory_contracts + WHERE run_id = ? AND contract_type = ? + ORDER BY priority DESC, created_at DESC + `); + const rows = stmt.all(runId, type) as unknown as unknown as Contract[]; + rows.forEach(row => this._markAccessed(row.id)); + this._logOperation(runId, "search", type, rows.length > 0); + return rows.map(row => ({ + ...row, + value: JSON.parse(row.value as string) + })); + } + + /** + * Get all contracts for a run + */ + static getAllForRun(runId: string, limit: number = 100): Contract[] { + const db = getDb(); + + // Enforce max contracts per run - evict lowest priority oldest first + const countStmt = db.prepare(`SELECT COUNT(*) as count FROM memory_contracts WHERE run_id = ?`); + const { count } = countStmt.get(runId) as { count: number }; + + if (count > limit) { + this._evictOldest(runId, count - limit); + } + + const stmt = db.prepare(` + SELECT * FROM memory_contracts + WHERE run_id = ? + ORDER BY priority DESC, created_at DESC + LIMIT ? + `); + const rows = stmt.all(runId, limit) as unknown as Contract[]; + return rows.map(row => ({ + ...row, + value: JSON.parse(row.value as string) + })); + } + + /** + * Get contracts by priority threshold + */ + static getByPriority(runId: string, minPriority: number = 5): Contract[] { + const db = getDb(); + const stmt = db.prepare(` + SELECT * FROM memory_contracts + WHERE run_id = ? AND priority >= ? + ORDER BY priority DESC, created_at DESC + `); + const rows = stmt.all(runId, minPriority) as unknown as Contract[]; + rows.forEach(row => this._markAccessed(row.id)); + return rows.map(row => ({ + ...row, + value: JSON.parse(row.value as string) + })); + } + + /** + * Create a checkpoint + */ + static createCheckpoint(runId: string, stepId: string, data: any): string { + const db = getDb(); + const id = crypto.randomUUID(); + const now = new Date().toISOString(); + + // Prune old checkpoints (keep last 10) + const pruneStmt = db.prepare(` + DELETE FROM memory_checkpoints + WHERE id IN ( + SELECT id FROM memory_checkpoints + WHERE run_id = ? + ORDER BY created_at DESC + LIMIT -1 OFFSET 10 + ) + `); + pruneStmt.run(runId); + + const stmt = db.prepare(` + INSERT INTO memory_checkpoints (id, run_id, step_id, checkpoint_data, created_at) + VALUES (?, ?, ?, ?, ?) + `); + stmt.run(id, runId, stepId, JSON.stringify(data), now); + this._logOperation(runId, "checkpoint", stepId, true); + return id; + } + + /** + * Get latest checkpoint for a run + */ + static getLatestCheckpoint(runId: string): any | null { + const db = getDb(); + const stmt = db.prepare(` + SELECT * FROM memory_checkpoints + WHERE run_id = ? + ORDER BY created_at DESC + LIMIT 1 + `); + const row = stmt.get(runId) as any; + + if (row) { + this._logOperation(runId, "restore", row.step_id, true); + return { + ...row, + checkpoint_data: JSON.parse(row.checkpoint_data) + }; + } + return null; + } + + /** + * Search contracts by keyword (simple substring match) + */ + static searchByKeyword(runId: string, keyword: string): Contract[] { + const db = getDb(); + const stmt = db.prepare(` + SELECT * FROM memory_contracts + WHERE run_id = ? AND (key LIKE ? OR value LIKE ?) + ORDER BY priority DESC, created_at DESC + `); + const pattern = `%${keyword}%`; + const rows = stmt.all(runId, pattern, pattern) as unknown as Contract[]; + rows.forEach(row => this._markAccessed(row.id)); + this._logOperation(runId, "keyword_search", keyword, rows.length > 0); + return rows.map(row => ({ + ...row, + value: JSON.parse(row.value as string) + })); + } + + /** + * Get working set - highest priority + most recently accessed contracts + */ + static getWorkingSet(runId: string, maxSize: number = 20): Contract[] { + const db = getDb(); + const stmt = db.prepare(` + SELECT * FROM memory_contracts + WHERE run_id = ? + ORDER BY + (priority * 10 + COALESCE(access_count, 0)) DESC, + accessed_at DESC + LIMIT ? + `); + const rows = stmt.all(runId, maxSize) as unknown as Contract[]; + return rows.map(row => ({ + ...row, + value: JSON.parse(row.value as string) + })); + } + + /** + * Delete all contracts for a run + */ + static clearRun(runId: string): void { + const db = getDb(); + db.prepare("DELETE FROM memory_contracts WHERE run_id = ?").run(runId); + db.prepare("DELETE FROM memory_checkpoints WHERE run_id = ?").run(runId); + db.prepare("DELETE FROM session_index WHERE run_id = ?").run(runId); + this._logOperation(runId, "clear", null, true); + } + + /** + * Mark a contract as accessed + */ + private static _markAccessed(id: string): void { + const db = getDb(); + const now = new Date().toISOString(); + const stmt = db.prepare(` + UPDATE memory_contracts + SET accessed_at = ?, access_count = COALESCE(access_count, 0) + 1 + WHERE id = ? + `); + stmt.run(now, id); + } + + /** + * Evict oldest/lowest priority contracts + */ + private static _evictOldest(runId: string, count: number): void { + const db = getDb(); + const stmt = db.prepare(` + DELETE FROM memory_contracts + WHERE id IN ( + SELECT id FROM memory_contracts + WHERE run_id = ? + ORDER BY priority ASC, created_at ASC + LIMIT ? + ) + `); + stmt.run(runId, count); + } + + /** + * Log a memory operation for auditing + */ + private static _logOperation( + runId: string, + operation: string, + key: string | null, + success: boolean, + details: any = null + ): void { + const db = getDb(); + const id = crypto.randomUUID(); + const now = new Date().toISOString(); + const stmt = db.prepare(` + INSERT INTO session_index (id, run_id, operation, key, success, details, timestamp) + VALUES (?, ?, ?, ?, ?, ?, ?) + `); + stmt.run(id, runId, operation, key, success ? 1 : 0, details ? JSON.stringify(details) : null, now); + } + + /** + * Get session audit log for a run + */ + static getSessionLog(runId: string, limit: number = 100): any[] { + const db = getDb(); + const stmt = db.prepare(` + SELECT * FROM session_index + WHERE run_id = ? + ORDER BY timestamp DESC + LIMIT ? + `); + return stmt.all(runId, limit) as any[]; + } +} diff --git a/src/lib/qmd.ts b/src/lib/qmd.ts new file mode 100644 index 00000000..bd229326 --- /dev/null +++ b/src/lib/qmd.ts @@ -0,0 +1,190 @@ +import { MemoryContract } from "./memory.js"; +import { getDb } from "../db.js"; + +/** + * QMD (Query-Model-Data) Layer - Semantic search for memory contracts + * Provides ranked retrieval using multiple strategies: + * - Exact match + * - Keyword/substring match + * - N-gram overlap (semantic similarity) + * - Priority boosting + */ + +interface SearchResult { + contract: any; + score: number; + strategy: string; +} + +/** + * Generate n-grams from text + */ +function ngrams(text: string, n: number = 3): string[] { + const normalized = text.toLowerCase().replace(/[^a-z0-9]/g, ' ').split(/\s+/).filter(w => w.length > 2); + const grams: string[] = []; + for (let i = 0; i <= normalized.length - n; i++) { + grams.push(normalized.slice(i, i + n).join(' ')); + } + return grams; +} + +/** + * Calculate Jaccard similarity between two sets + */ +function jaccardSimilarity(setA: string[], setB: string[]): number { + const a = new Set(setA); + const b = new Set(setB); + const intersection = new Set([...a].filter(x => b.has(x))); + const union = new Set([...a, ...b]); + return intersection.size / union.size; +} + +/** + * Calculate relevance score for a contract against a query + */ +function calculateScore(contract: any, query: string): { score: number; strategy: string } { + const queryLower = query.toLowerCase(); + const keyLower = contract.key.toLowerCase(); + const valueStr = JSON.stringify(contract.value).toLowerCase(); + + // Exact key match (highest priority) + if (keyLower === queryLower) { + return { score: 1.0, strategy: 'exact_key' }; + } + + // Key contains query + if (keyLower.includes(queryLower)) { + return { score: 0.9, strategy: 'key_contains' }; + } + + // Value contains query + if (valueStr.includes(queryLower)) { + return { score: 0.7, strategy: 'value_contains' }; + } + + // N-gram semantic similarity + const queryGrams = ngrams(query); + const contractText = `${keyLower} ${valueStr}`; + const contractGrams = ngrams(contractText); + + if (queryGrams.length > 0 && contractGrams.length > 0) { + const similarity = jaccardSimilarity(queryGrams, contractGrams); + if (similarity > 0.1) { + return { score: 0.5 + (similarity * 0.3), strategy: 'semantic' }; + } + } + + return { score: 0, strategy: 'no_match' }; +} + +/** + * Query the memory system with ranked results + */ +export function queryMemory(runId: string, query: string, options: { + type?: 'constraint' | 'decision' | 'context' | 'checkpoint'; + minPriority?: number; + limit?: number; + minScore?: number; +} = {}): SearchResult[] { + const { type, minPriority = 1, limit = 10, minScore = 0.3 } = options; + + // Get candidate contracts + let contracts: any[]; + if (type) { + contracts = MemoryContract.searchContracts(runId, type); + } else { + contracts = MemoryContract.getAllForRun(runId, 100); + } + + // Filter by priority + contracts = contracts.filter(c => c.priority >= minPriority); + + // Score and rank + const results: SearchResult[] = contracts + .map(contract => { + const { score, strategy } = calculateScore(contract, query); + // Boost by priority (0.05 per priority level) + const priorityBoost = (contract.priority - 5) * 0.05; + return { + contract, + score: Math.min(1.0, score + priorityBoost), + strategy + }; + }) + .filter(r => r.score >= minScore) + .sort((a, b) => b.score - a.score) + .slice(0, limit); + + // Log the query + MemoryContract.getSessionLog(runId); // Just to ensure logging + + return results; +} + +/** + * Find related contracts using graph traversal + * Contracts are related if they share keywords or were accessed together + */ +export function findRelated(runId: string, key: string, depth: number = 1): any[] { + const db = getDb(); + const visited = new Set(); + const related: any[] = []; + + function traverse(currentKey: string, currentDepth: number) { + if (currentDepth > depth || visited.has(currentKey)) return; + visited.add(currentKey); + + // Find contracts accessed in same session + const coAccessed = db.prepare(` + SELECT DISTINCT c.* FROM memory_contracts c + JOIN session_index s1 ON s1.key = ? + JOIN session_index s2 ON s2.run_id = s1.run_id + AND s2.timestamp BETWEEN datetime(s1.timestamp, '-5 minutes') AND datetime(s1.timestamp, '+5 minutes') + WHERE c.key = s2.key AND c.run_id = ? AND c.key != ? + `).all(currentKey, runId, currentKey) as any[]; + + for (const contract of coAccessed) { + if (!visited.has(contract.key)) { + related.push(contract); + traverse(contract.key, currentDepth + 1); + } + } + } + + traverse(key, 0); + return related; +} + +/** + * Smart suggest - suggest relevant contracts based on current context + */ +export function suggestRelevant(runId: string, context: string, limit: number = 5): any[] { + // Extract keywords from context + const keywords = context.toLowerCase() + .replace(/[^a-z0-9\s]/g, ' ') + .split(/\s+/) + .filter(w => w.length > 3 && !['this', 'that', 'with', 'from', 'have', 'been', 'they', 'will', 'would', 'there', 'their'].includes(w)); + + // Score contracts by keyword overlap + const allContracts = MemoryContract.getAllForRun(runId, 100); + const scored = allContracts.map(contract => { + const contractText = `${contract.key} ${JSON.stringify(contract.value)}`.toLowerCase(); + const matches = keywords.filter(k => contractText.includes(k)).length; + return { contract, score: matches / keywords.length }; + }); + + return scored + .filter(s => s.score > 0) + .sort((a, b) => b.score - a.score) + .slice(0, limit) + .map(s => s.contract); +} + +/** + * Export search interface + */ +export const QMD = { + query: queryMemory, + findRelated, + suggestRelevant +}; diff --git a/src/server/index.html b/src/server/index.html index f7624a9a..f28fa059 100644 --- a/src/server/index.html +++ b/src/server/index.html @@ -215,6 +215,10 @@

antfarm dashboard

Medic + Auto-refresh: 30s
Select a workflow
@@ -222,6 +226,44 @@

antfarm dashboard

Workflow Medic
Loading...
+
+
🎮 Trooper Clash - Azure Backend Plan
+
+
+
Infrastructure Stack
+
+
Azure PlayFabAuth, Economy, Matchmaking
+
Container AppsDedicated game servers
+
Static Web AppsWebGL demo hosting
+
SignalR ServiceReal-time features
+
Cosmos DBMatch history, replays
+
Blob + CDNAsset delivery
+
+
+
+
Cost Estimates
+
+
Launch (50K MAU)~$44/month
+
Growth (100K MAU)~$467/month
+
Scale (500K MAU)~$1,658/month
+
+
+
+
Key Benefits
+
    +
  • Unity-native backend (PlayFab)
    • +
  • Scale-to-zero game servers
    • +
  • Global CDN for WebGL demo
    • +
  • Server-authoritative (anti-cheat)
    • +
  • GDPR/COPPA compliant
    • +
+
+
+ Status: Architecture defined, ready for implementation
+ Next: PlayFab SDK integration → Container Apps deployment +
+
+
@@ -511,6 +553,24 @@

Stories

{ + if (!projectsPanelOpen) return; + const panel = document.getElementById('projects-panel'); + const btn = document.getElementById('projects-btn'); + if (!panel.contains(e.target) && !btn.contains(e.target)) { + projectsPanelOpen = false; + panel.classList.remove('open'); + } +}); + async function loadMedicStatus() { try { const status = await fetchJSON('/api/medic/status'); diff --git a/tests/memory.test.ts b/tests/memory.test.ts new file mode 100644 index 00000000..998c4c7e --- /dev/null +++ b/tests/memory.test.ts @@ -0,0 +1,279 @@ +import { describe, it, beforeEach, afterEach } from "node:test"; +import assert from "node:assert/strict"; +import { MemoryContract } from "../dist/lib/memory.js"; +import { getDb } from "../dist/db.js"; + +// Test database path +const TEST_DB_PATH = "/tmp/antfarm-memory-test.db"; + +function createTestRun(db: any, runId: string) { + const now = new Date().toISOString(); + db.prepare(` + INSERT OR IGNORE INTO runs (id, workflow_id, task, status, created_at, updated_at) + VALUES (?, 'test-workflow', 'test task', 'running', ?, ?) + `).run(runId, now, now); +} + +describe("MemoryContract", () => { + let runId: string; + + beforeEach(() => { + runId = `test-run-${Date.now()}`; + // Use test database + process.env.ANTFARM_DB_PATH = TEST_DB_PATH; + // Force fresh connection and create run + const db = getDb(); + createTestRun(db, runId); + }); + + afterEach(() => { + // Clean up + try { + const db = getDb(); + db.prepare("DELETE FROM memory_contracts WHERE run_id = ?").run(runId); + db.prepare("DELETE FROM memory_checkpoints WHERE run_id = ?").run(runId); + db.prepare("DELETE FROM session_index WHERE run_id = ?").run(runId); + } catch { + // Ignore cleanup errors + } + }); + + describe("setContract / getContract", () => { + it("should create and retrieve a contract", () => { + const id = MemoryContract.setContract( + runId, + "constraint", + "test_key", + { foo: "bar" }, + 8 + ); + + assert.ok(id); + assert.strictEqual(id.length, 36); // UUID length + + const contract = MemoryContract.getContract(runId, "test_key"); + assert.ok(contract); + assert.strictEqual(contract.contract_type, "constraint"); + assert.strictEqual(contract.key, "test_key"); + assert.deepStrictEqual(contract.value, { foo: "bar" }); + assert.strictEqual(contract.priority, 8); + }); + + it("should update existing contract on conflict", () => { + MemoryContract.setContract(runId, "context", "same_key", { v: 1 }, 5); + MemoryContract.setContract(runId, "constraint", "same_key", { v: 2 }, 10); + + const contract = MemoryContract.getContract(runId, "same_key"); + assert.strictEqual(contract?.contract_type, "constraint"); + assert.deepStrictEqual(contract?.value, { v: 2 }); + assert.strictEqual(contract?.priority, 10); + }); + + it("should return null for non-existent key", () => { + const contract = MemoryContract.getContract(runId, "does_not_exist"); + assert.strictEqual(contract, null); + }); + + it("should track access count", () => { + MemoryContract.setContract(runId, "context", "tracked_key", { data: true }); + + MemoryContract.getContract(runId, "tracked_key"); + MemoryContract.getContract(runId, "tracked_key"); + + const contract = MemoryContract.getContract(runId, "tracked_key"); + assert.ok(contract?.access_count >= 2); + }); + }); + + describe("searchContracts", () => { + it("should filter by type", () => { + MemoryContract.setContract(runId, "constraint", "c1", {}, 10); + MemoryContract.setContract(runId, "constraint", "c2", {}, 9); + MemoryContract.setContract(runId, "decision", "d1", {}, 8); + MemoryContract.setContract(runId, "context", "ctx1", {}, 5); + + const constraints = MemoryContract.searchContracts(runId, "constraint"); + assert.strictEqual(constraints.length, 2); + assert.ok(constraints.every(c => c.contract_type === "constraint")); + }); + + it("should sort by priority desc, then created desc", () => { + MemoryContract.setContract(runId, "context", "low", {}, 1); + MemoryContract.setContract(runId, "context", "high", {}, 10); + MemoryContract.setContract(runId, "context", "med", {}, 5); + + const results = MemoryContract.searchContracts(runId, "context"); + assert.strictEqual(results[0].key, "high"); + assert.strictEqual(results[1].key, "med"); + assert.strictEqual(results[2].key, "low"); + }); + }); + + describe("getAllForRun", () => { + it("should return all contracts for a run", () => { + MemoryContract.setContract(runId, "constraint", "k1", {}, 10); + MemoryContract.setContract(runId, "decision", "k2", {}, 5); + MemoryContract.setContract(runId, "context", "k3", {}, 1); + + const all = MemoryContract.getAllForRun(runId); + assert.strictEqual(all.length, 3); + }); + + it("should respect limit", () => { + for (let i = 0; i < 10; i++) { + MemoryContract.setContract(runId, "context", `key${i}`, {}, 5); + } + + const limited = MemoryContract.getAllForRun(runId, 5); + assert.strictEqual(limited.length, 5); + }); + }); + + describe("getByPriority", () => { + it("should filter by minimum priority", () => { + MemoryContract.setContract(runId, "context", "high", {}, 10); + MemoryContract.setContract(runId, "context", "med", {}, 5); + MemoryContract.setContract(runId, "context", "low", {}, 1); + + const highOnly = MemoryContract.getByPriority(runId, 8); + assert.strictEqual(highOnly.length, 1); + assert.strictEqual(highOnly[0].key, "high"); + }); + }); + + describe("searchByKeyword", () => { + it("should find contracts by key substring", () => { + MemoryContract.setContract(runId, "context", "user_auth_config", { data: "x" }); + MemoryContract.setContract(runId, "context", "database_schema", { data: "y" }); + MemoryContract.setContract(runId, "context", "api_endpoints", { data: "z" }); + + const results = MemoryContract.searchByKeyword(runId, "auth"); + assert.strictEqual(results.length, 1); + assert.strictEqual(results[0].key, "user_auth_config"); + }); + + it("should find contracts by value substring", () => { + MemoryContract.setContract(runId, "context", "key1", { description: "authentication system" }); + MemoryContract.setContract(runId, "context", "key2", { description: "database connection" }); + + const results = MemoryContract.searchByKeyword(runId, "authentication"); + assert.strictEqual(results.length, 1); + assert.strictEqual(results[0].key, "key1"); + }); + }); + + describe("getWorkingSet", () => { + it("should return highest priority contracts first", () => { + MemoryContract.setContract(runId, "context", "low", {}, 1); + MemoryContract.setContract(runId, "context", "high", {}, 10); + MemoryContract.setContract(runId, "context", "med", {}, 5); + + const workingSet = MemoryContract.getWorkingSet(runId, 2); + assert.strictEqual(workingSet.length, 2); + assert.strictEqual(workingSet[0].key, "high"); + assert.strictEqual(workingSet[1].key, "med"); + }); + + it("should boost frequently accessed contracts", () => { + MemoryContract.setContract(runId, "context", "frequent", {}, 5); + MemoryContract.setContract(runId, "context", "rare", {}, 6); + + // Access frequent multiple times + for (let i = 0; i < 10; i++) { + MemoryContract.getContract(runId, "frequent"); + } + + const workingSet = MemoryContract.getWorkingSet(runId, 2); + // Frequent should be first due to access count boost + assert.strictEqual(workingSet[0].key, "frequent"); + }); + }); + + describe("Checkpoint System", () => { + it("should create and retrieve checkpoints", () => { + const checkpointData = { + completedStories: ["story-1"], + currentStory: "story-2", + branch: "feature/test" + }; + + const id = MemoryContract.createCheckpoint(runId, "step-5", checkpointData); + assert.ok(id); + + const retrieved = MemoryContract.getLatestCheckpoint(runId); + assert.ok(retrieved); + assert.strictEqual(retrieved.step_id, "step-5"); + assert.deepStrictEqual(retrieved.checkpoint_data, checkpointData); + }); + + it("should return most recent checkpoint", async () => { + MemoryContract.createCheckpoint(runId, "step-1", { v: 1 }); + await new Promise(r => setTimeout(r, 10)); + MemoryContract.createCheckpoint(runId, "step-2", { v: 2 }); + await new Promise(r => setTimeout(r, 10)); + MemoryContract.createCheckpoint(runId, "step-3", { v: 3 }); + + const latest = MemoryContract.getLatestCheckpoint(runId); + assert.ok(latest); + assert.deepStrictEqual(latest?.checkpoint_data, { v: 3 }); + }); + + it("should prune old checkpoints (keep last 10)", async () => { + // Create 15 checkpoints with small delays + for (let i = 0; i < 15; i++) { + MemoryContract.createCheckpoint(runId, `step-${i}`, { index: i }); + await new Promise(r => setTimeout(r, 5)); + } + + // Check that only recent ones exist (at most 10-11 due to timing) + const db = getDb(); + const count = db.prepare( + "SELECT COUNT(*) as count FROM memory_checkpoints WHERE run_id = ?" + ).get(runId) as { count: number }; + + assert.ok(count.count <= 11, `Expected at most 11 checkpoints, got ${count.count}`); + }); + }); + + describe("Session Index / Audit Log", () => { + it("should log write operations", () => { + MemoryContract.setContract(runId, "context", "logged_key", {}); + + const logs = MemoryContract.getSessionLog(runId); + const writeLog = logs.find(l => l.operation === "write" && l.key === "logged_key"); + assert.ok(writeLog); + assert.strictEqual(writeLog.success, 1); + }); + + it("should log read operations", () => { + MemoryContract.setContract(runId, "context", "read_key", {}); + MemoryContract.getContract(runId, "read_key"); + + const logs = MemoryContract.getSessionLog(runId); + const readLog = logs.find(l => l.operation === "read" && l.key === "read_key"); + assert.ok(readLog); + assert.strictEqual(readLog.success, 1); + }); + + it("should log failed reads", () => { + MemoryContract.getContract(runId, "non_existent_key"); + + const logs = MemoryContract.getSessionLog(runId); + const failedLog = logs.find(l => l.operation === "read" && l.key === "non_existent_key"); + assert.ok(failedLog); + assert.strictEqual(failedLog.success, 0); + }); + }); + + describe("clearRun", () => { + it("should delete all data for a run", () => { + MemoryContract.setContract(runId, "context", "k1", {}); + MemoryContract.createCheckpoint(runId, "step-1", {}); + + MemoryContract.clearRun(runId); + + assert.strictEqual(MemoryContract.getAllForRun(runId).length, 0); + assert.strictEqual(MemoryContract.getLatestCheckpoint(runId), null); + }); + }); +}); diff --git a/tests/qmd.test.ts b/tests/qmd.test.ts new file mode 100644 index 00000000..8b1f6678 --- /dev/null +++ b/tests/qmd.test.ts @@ -0,0 +1,239 @@ +import { describe, it, beforeEach, afterEach } from "node:test"; +import assert from "node:assert/strict"; +import { QMD } from "../dist/lib/qmd.js"; +import { MemoryContract } from "../dist/lib/memory.js"; +import { getDb } from "../dist/db.js"; + +const TEST_DB_PATH = "/tmp/antfarm-qmd-test.db"; + +function createTestRun(db: any, runId: string) { + const now = new Date().toISOString(); + db.prepare(` + INSERT OR IGNORE INTO runs (id, workflow_id, task, status, created_at, updated_at) + VALUES (?, 'test-workflow', 'test task', 'running', ?, ?) + `).run(runId, now, now); +} + +describe("QMD Query Layer", () => { + let runId: string; + + beforeEach(() => { + runId = `test-run-${Date.now()}`; + process.env.ANTFARM_DB_PATH = TEST_DB_PATH; + // Force fresh connection and create run + const db = getDb(); + createTestRun(db, runId); + }); + + afterEach(() => { + try { + MemoryContract.clearRun(runId); + } catch { + // Ignore cleanup errors + } + }); + + describe("query", () => { + it("should find exact key matches with highest score", () => { + MemoryContract.setContract(runId, "constraint", "database_schema", { tables: ["users"] }, 10); + MemoryContract.setContract(runId, "context", "database_config", { settings: {} }, 5); + + const results = QMD.query(runId, "database_schema"); + + assert.ok(results.length > 0); + assert.strictEqual(results[0].contract.key, "database_schema"); + assert.strictEqual(results[0].score, 1.0); + assert.strictEqual(results[0].strategy, "exact_key"); + }); + + it("should find key contains matches", () => { + MemoryContract.setContract(runId, "constraint", "user_authentication_flow", { method: "jwt" }, 8); + MemoryContract.setContract(runId, "context", "api_documentation", {}, 5); + + const results = QMD.query(runId, "authentication"); + + assert.ok(results.length > 0); + assert.strictEqual(results[0].strategy, "key_contains"); + assert.ok(results[0].score >= 0.9); + }); + + it("should find value contains matches", () => { + MemoryContract.setContract(runId, "decision", "auth_method", { strategy: "oauth2 with google provider" }, 7); + + const results = QMD.query(runId, "google"); + + assert.ok(results.length > 0); + assert.strictEqual(results[0].strategy, "value_contains"); + }); + + it("should use semantic similarity for related terms", () => { + MemoryContract.setContract(runId, "context", "orm_setup", { + description: "Prisma ORM with PostgreSQL database connection pooling" + }, 5); + + // Query with related but not exact terms + const results = QMD.query(runId, "sql database connection", { minScore: 0.2 }); + + assert.ok(results.length > 0); + // Could be semantic or value_contains depending on overlap + assert.ok(['semantic', 'value_contains'].includes(results[0].strategy)); + }); + + it("should filter by type", () => { + MemoryContract.setContract(runId, "constraint", "c1", {}, 10); + MemoryContract.setContract(runId, "decision", "d1", {}, 10); + MemoryContract.setContract(runId, "context", "ctx1", {}, 10); + + const results = QMD.query(runId, "1", { type: "constraint" }); + + assert.ok(results.every(r => r.contract.contract_type === "constraint")); + }); + + it("should filter by minimum priority", () => { + MemoryContract.setContract(runId, "context", "high_priority", {}, 10); + MemoryContract.setContract(runId, "context", "low_priority", {}, 1); + + const results = QMD.query(runId, "priority", { minPriority: 5 }); + + assert.ok(results.every(r => r.contract.priority >= 5)); + }); + + it("should respect limit option", () => { + for (let i = 0; i < 20; i++) { + MemoryContract.setContract(runId, "context", `key${i}`, {}, 5); + } + + const results = QMD.query(runId, "key", { limit: 5 }); + + assert.strictEqual(results.length, 5); + }); + + it("should return empty array when no matches above threshold", () => { + MemoryContract.setContract(runId, "context", "unrelated", { data: "xyz" }, 5); + + const results = QMD.query(runId, "completely different topic", { minScore: 0.5 }); + + assert.strictEqual(results.length, 0); + }); + + it("should rank results by score descending", () => { + MemoryContract.setContract(runId, "constraint", "exact", {}, 5); // exact match + MemoryContract.setContract(runId, "context", "exact_match", {}, 5); // contains match + MemoryContract.setContract(runId, "context", "something_else", { exact: true }, 5); // value match + + const results = QMD.query(runId, "exact"); + + assert.ok(results.length >= 3); + // Scores should be descending + for (let i = 1; i < results.length; i++) { + assert.ok(results[i].score <= results[i-1].score); + } + }); + }); + + describe("findRelated", () => { + it("should find co-accessed contracts", () => { + // Create contracts + MemoryContract.setContract(runId, "constraint", "schema", {}, 10); + MemoryContract.setContract(runId, "decision", "orm", {}, 8); + MemoryContract.setContract(runId, "context", "tech_stack", {}, 5); + + // Access both contracts to create relationship via session log + MemoryContract.getContract(runId, "schema"); + MemoryContract.getContract(runId, "orm"); + + const related = QMD.findRelated(runId, "schema", 1); + + // Should return an array (may or may not find orm depending on timing) + assert.ok(Array.isArray(related)); + }); + + it("should respect depth limit", () => { + MemoryContract.setContract(runId, "constraint", "a", {}, 10); + MemoryContract.setContract(runId, "constraint", "b", {}, 9); + MemoryContract.setContract(runId, "constraint", "c", {}, 8); + + const related = QMD.findRelated(runId, "a", 0); + + // Depth 0 should only return immediate relations (none in this case without logs) + assert.strictEqual(related.length, 0); + }); + }); + + describe("suggestRelevant", () => { + it("should suggest contracts matching context keywords", () => { + MemoryContract.setContract(runId, "constraint", "auth_middleware", { + description: "JWT authentication middleware for express routes" + }, 10); + + MemoryContract.setContract(runId, "decision", "database", { + description: "PostgreSQL with Prisma ORM" + }, 8); + + const suggestions = QMD.suggestRelevant( + runId, + "I need to implement user login with JWT tokens for authentication", + 5 + ); + + assert.ok(suggestions.length > 0); + // Should suggest auth-related contract + assert.ok(suggestions.some(s => s.key === "auth_middleware")); + }); + + it("should filter out common words", () => { + MemoryContract.setContract(runId, "context", "test_key", { data: "value" }, 5); + + // Context with only common words + const suggestions = QMD.suggestRelevant( + runId, + "this that with from have been they will would there their", + 5 + ); + + // Should not match anything since all words are filtered + assert.strictEqual(suggestions.length, 0); + }); + + it("should respect limit", () => { + for (let i = 0; i < 10; i++) { + MemoryContract.setContract(runId, "context", `key${i}`, { data: `value${i}` }, 5); + } + + const suggestions = QMD.suggestRelevant(runId, "value0 value1 value2", 3); + + assert.ok(suggestions.length <= 3); + }); + }); + + describe("n-gram similarity", () => { + it("should match semantically similar terms", () => { + MemoryContract.setContract(runId, "context", "database_connection", { + description: "PostgreSQL connection pooling configuration" + }, 5); + + // Different words but semantically related + const results = QMD.query(runId, "sql database pool", { minScore: 0.1 }); + + // May or may not match depending on n-gram overlap + if (results.length > 0) { + assert.ok(['semantic', 'value_contains'].includes(results[0].strategy)); + } + }); + + it("should score higher for more n-gram overlap", () => { + MemoryContract.setContract(runId, "context", "a", { text: "foo bar baz qux" }, 5); + MemoryContract.setContract(runId, "context", "b", { text: "foo bar xyz" }, 5); + + const results = QMD.query(runId, "foo bar baz"); + + // 'a' should rank higher due to more overlap + const resultA = results.find(r => r.contract.key === "a"); + const resultB = results.find(r => r.contract.key === "b"); + + if (resultA && resultB) { + assert.ok(resultA.score > resultB.score); + } + }); + }); +}); diff --git a/workflows/bug-fix/workflow.yml b/workflows/bug-fix/workflow.yml index 1ec1126a..bc1dd72a 100644 --- a/workflows/bug-fix/workflow.yml +++ b/workflows/bug-fix/workflow.yml @@ -9,7 +9,7 @@ description: | PR agent creates the pull request. polling: - model: default + model: devstral-small-2:latest timeoutSeconds: 120 agents: diff --git a/workflows/feature-dev/workflow.yml b/workflows/feature-dev/workflow.yml index 511c0019..6d5e867c 100644 --- a/workflows/feature-dev/workflow.yml +++ b/workflows/feature-dev/workflow.yml @@ -10,7 +10,7 @@ description: | Then integration/E2E testing, PR creation, and code review. polling: - model: default + model: devstral-small-2:latest timeoutSeconds: 120 agents: @@ -93,18 +93,29 @@ steps: TASK: {{task}} + REQUIRED_MEMORY_SEARCH: + Working Set: {{memory_working_set}} + Constraints: {{memory_constraints}} + Decisions: {{memory_decisions}} + Checkpoint: {{checkpoint_step}} + + You MUST acknowledge key constraints and decisions above before proceeding. + MEMORY_ACKNOWLEDGED: (list which constraints/decisions you considered) + Instructions: - 1. Explore the codebase to understand the stack, conventions, and patterns - 2. Break the task into small user stories (max 20) - 3. Order by dependency: schema/DB first, backend, frontend, integration - 4. Each story must fit in one developer session (one context window) - 5. Every acceptance criterion must be mechanically verifiable - 6. Always include "Typecheck passes" as the last criterion in every story - 7. Every story MUST include test criteria — "Tests for [feature] pass" - 8. The developer is expected to write tests as part of each story + 1. Review memory constraints and decisions above + 2. Explore the codebase to understand the stack, conventions, and patterns + 3. Break the task into small user stories (max 20) + 4. Order by dependency: schema/DB first, backend, frontend, integration + 5. Each story must fit in one developer session (one context window) + 6. Every acceptance criterion must be mechanically verifiable + 7. Always include "Typecheck passes" as the last criterion in every story + 8. Every story MUST include test criteria — "Tests for [feature] pass" + 9. The developer is expected to write tests as part of each story Reply with: STATUS: done + MEMORY_ACKNOWLEDGED: (confirm you reviewed constraints/decisions) REPO: /path/to/repo BRANCH: feature-branch-name STORIES_JSON: [ ... array of story objects ... ] @@ -124,17 +135,27 @@ steps: REPO: {{repo}} BRANCH: {{branch}} + REQUIRED_MEMORY_SEARCH: + Working Set: {{memory_working_set}} + Constraints: {{memory_constraints}} + Decisions: {{memory_decisions}} + Checkpoint: {{checkpoint_step}} + + MEMORY_ACKNOWLEDGED: (list which constraints/decisions you considered) + Instructions: - 1. cd into the repo - 2. Create the feature branch (git checkout -b {{branch}}) - 3. Read package.json, CI config, test config to understand the build/test setup - 4. Ensure .gitignore exists — if missing, create one appropriate for the detected stack (must include .env, node_modules/, *.key, *.pem at minimum) - 5. Run the build to establish a baseline - 6. Run the tests to establish a baseline - 7. Report what you found + 1. Review memory constraints and decisions above + 2. cd into the repo + 3. Create the feature branch (git checkout -b {{branch}}) + 4. Read package.json, CI config, test config to understand the build/test setup + 5. Ensure .gitignore exists — if missing, create one appropriate for the detected stack (must include .env, node_modules/, *.key, *.pem at minimum) + 6. Run the build to establish a baseline + 7. Run the tests to establish a baseline + 8. Report what you found Reply with: STATUS: done + MEMORY_ACKNOWLEDGED: (confirm you reviewed constraints/decisions) BUILD_CMD: TEST_CMD: CI_NOTES: @@ -178,21 +199,33 @@ steps: PROGRESS LOG: {{progress}} + REQUIRED_MEMORY_SEARCH: + Working Set: {{memory_working_set}} + Constraints: {{memory_constraints}} + Decisions: {{memory_decisions}} + Checkpoint: {{checkpoint_step}} + + You MUST review and acknowledge key constraints and decisions before proceeding. + Instructions: - 1. Read progress-{{run_id}}.txt — especially the Codebase Patterns section - 2. Pull latest on the branch - 3. Implement this story only - 4. Write tests for this story's functionality - 5. Run typecheck / build - 6. Run tests to confirm they pass - 7. Commit: feat: {{current_story_id}} - {{current_story_title}} - 8. Rewrite progress-{{run_id}}.txt with updated story results and patterns - 9. Update Codebase Patterns if you found reusable patterns + 1. Review memory constraints and decisions above + 2. Read progress-{{run_id}}.txt — especially the Codebase Patterns section + 3. Pull latest on the branch + 4. Implement this story only + 5. Write tests for this story's functionality + 6. Run typecheck / build + 7. Run tests to confirm they pass + 8. Commit: feat: {{current_story_id}} - {{current_story_title}} + 9. Rewrite progress-{{run_id}}.txt with updated story results and patterns + 10. Update Codebase Patterns if you found reusable patterns Reply with: STATUS: done + MEMORY_ACKNOWLEDGED: (confirm you reviewed constraints/decisions) CHANGES: what you implemented TESTS: what tests you wrote + key_decision: (if you made any critical decisions that future steps should know) + key_constraint: (if you identified any constraints future steps must respect) expects: "STATUS: done" max_retries: 2 on_fail: diff --git a/workflows/security-audit/workflow.yml b/workflows/security-audit/workflow.yml index 08657aff..359c3ec1 100644 --- a/workflows/security-audit/workflow.yml +++ b/workflows/security-audit/workflow.yml @@ -9,7 +9,7 @@ description: | Verifier confirms each fix. Tester runs final integration validation. PR agent creates the pull request. polling: - model: default + model: devstral-small-2:latest timeoutSeconds: 120 agents: @@ -99,12 +99,18 @@ steps: TASK: {{task}} + REQUIRED_MEMORY_SEARCH: + Working Set: {{memory_working_set}} + Constraints: {{memory_constraints}} + Decisions: {{memory_decisions}} + Instructions: - 1. Explore the codebase — understand the stack, framework, dependencies - 2. Run `npm audit` (or equivalent) if a package manager is present - 3. Scan for hardcoded secrets: API keys, passwords, tokens, private keys in source - 4. Check for .env files committed to the repo - 5. Scan for common vulnerabilities: + 1. Review memory constraints and decisions above + 2. Explore the codebase — understand the stack, framework, dependencies + 3. Run `npm audit` (or equivalent) if a package manager is present + 4. Scan for hardcoded secrets: API keys, passwords, tokens, private keys in source + 5. Check for .env files committed to the repo + 6. Scan for common vulnerabilities: - SQL injection (raw queries, string concatenation in queries) - XSS (unescaped user input in templates/responses) - CSRF (missing CSRF tokens on state-changing endpoints) @@ -115,12 +121,13 @@ steps: - Missing input validation on API endpoints - Insecure file permissions - Exposed environment variables - 6. Review auth/session handling (token expiry, session fixation, cookie flags) - 7. Check security headers (CORS, CSP, HSTS, X-Frame-Options) - 8. Document every finding with severity, file, line, and description + 7. Review auth/session handling (token expiry, session fixation, cookie flags) + 8. Check security headers (CORS, CSP, HSTS, X-Frame-Options) + 9. Document every finding with severity, file, line, and description Reply with: STATUS: done + MEMORY_ACKNOWLEDGED: (confirm you reviewed constraints/decisions) REPO: /path/to/repo BRANCH: security-audit-YYYY-MM-DD VULNERABILITY_COUNT: @@ -142,13 +149,19 @@ steps: VULNERABILITY_COUNT: {{vulnerability_count}} FINDINGS: {{findings}} + REQUIRED_MEMORY_SEARCH: + Working Set: {{memory_working_set}} + Constraints: {{memory_constraints}} + Decisions: {{memory_decisions}} + Instructions: - 1. Deduplicate findings (same root cause = one fix) - 2. Group related issues (e.g., multiple XSS from same missing sanitizer = one fix) - 3. Rank by: exploitability × impact (critical > high > medium > low) - 4. Create a prioritized fix plan — max 20 fixes - 5. If more than 20 issues, pick the top 20 by severity; note deferred items - 6. Output each fix as a story in STORIES_JSON format + 1. Review memory constraints and decisions above + 2. Deduplicate findings (same root cause = one fix) + 3. Group related issues (e.g., multiple XSS from same missing sanitizer = one fix) + 4. Rank by: exploitability × impact (critical > high > medium > low) + 5. Create a prioritized fix plan — max 20 fixes + 6. If more than 20 issues, pick the top 20 by severity; note deferred items + 7. Output each fix as a story in STORIES_JSON format Each story object must have: - id: "fix-001", "fix-002", etc. @@ -159,6 +172,7 @@ steps: Reply with: STATUS: done + MEMORY_ACKNOWLEDGED: (confirm you reviewed constraints/decisions) FIX_PLAN: CRITICAL_COUNT: HIGH_COUNT: @@ -177,15 +191,22 @@ steps: REPO: {{repo}} BRANCH: {{branch}} + REQUIRED_MEMORY_SEARCH: + Working Set: {{memory_working_set}} + Constraints: {{memory_constraints}} + Decisions: {{memory_decisions}} + Instructions: - 1. cd into the repo - 2. Create the security branch (git checkout -b {{branch}} from main) - 3. Read package.json, CI config, test config to understand build/test setup - 4. Run the build to establish a baseline - 5. Run the tests to establish a baseline + 1. Review memory constraints and decisions above + 2. cd into the repo + 3. Create the security branch (git checkout -b {{branch}} from main) + 4. Read package.json, CI config, test config to understand build/test setup + 5. Run the build to establish a baseline + 6. Run the tests to establish a baseline Reply with: STATUS: done + MEMORY_ACKNOWLEDGED: (confirm you reviewed constraints/decisions) BUILD_CMD: TEST_CMD: BASELINE: @@ -228,20 +249,32 @@ steps: PROGRESS LOG: {{progress}} + REQUIRED_MEMORY_SEARCH: + Working Set: {{memory_working_set}} + Constraints: {{memory_constraints}} + Decisions: {{memory_decisions}} + Checkpoint: {{checkpoint_step}} + + You MUST review and acknowledge key constraints before implementing fixes. + Instructions: - 1. cd into the repo, pull latest on the branch - 2. Read the vulnerability description carefully - 3. Implement the fix — minimal, targeted changes only - 4. Write a regression test that verifies the vulnerability is patched - 5. Run {{build_cmd}} to verify the build passes - 6. Run {{test_cmd}} to verify all tests pass - 7. Commit: fix(security): brief description - 8. Rewrite progress-{{run_id}}.txt with updated story results and patterns + 1. Review memory constraints and decisions above + 2. cd into the repo, pull latest on the branch + 3. Read the vulnerability description carefully + 4. Implement the fix — minimal, targeted changes only + 5. Write a regression test that verifies the vulnerability is patched + 6. Run {{build_cmd}} to verify the build passes + 7. Run {{test_cmd}} to verify all tests pass + 8. Commit: fix(security): brief description + 9. Rewrite progress-{{run_id}}.txt with updated story results and patterns Reply with: STATUS: done + MEMORY_ACKNOWLEDGED: (confirm you reviewed constraints/decisions) CHANGES: what was fixed REGRESSION_TEST: what test was added + key_decision: (if you made any critical security decisions) + key_constraint: (if you identified constraints future fixes must respect) expects: "STATUS: done" max_retries: 2 on_fail: