diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index c9436a0..5b33de2 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -83,6 +83,8 @@ Use SemaphoreSlim over lock in async contexts. Use System.Threading.Lock for the lock. ## Misc +All non‑standard library dependencies must be isolated to integrations. Only integrations may depend on integrations. + One class per file unless describing "dumb" data types like records. We have enabled implicit usings globally. diff --git a/INDEX.md b/INDEX.md index 4be146a..48274c5 100644 --- a/INDEX.md +++ b/INDEX.md @@ -1,35 +1,37 @@ -# Code +# Index Project overview: see [README](/README.md). -## Coven Engine (Coven.Core) -- [Coven.Core](/src/Coven.Core/) -- [Coven.Core.Tests](/src/Coven.Core.Tests/) +## Documentation +- Root: [README](/README.md), [CONTRIBUTING](/CONTRIBUTING.md), [AGENTS](/AGENTS.md) +- Licensing: [COMMERCIAL-TERMS](/COMMERCIAL-TERMS.md), [LICENSE](/LICENSE), [NOTICE](/NOTICE) +- Architecture: [README](/architecture/README.md), [Journaling and Scriveners](/architecture/Journaling-and-Scriveners.md), [Abstractions and Branches](/architecture/Abstractions-and-Branches.md), [Windowing and Shattering](/architecture/Windowing-and-Shattering.md), [Licensing](/architecture/Licensing.md) +- Build/Release: [Release Process](/build/ReleaseProcess.md), [VERSION](/build/VERSION) -## Coven with Agents (Coven.Spellcasting) -- [Coven.Spellcasting](/src/Coven.Spellcasting/) +## Projects (src) +- Solution: [/src/Coven.sln](/src/Coven.sln) -## Coven Infrastructure for Chat -- [Coven.Chat](/src/Coven.Chat/) +- Core: [/src/Coven.Core](/src/Coven.Core/) ([README](/src/Coven.Core/README.md)) +- Streaming: [/src/Coven.Core.Streaming](/src/Coven.Core.Streaming/) ([README](/src/Coven.Core.Streaming/README.md)) +- Daemonology: [/src/Coven.Daemonology](/src/Coven.Daemonology/) ([README](/src/Coven.Daemonology/README.md)) +- Transmutation: [/src/Coven.Transmutation](/src/Coven.Transmutation/) ([README](/src/Coven.Transmutation/README.md)) +- Spellcasting: [/src/Coven.Spellcasting](/src/Coven.Spellcasting/) ([README](/src/Coven.Spellcasting/README.md)) -# Architecture Guide +- Chat (branch): [/src/Coven.Chat](/src/Coven.Chat/) ([README](/src/Coven.Chat/README.md)) +- Chat Console (leaf): [/src/Coven.Chat.Console](/src/Coven.Chat.Console/) ([README](/src/Coven.Chat.Console/README.md)) +- Chat Discord (leaf): [/src/Coven.Chat.Discord](/src/Coven.Chat.Discord/) ([README](/src/Coven.Chat.Discord/README.md)) -Explore by topic. All paths below are under `/architecture`. +- Agents (branch): [/src/Coven.Agents](/src/Coven.Agents/) ([README](/src/Coven.Agents/README.md)) +- Agents OpenAI (leaf): [/src/Coven.Agents.OpenAI](/src/Coven.Agents.OpenAI/) ([README](/src/Coven.Agents.OpenAI/README.md)) -- [Architecture README](/architecture/README.md) +- Tests: [/src/Coven.Core.Tests](/src/Coven.Core.Tests/), [/src/Coven.Daemonology.Tests](/src/Coven.Daemonology.Tests/) -## Core Components -- [Coven.Core](/architecture/Coven.Core.md) -- [Coven.Chat](/architecture/Coven.Chat.md) -- [Coven.Daemonology](/architecture/Coven.Daemonology.md) -- [Coven.Spellcasting](/architecture/Coven.Spellcasting.md) +## Samples +- 01.DiscordAgent: [/src/samples/01.DiscordAgent](/src/samples/01.DiscordAgent/) ([README](/src/samples/01.DiscordAgent/README.md)) -## Integrations -- [Coven.Chat.Console](/architecture/Coven.Chat.Console.md) -- [Coven.Chat.Discord](/architecture/Coven.Chat.Discord.md) -- [Coven.Codex](/architecture/Coven.Codex.md) -- [Coven.OpenAI](/architecture/Coven.OpenAI.md) -- [Coven.Spellcasting.MCP](/architecture/Coven.Spellcasting.MCP.md) - -## Meta -- [Licensing](/architecture/Licensing.md) +## Toys +- [/src/toys/Coven.Toys.ConsoleChat](/src/toys/Coven.Toys.ConsoleChat/) +- [/src/toys/Coven.Toys.ConsoleOpenAI](/src/toys/Coven.Toys.ConsoleOpenAI/) +- [/src/toys/Coven.Toys.ConsoleOpenAIStreaming](/src/toys/Coven.Toys.ConsoleOpenAIStreaming/) +- [/src/toys/Coven.Toys.DiscordChat](/src/toys/Coven.Toys.DiscordChat/) +- [/src/toys/Coven.Toys.DiscordStreaming](/src/toys/Coven.Toys.DiscordStreaming/) diff --git a/README.md b/README.md index 27287dc..25878e3 100644 --- a/README.md +++ b/README.md @@ -127,7 +127,7 @@ await foreach ((long _, AgentEntry? entry) in _agents.TailAsync(0, cancellationT ### Extensibility -Window policies: tune output chunking/summarization. Example (from Sample 01 `Program.cs`): +Semantic windowing: policies define when streamed messages are ready for decision‑making (not fixed turns). See: `architecture/Windowing-and-Shattering.md`. ```csharp // Paragraph-first + tighter max-length for agent outputs diff --git a/architecture/Abstractions-and-Branches.md b/architecture/Abstractions-and-Branches.md new file mode 100644 index 0000000..b088ef8 --- /dev/null +++ b/architecture/Abstractions-and-Branches.md @@ -0,0 +1,39 @@ +# Abstractions and Branches: + +Single takeaway: Integrate with Chat and Agents; swap leaves (Discord, Console, OpenAI) without changing your application logic. + +## Spine, Branches, Leaves +- Spine: your ritual pipeline (MagikBlocks) orchestrating work. +- Branches: Chat and Agents abstractions that expose typed journals and services. +- Leaves: integrations translating abstractions to external systems (e.g., Discord, OpenAI). + +Your block logic writes/reads journal entries from Chat/Agents. The specific leaf is free to change (or multiply) behind the branch boundary. + +## Directionality +- Spine: your block/user code lives here. +- Efferent: spine → leaves (outbound from your code to adapters). +- Afferent: leaves → spine (inbound to your code from adapters). + +## Chat +- Contract: `IScrivener` representing inbound (afferent) and outbound (efferent) chat entries. +- Examples: Discord and Console leaves both implement chat daemons and journals. +- Windowing: chat drafts/outputs can be controlled via windowing policies. + +## Agents +- Contract: `IScrivener` representing prompts, thoughts, responses, and streaming chunks. +- Streaming: leaves can stream agent outputs incrementally; window policies decide emission. +- Templating: use `ITransmuter` to shape request/response items (context, persona, metadata). + +## Swap Without Rewrites +- Replace Discord with Console by changing DI registration; keep Chat code unchanged. +- Switch AI providers or models by updating the Agents leaf; keep Agent journaling unchanged. +- Combine multiple leaves (e.g., Discord + Console) and route via the same branch journals. + +## DI Patterns +- Register branches in DI via extension methods; prefer `TryAdd` within libraries. +- Start leaf daemons in a block and optionally wait for `Status.Running`. +- Keep application code unaware of leaf‑specific SDKs. + +## Related +- Journaling/Scriveners for boundary decoupling. +- Windowing/Shattering for streaming behavior. diff --git a/architecture/Coven.Chat.Console.md b/architecture/Coven.Chat.Console.md deleted file mode 100644 index c316210..0000000 --- a/architecture/Coven.Chat.Console.md +++ /dev/null @@ -1,62 +0,0 @@ -# Coven.Chat.Console - -Wires stdin/stdout into Coven as `ChatEntry` using the same journal+transmuter patterns as the Discord adapter. - -## Overview -- Reads user input from stdin as `ConsoleIncoming` and transmutes to `ChatIncoming`. -- Writes `ChatOutgoing` as `ConsoleOutgoing` and prints to stdout; records `ConsoleAck`/`ChatAck` to avoid loops. -- Uses `IScrivener` journals and an `IBiDirectionalTransmuter` to bridge Console↔Chat. - -## DI Registration -```csharp -using Coven.Chat.Console; -using Microsoft.Extensions.DependencyInjection; - -services.AddConsoleChat(new ConsoleClientConfig { - InputSender = "console", - OutputSender = "BOT" -}); -``` - -Registered services: -- `IScrivener`: default in-memory if not provided. -- `IScrivener`: `ConsoleScrivener` over a keyed internal `InMemoryScrivener`. -- `ConsoleGatewayConnection`: stdin/out bridge. -- `ConsoleTransmuter`: Console↔Chat mapping. -- `ConsoleChatSessionFactory`, `ConsoleChatSession`, `ConsoleChatDaemon`. - -## Lifecycle -Start the daemon and tail the `ChatEntry` journal. Example echo flow: -```csharp -using Coven.Chat; -using Coven.Chat.Console; -using Coven.Core; -using Coven.Core.Builder; -using Microsoft.Extensions.DependencyInjection; -using Microsoft.Extensions.Hosting; - -HostApplicationBuilder b = Host.CreateApplicationBuilder(args); -b.Services.AddConsoleChat(new ConsoleClientConfig { InputSender = "console", OutputSender = "BOT" }); -b.Services.BuildCoven(c => c.MagikBlock().Done()); -IHost host = b.Build(); -ICoven coven = host.Services.GetRequiredService(); -await coven.Ritual(new Empty()); - -sealed class EchoBlock(ContractDaemon daemon, IScrivener journal) : IMagikBlock -{ - public async Task DoMagik(Empty _, CancellationToken ct = default) - { - await daemon.Start(ct); - await foreach ((_, ChatEntry e) in journal.TailAsync(0, ct)) - if (e is ChatIncoming i) - await journal.WriteAsync(new ChatOutgoing("BOT", i.Text), ct); - return _; - } -} -``` - -## Notes -- Line-delimited input: each stdin line becomes one message. -- Acks are not pumped across journals; they only confirm local writes. -- All operations honor cooperative cancellation. - diff --git a/architecture/Coven.Chat.Discord.md b/architecture/Coven.Chat.Discord.md deleted file mode 100644 index e69de29..0000000 diff --git a/architecture/Coven.Chat.md b/architecture/Coven.Chat.md deleted file mode 100644 index e99d089..0000000 --- a/architecture/Coven.Chat.md +++ /dev/null @@ -1,67 +0,0 @@ -# Coven.Chat - -Lightweight chat abstractions for Daemon IO using Coven.Core journaling (`IScrivener`). - -- Namespace: `Coven.Chat` -- Core: `IScrivener` (append, tail, wait) and chat entry types (e.g., `ChatThought`, `ChatResponse`) -- Goal: Make message exchange deterministic and testable without prescribing a transport - ---- - -## Principles - -- Typed entries: `ChatEntry` is the exact entry type your app exchanges. -- DI‑first: Resolve `IScrivener` and any hosts via your container. Implementations use constructor injection. -- Storage‑agnostic: Sequence numbers, timestamps, and I/O are implementation details; your code sees `ChatEntry` and returned positions for chaining. - ---- - -## Usage Patterns - -### Send a message - -```csharp -// Given IScrivener via DI -await scrivener.WriteAsync(new ChatThought("agent", "Starting up..."), ct); -``` - -### Wait for the next response - -```csharp -long after = 0; // or last anchor position -var (_, response) = await scrivener.WaitForAsync(after, ct); -Console.WriteLine($"user said: {response.Text}"); -``` - -### Ask (prompt → await answer) - -```csharp -var anchor = await scrivener.WriteAsync(new ChatThought("agent", "How can I help?"), ct); -var (_, reply) = await scrivener.WaitForAsync(anchor, ct); -``` - -### Stream updates to a UI - -```csharp -await foreach (var (_, entry) in scrivener.TailAsync(0, ct)) -{ - switch (entry) - { - case ChatThought t: ui.ShowThought(t); break; - case ChatResponse r: ui.ShowResponse(r); break; - // add project‑specific entries as needed - } -} -``` - -### Backward paging (last N entries) - -```csharp -var items = new List(); -await foreach (var (_, entry) in scrivener.ReadBackwardAsync(long.MaxValue, ct)) -{ - items.Add(entry); - if (items.Count == 50) break; -} -items.Reverse(); // chronological -``` diff --git a/architecture/Coven.Codex.md b/architecture/Coven.Codex.md deleted file mode 100644 index e69de29..0000000 diff --git a/architecture/Coven.Core.md b/architecture/Coven.Core.md deleted file mode 100644 index e69de29..0000000 diff --git a/architecture/Coven.Daemonology.md b/architecture/Coven.Daemonology.md deleted file mode 100644 index 279de27..0000000 --- a/architecture/Coven.Daemonology.md +++ /dev/null @@ -1,25 +0,0 @@ -# Coven.Daemonology - -The Daemonology library contains light-weight scaffolding for building long-running services. - -## IDaemon -IDaemon represents the very minimum necessary to run a long-running service. -- Start(cancellation): Begins running a long-running service. -- Shutdown(cancellation): Shuts down a long-running service gracefully. -- Status: A property that represents what the service is currently doing when it is called. - -## ContractDaemon -ContractDaemon is an abstract base that coordinates status and failure waiters without polling. It offers: - -- `Task WaitFor(Status, cancellation)` — await a specific status. -- `Task WaitForFailure(cancellation)` — observe the first failure. -- `protected void Transition(Status)` — single mutation point for status changes. -- `protected void Fail(Exception)` — publish first failure and wake failure waiters. - -## Status enum -We use a simple enum to represent different states. Failure is not a state; it is a separate signal. - -Possible values are: -- Stopped -- Running -- Completed diff --git a/architecture/Coven.OpenAI.md b/architecture/Coven.OpenAI.md deleted file mode 100644 index e69de29..0000000 diff --git a/architecture/Coven.Spellcasting.MCP.md b/architecture/Coven.Spellcasting.MCP.md deleted file mode 100644 index e69de29..0000000 diff --git a/architecture/Coven.Spellcasting.md b/architecture/Coven.Spellcasting.md deleted file mode 100644 index 8b9a90a..0000000 --- a/architecture/Coven.Spellcasting.md +++ /dev/null @@ -1,22 +0,0 @@ -# Coven.Spellcasting.Spells - -Spells represent a tool call that the agent can intentionally invoke. They are implemented as well-typed .NET classes, unified under a single, shared contract for metadata. - -## Unified Contract -- ISpell forms: - - `ISpell` (zero-arg): `Task CastSpell()`. - - `ISpell` (unary): `Task CastSpell(TIn input)`. - - `ISpell` (binary): `Task CastSpell(TIn input)`. - - Defaults: The n-ary spell interfaces provide default `Definition` values: - - Zero: friendly name from the spell type; no schemas. - - Unary: friendly name + input schema from `TIn`. - - Binary: friendly name + input schema from `TIn`, output schema from `TOut`. - - Spellbooks remain the source of truth and may override names/schemas supplied to agents. - -## Spell Registration -Because spells can be cast from outside a C# context, their shapes (name + JSON schemas) must be available: -- The Spellbook is the source of truth for names and schemas. It supplies an `IReadOnlyList` to agents. -- Schema generation occurs during Coven finalization (`.Done()`), ensuring deterministic names/schemas. - -## DI -Spells are DI-friendly. Invocation constructs spells via the container so all dependencies are satisfied at call time. \ No newline at end of file diff --git a/architecture/Journaling-and-Scriveners.md b/architecture/Journaling-and-Scriveners.md new file mode 100644 index 0000000..83b9019 --- /dev/null +++ b/architecture/Journaling-and-Scriveners.md @@ -0,0 +1,46 @@ +# Journaling and Scriveners + +Scriveners are append‑only journals that record typed entries. They decouple producers from consumers at architectural boundaries, enable replay/time‑travel, and make streaming deterministic. + +## Why Journals? +- Audit and replay: every meaningful event is recorded for debugging and compliance. +- Decoupling: producers write entries without knowing who consumes them. +- Testing: deterministic inputs (append entries) yield deterministic outputs (tail reads). +- Streaming: incremental entries can be windowed/shattered into UX‑friendly outputs. + +## Core Behaviors +- Append‑only: entries are added with a monotonically increasing position. +- Tail: readers observe `(position, entry)` tuples starting from a position (often `0` or the latest). +- Typed streams: each journal is strongly typed (e.g., `ChatEntry`, `AgentEntry`). + +## Boundary Decoupling +- Spine ↔ Branch: MagikBlocks read/write `IScrivener`; branches/leaves observe journals and act. +- Multi‑writer, multi‑reader: independent components coordinate only through entries, not callbacks. +- Recovery: restartable components can reconstruct state by replaying entries from position `0`. + +## Patterns +- Directionality: efferent entries move away from the spine (outbound to leaves); afferent entries move toward the spine (inbound from leaves). +- Input/Output symmetry: design afferent vs efferent entries explicitly to clarify direction. +- Completion markers: write a dedicated “completed” entry to flush buffers and finalize windows. +- Idempotent consumers: consuming the same entry twice yields the same effect (or is a no‑op). +- Pure transmutation: map entries to entries with `ITransmuter`/`IBatchTransmuter` without side‑effects. + +## Common Entry Families +- Chat: `ChatAfferent`, `ChatEfferentDraft`/`ChatEfferent`, `ChatChunk`, `ChatStreamCompleted`. +- Agents: `AgentPrompt`, `AgentThought`, `AgentResponse`, `AgentAfferentChunk`, `AgentAfferentThoughtChunk`. +- Daemons: `DaemonEvent` for status changes and failures. + +## Operational Tips +- Use a single scrivener per flow in DI; avoid accidental duplicates. +- Avoid long‑running synchronous work in consumers; prefer daemons that tail asynchronously. +- Treat journals as the source of truth for cross‑component communication. + +## Related +- Windowing/Shattering: see “Windowing and Shattering”. +- Daemon lifecycle: see `src/Coven.Daemonology/README.md`. + +## Acceptance Criteria +- Describes the role of scriveners in decoupling boundaries. +- Explains append/tail and typed streams with concrete families. +- Provides patterns for completion markers and pure transmutation. +- Links to related topics and package docs. diff --git a/architecture/README.md b/architecture/README.md index cdab9d8..700aab5 100644 --- a/architecture/README.md +++ b/architecture/README.md @@ -1,19 +1,63 @@ # Architecture Guide -Explore by component. Standard patterns and integrations live in the root [README](../README.md). All docs here are flattened and named after their primary namespace for quick discovery. +This folder documents only cross‑cutting architecture — the concepts and patterns that apply across the whole system. Package‑specific docs live next to their code. -## Standards +Single takeaway: Coven provides Chat and Agent abstractions so your code talks to branches, not leaves. You integrate once with Chat/Agents and remain insulated from specific adapters like Discord or OpenAI. -- Flat structure: Keep docs in this folder; avoid deep trees. Use a subfolder only for large bundles that cannot be reasonably flattened. -- Titles: H1 must map to the primary namespace (e.g., `Coven.Spellcasting`). -- Scope: Component descriptions only; do not include detailed code that already exists in the codebase. Include only usage snippets that users must write. -- Independence: Document components independently; wiring guides come later, once components stand alone. -- Size: Keep docs focused and readable. Target ~240 lines to reduce the chance of missing key details. -- Canonical patterns: If a pattern is required for all users, place it in the repo root [README](../README.md) (not here). -- Deprecations: When we delete or deprecate a feature, purge its docs immediately. -- Index hygiene: Update this README and the repo [INDEX](../INDEX.md) whenever docs change. -- All non-standard library dependencies must be isolated to integrations. -- Only integrations may depend on integrations. +## Critical Subtopics (Summaries) + +- Daemons & Lifecycle: All long‑running work runs as daemons implementing a minimal lifecycle (Start/Shutdown with `Status` and failure journaling). Orchestration starts daemons inside a MagikBlock and can deterministically await `Running` or handle failures. This keeps background processing testable and predictable. See: `src/Coven.Daemonology/README.md`. + +- Journaling & Scriveners: Append‑only, typed journals decouple producers from consumers, enable replay/time‑travel, and make streaming deterministic. Blocks, branches, and leaves communicate by writing/reading entries, not by callbacks. See: [Journaling and Scriveners](./Journaling-and-Scriveners.md). + +- Abstractions: Chat & Agents: Write app logic against branches (Chat/Agents) so you can swap leaves (Discord, Console, OpenAI) without touching your spine. Typed entries model afferent/efferent flows; templating and streaming are layered on top. See: [Abstractions and Branches](./Abstractions-and-Branches.md). + +- Windowing & Shattering (Semantic Windowing): Policies define when buffered, streamed messages are ready for decision‑making. Optional shatter splits outputs (e.g., paragraphs). Completion markers ensure deterministic flush. See: [Windowing and Shattering](./Windowing-and-Shattering.md). + +## Directionality (Afferent vs Efferent) + +- Efferent: messages flowing away from your block code (spine) toward leaves (adapters/integrations). +- Afferent: messages flowing from leaves back toward your block code (spine). +- Rule of thumb: block/user code lives at the spine; efferent goes out, afferent comes in. + +## Examples: Mapping Concepts to Sample 01 (Discord Agent) + +Concrete examples help ground the vocabulary above. The Sample 01 app wires Discord chat to an OpenAI‑backed agent via a simple router MagikBlock. + +- Spine: the ritual executes a single `RouterBlock` MagikBlock. + - File: `src/samples/01.DiscordAgent/RouterBlock.cs` + +- Branches: app logic targets Chat and Agents abstractions. + - Chat branch types used by the router: `ChatEntry`, `ChatAfferent`, `ChatEfferentDraft` (from `Coven.Chat`). + - Agents branch types used by the router: `AgentEntry`, `AgentPrompt`, `AgentResponse`, `AgentThought` (from `Coven.Agents`). + +- Leaves (adapters): concrete integrations that translate branches to external systems. + - Discord chat: `Coven.Chat.Discord` (gateway/session/scrivener/daemon). Configured via `DiscordClientConfig` in `Program.cs`. + - OpenAI agent: `Coven.Agents.OpenAI` (gateway/session/scrivener/daemon). Configured via `OpenAIClientConfig` in `Program.cs`. + +- Daemons: long‑running services started inside the router block. + - Discord chat daemon (`DiscordChatDaemon`) tails Discord and writes `ChatAfferent` entries. + - OpenAI agent daemon (`OpenAIAgentDaemon`) processes prompts and writes streamed agent chunks/entries. + - Optional streaming daemons (from `Coven.Core.Streaming`) may be registered by branches to window chunk streams into outputs. + +- Journals (Scriveners): append‑only logs connecting router↔branches↔leaves. + - Chat journal: `IScrivener` (Discord adapter implements its own scrivener; router reads/writes entries). + - Agent journal: `IScrivener` (OpenAI adapter scrivener; router reads/writes entries). + +- Directionality in the router logic: + - Afferent (inbound): Discord adapter writes `ChatAfferent` when a user posts in the channel; router reads and forwards as an `AgentPrompt`. + - Efferent (outbound): When the agent emits an `AgentResponse`, router writes a `ChatEfferentDraft` so the Discord adapter sends a message to the channel. + - Thoughts: `AgentThought` entries are internal by default; the sample shows how to optionally surface them to chat. + +- Streaming and windowing (semantic readiness): + - OpenAI adapter streams agent output as chunks. Window policies (e.g., paragraph or max‑length) determine when to emit a user‑visible response. + - Sample 01 demonstrates how to override windowing or templating via DI in `Program.cs`. + +- Transmutation/templating examples: + - Sample mapping of OpenAI entries to response items for prompt/answer templating: `DiscordOpenAITemplatingTransmuter` in `src/samples/01.DiscordAgent/DiscordOpenAITemplatingTransmuter.cs`. + +- Configuration entry points (minimal app wire‑up): + - File: `src/samples/01.DiscordAgent/Program.cs` — registers Discord chat and OpenAI agents, enables optional streaming behavior, and builds the ritual. ## Cancellation Tokens @@ -24,18 +68,29 @@ Explore by component. Standard patterns and integrations live in the root [READM - Exceptions: Treat `OperationCanceledException` as cooperative shutdown; don’t log it as an error or wrap it. - I/O: Prefer token-aware APIs; if missing, use a cancel-aware pattern (avoid `WaitAsync` if a better token overload exists). -## Core Components -- [Coven.Core](Coven.Core.md): Core runtime (MagikBlocks, builder, routing, board, IScrivener, ITransmuter). -- [Coven.Chat](Coven.Chat.md): Patterns for enabling conversation from external sources (Console or Discord for example) -- [Coven.Daemonology](Coven.Daemonology.md): Describes core features for long running hosts. -- [Coven.Spellcasting](Coven.Spellcasting.md): Spell contracts and schema conventions. This is how we wire in tools. - -## Integrations -- [Coven.Chat.Console](Coven.Chat.Console.md): Wires stdin/stdout into Coven as ChatEntry. -- [Coven.Chat.Discord](Coven.Chat.Discord.md): Wires discord messages into Coven as ChatEntry. -- [Coven.Codex](Coven.Codex.md): Implementation of Codex CLI Daemon. -- [Coven.OpenAI](Coven.OpenAI.md): Implementation of OpenAI responses API as a Coven-style Daemon. -- [Coven.Spellcasting.MCP](Coven.Spellcasting.MCP.md): Enables consumption of spells as MCP tools. + +## Documentation Standards + +- Purpose: This folder documents cross‑cutting architecture only (concepts, lifecycles, policies, contracts, patterns). Avoid package‑specific API docs here. +- Location: Package/project documentation lives next to code at `src//README.md` (usage, configuration, examples, changelogs). +- Structure: Keep a flat layout in `architecture/`; +- Titles: H1 names are topic‑based (e.g., `Windowing and Shattering`, `Journaling`), not package/namespace names. +- Scope: Show patterns with minimal examples; link to package READMEs for concrete types and APIs. Do not duplicate per‑package documentation. +- Independence: Cross‑cutting docs should remain implementation‑agnostic and stable across refactors; reference contracts and behaviors, not specific classes. +- Size: Keep each document focused and readable (target ~240 lines) to reduce drift and ease maintenance. +- Canonical patterns: Repo‑wide mandatory patterns live in the root [README](../README.md); use `architecture/` for deeper rationale and trade‑offs. +- Deprecations: Remove or relocate obsolete cross‑cutting docs immediately. Retire package‑specific pages from `architecture/` and move details to `src//README.md`. +- Index hygiene: Keep this README and the repo [INDEX](../INDEX.md) up to date; link to both cross‑cutting topics and package READMEs when relevant. + + +## Topics +- Cross‑cutting topics live here. Current topics include the guidance in this file (e.g., Cancellation Tokens) and the pages below: + - [Journaling and Scriveners](./Journaling-and-Scriveners.md) + - [Abstractions and Branches](./Abstractions-and-Branches.md) + - [Windowing and Shattering](./Windowing-and-Shattering.md) + +## Where to Find Package Docs +- Per‑package documentation: see `src//README.md` (usage, configuration, examples). ## Meta/Misc - [Licensing](./Licensing.md) diff --git a/architecture/Windowing-and-Shattering.md b/architecture/Windowing-and-Shattering.md new file mode 100644 index 0000000..7f4f156 --- /dev/null +++ b/architecture/Windowing-and-Shattering.md @@ -0,0 +1,46 @@ +# Windowing and Shattering + +Control how streaming chunks become user‑visible outputs. Policies decide when to emit; shatter rules split outputs for better UX. + +## Concepts +- Chunks: granular streaming fragments written to journals (e.g., `AgentAfferentChunk`). +- Window: buffered view over pending chunks, passed to `IWindowPolicy`. +- Emit: when a policy returns true, buffered chunks are batch‑transmuted into an output. +- Shatter: optional split of an output into multiple entries (e.g., paragraphs). +- Completion: a marker entry causes the buffer to flush deterministically. + +## Semantic Windowing + +Instead of fixed “turns,” Coven supports semantic windowing: policies determine when a buffered window of incoming messages is ready for decision‑making. Readiness is about meaning, not sequence count. + +- Readiness examples: + - Content boundary reached (paragraph/sentence end, code block closed). + - Thought summary/marker emitted indicating a coherent chunk is available. + - Safety thresholds crossed (token/length caps) to prevent over‑accumulation. + - Time‑based debounce to avoid emitting on every tiny chunk while staying responsive. + - External/user signals (e.g., user stop, domain event) indicating a decision point. + +- Decisions at readiness: + - Emit an interim or final response to the user. + - Advance workflow (e.g., route to a block, persist a checkpoint). + - Trigger downstream processing that benefits from coherent input windows. + +- Implementing semantics: + - Encode readiness in `IWindowPolicy`; compose multiple semantics with `CompositeWindowPolicy` (logical OR). + - Use `IBatchTransmuter` to convert a ready window into a decision artifact (e.g., response or structured record); return a remainder if partially consumed. + - Use completion markers to drain any residual buffered content deterministically. + +## Why It Matters +- Responsiveness: stream partial outputs without sacrificing coherence. +- Control: combine policies (e.g., paragraph OR max‑length) to match UX goals. +- Determinism: completion guarantees final flush and consistent replay. + +## Typical Policies +- Final‑only: emit only on completion. +- Paragraph‑first: emit on paragraph boundaries. +- Max‑length: emit when buffered size exceeds a threshold. +- Composite: OR multiple policies for flexible behavior. + +## Related +- See `src/Coven.Core.Streaming/README.md` for API details and examples. +- Used by Chat and Agents branches to shape streaming UX. diff --git a/src/Coven.Agents.OpenAI/OpenAITransmuter.cs b/src/Coven.Agents.OpenAI/OpenAITransmuter.cs index 32ea26d..f9c2210 100644 --- a/src/Coven.Agents.OpenAI/OpenAITransmuter.cs +++ b/src/Coven.Agents.OpenAI/OpenAITransmuter.cs @@ -4,6 +4,10 @@ namespace Coven.Agents.OpenAI; +/// +/// Maps between OpenAI-specific entries and generic Agent entries. +/// Afferent: OpenAI → Agent; Efferent: Agent → OpenAI. +/// internal sealed class OpenAITransmuter : IBiDirectionalTransmuter { public Task TransmuteAfferent(OpenAIEntry Input, CancellationToken cancellationToken) @@ -34,8 +38,8 @@ public Task TransmuteEfferent(AgentEntry Output, CancellationToken AgentPrompt prompt => Task.FromResult(new OpenAIEfferent(prompt.Sender, prompt.Text)), AgentResponse response => Task.FromResult(new OpenAIAck(response.Sender, response.Text)), AgentThought thought => Task.FromResult(new OpenAIAck(thought.Sender, thought.Text)), - AgentEfferentChunk affChunk => Task.FromResult(new OpenAIAck(affChunk.Sender, affChunk.Text)), - AgentAfferentChunk effChunk => Task.FromResult(new OpenAIAck(effChunk.Sender, effChunk.Text)), + AgentEfferentChunk efferentChunk => Task.FromResult(new OpenAIAck(efferentChunk.Sender, efferentChunk.Text)), + AgentAfferentChunk afferentChunk => Task.FromResult(new OpenAIAck(afferentChunk.Sender, afferentChunk.Text)), // Streaming efferent thought drafts map to OpenAI efferent thought chunk (not forwarded by gateway today) AgentEfferentThoughtChunk etChunk => Task.FromResult(new OpenAIEfferentThoughtChunk(etChunk.Sender, etChunk.Text)), // Afferent thought drafts are not sent outward; ack for completeness diff --git a/src/Coven.Agents.OpenAI/README.md b/src/Coven.Agents.OpenAI/README.md new file mode 100644 index 0000000..b1549d7 --- /dev/null +++ b/src/Coven.Agents.OpenAI/README.md @@ -0,0 +1,63 @@ +# Coven.Agents.OpenAI + +OpenAI agent integration (official .NET SDK). Registers journals, gateway/session, transmuters, and optional streaming/windowing. + +## What’s Inside + +- Config: `OpenAIClientConfig` (API key, model, optional org/project; reasoning options). +- Registration: `AddOpenAIAgents(config, registration => ...)`. +- Gateways: request vs streaming connections. +- Journals: `IScrivener`, `IScrivener`. +- Transmuters: `OpenAITransmuter` (OpenAI↔Agent), `OpenAIEntryToResponseItemTransmuter` (templating), `OpenAIResponseOptionsTransmuter`. +- Windowing: default policies for response chunks and thought chunks when streaming is enabled. +- Daemons: `OpenAIAgentDaemon` and windowing daemons (when streaming). + +## Quick Start + +```csharp +using Coven.Agents.OpenAI; + +OpenAIClientConfig cfg = new() +{ + ApiKey = Environment.GetEnvironmentVariable("OPENAI_API_KEY")!, + Model = Environment.GetEnvironmentVariable("OPENAI_MODEL") ?? "gpt-5-2025-08-07", +}; + +services.AddOpenAIAgents(cfg, registration => +{ + registration.EnableStreaming(); // optional, enables windowing daemons +}); +``` + +## Override Policies (Streaming) + +```csharp +using Coven.Core.Streaming; +using Coven.Agents; + +// Response chunks: paragraphs OR soft cap +services.AddScoped>(_ => + new CompositeWindowPolicy( + new AgentParagraphWindowPolicy(), + new AgentMaxLengthWindowPolicy(4096))); + +// Thought chunks: summary marker OR soft cap +services.AddScoped>(_ => + new CompositeWindowPolicy( + new AgentThoughtSummaryMarkerWindowPolicy(), + new AgentThoughtMaxLengthWindowPolicy(4096))); +``` + +## Templating (Optional) + +Provide an `ITransmuter` to inject context (usernames, system preamble, etc.). See sample `DiscordOpenAITemplatingTransmuter` in `src/samples/01.DiscordAgent`. + +## Requirements + +- Valid OpenAI API key and model name for your account. +- Network egress to OpenAI endpoints. + +## See Also + +- Branch: `Coven.Agents`. +- Architecture: Abstractions and Branches; Windowing and Shattering. diff --git a/src/Coven.Agents/README.md b/src/Coven.Agents/README.md new file mode 100644 index 0000000..f5181f5 --- /dev/null +++ b/src/Coven.Agents/README.md @@ -0,0 +1,35 @@ +# Coven.Agents + +Branch abstraction for AI agents. Defines typed agent entries and batch transmutation for streamed responses and thoughts. + +## What’s Inside + +- Entries: `AgentPrompt`, `AgentResponse`, `AgentThought`, `AgentAck`. +- Streaming entries: `AgentAfferentChunk`, `AgentEfferentChunk`, `AgentAfferentThoughtChunk`, `AgentEfferentThoughtChunk`, `AgentStreamCompleted`. +- Batch transmuters: `AgentAfferentBatchTransmuter` (response chunks → `AgentResponse`), `AgentAfferentThoughtBatchTransmuter` (thought chunks → `AgentThought`). + +## Why use it? + +- Decouple agent‑facing logic from specific providers (OpenAI, etc.). +- Stream agent output and surface user‑visible responses with semantic windowing. + +## Usage (Conceptual) + +Applications typically integrate a concrete leaf (e.g., `Coven.Agents.OpenAI`) which registers journals, daemons, and default policies. Your blocks read/write `AgentEntry`: + +```csharp +await _agents.WriteAsync(new AgentPrompt("user", "hello"), ct); + +await foreach ((long _, AgentEntry? entry) in _agents.TailAsync(0, ct)) +{ + if (entry is AgentResponse r) + { + // forward to chat + } +} +``` + +## See Also + +- Provider: `Coven.Agents.OpenAI`. +- Architecture: Abstractions and Branches; Windowing and Shattering. diff --git a/src/Coven.Chat.Console/ConsoleTransmuter.cs b/src/Coven.Chat.Console/ConsoleTransmuter.cs index 67b923f..7c13107 100644 --- a/src/Coven.Chat.Console/ConsoleTransmuter.cs +++ b/src/Coven.Chat.Console/ConsoleTransmuter.cs @@ -2,6 +2,10 @@ namespace Coven.Chat.Console; +/// +/// Maps between Console-specific entries and generic Chat entries. +/// Afferent: Console → Chat; Efferent: Chat → Console. +/// internal sealed class ConsoleTransmuter(ConsoleClientConfig config) : IBiDirectionalTransmuter { private readonly ConsoleClientConfig _config = config ?? throw new ArgumentNullException(nameof(config)); diff --git a/src/Coven.Chat.Console/README.md b/src/Coven.Chat.Console/README.md new file mode 100644 index 0000000..80e1e62 --- /dev/null +++ b/src/Coven.Chat.Console/README.md @@ -0,0 +1,30 @@ +# Coven.Chat.Console + +Console chat adapter (leaf). Bridges standard input/output to `Coven.Chat` entries and runs a console daemon. + +## What’s Inside + +- Config: `ConsoleClientConfig` (`InputSender`, `OutputSender`). +- Gateway + Session: connect console I/O to chat journal. +- Transmuter: `ConsoleTransmuter` (ConsoleEntry ↔ ChatEntry). +- Journals: `IScrivener`, `IScrivener`. +- Daemon: `ConsoleChatDaemon`. + +## Usage + +```csharp +using Coven.Chat.Console; + +services.AddConsoleChat(new ConsoleClientConfig +{ + InputSender = "console", + OutputSender = "BOT" +}); +``` + +This registers the console gateway/session, a chat journal (if none exists), a console scrivener, and the console daemon. Pair with `Coven.Chat` windowing if you want streamed drafts to become finalized chat messages. + +## See Also + +- Branch: `Coven.Chat` for windowing. +- Sample: See root README for swapping Discord→Console in Sample 01. diff --git a/src/Coven.Chat.Discord/README.md b/src/Coven.Chat.Discord/README.md new file mode 100644 index 0000000..adaccfd --- /dev/null +++ b/src/Coven.Chat.Discord/README.md @@ -0,0 +1,37 @@ +# Coven.Chat.Discord + +Discord chat adapter (leaf). Bridges a Discord channel to `Coven.Chat` entries with a daemon and default windowing suitable for Discord limits. + +## What’s Inside + +- Config: `DiscordClientConfig` (bot token, `ChannelId`). +- Gateway + Session: `DiscordGatewayConnection`, `DiscordChatSessionFactory`. +- Transmuter: `DiscordTransmuter` (DiscordEntry ↔ ChatEntry). +- Journals: `IScrivener`, `IScrivener`. +- Daemon: `DiscordChatDaemon`. +- Windowing defaults: paragraph OR 2000‑char cap; shatter drafts accordingly. + +## Prerequisites + +- Discord bot token with Message Content intent enabled. +- Bot invited to the server and has read/write permissions for the target channel. +- Channel ID (copy via Discord Developer Mode). + +## Usage + +```csharp +using Coven.Chat.Discord; + +services.AddDiscordChat(new DiscordClientConfig +{ + BotToken = Environment.GetEnvironmentVariable("DISCORD_BOT_TOKEN")!, + ChannelId = ulong.Parse(Environment.GetEnvironmentVariable("DISCORD_CHANNEL_ID")!) +}); +``` + +This registers the Discord client, session factory, journals, transmuter, daemon, and windowing/shattering tuned for Discord. + +## See Also + +- Branch: `Coven.Chat`. +- Sample: `src/samples/01.DiscordAgent`. diff --git a/src/Coven.Chat/README.md b/src/Coven.Chat/README.md new file mode 100644 index 0000000..19943a2 --- /dev/null +++ b/src/Coven.Chat/README.md @@ -0,0 +1,42 @@ +# Coven.Chat + +Branch abstraction for multi‑user chat. Defines typed chat entries, batch transmutation for chunked output, and DI helpers for chat windowing. + +## What’s Inside + +- Entries: `ChatAfferent`, `ChatEfferent`, `ChatEfferentDraft`, `ChatChunk`, `ChatStreamCompleted`, `ChatAck`. +- Windowing: DI extension `AddChatWindowing()` registers a windowing daemon for chat. +- Policies: `Windowing/*` paragraph/length/sentence window policies. +- Shattering: `Shattering/*` policies to split drafts into chunks (paragraph, sentence, max length). +- Transmuter: `ChatChunkBatchTransmuter` (chunks → `ChatEfferent`). + +## Why use it? + +- Write app logic once against `ChatEntry` and swap leaves (Console, Discord) without changing your code. +- Streamed responses become user‑visible messages based on semantic windowing policies. + +## Usage + +```csharp +using Coven.Chat; +using Coven.Core.Streaming; + +// Register chat windowing in DI +services.AddChatWindowing(); + +// Optionally override the window policy (OR composition) +services.AddScoped>(_ => + new CompositeWindowPolicy( + new ChatParagraphWindowPolicy(), + new ChatMaxLengthWindowPolicy(2000))); +``` + +## Entries at a Glance + +- Afferent (inbound to spine): `ChatAfferent`, `ChatAfferentDraft`, `ChatChunk`, `ChatStreamCompleted`. +- Efferent (outbound to users): `ChatEfferentDraft` (draft), `ChatEfferent` (final), `ChatAck` (local loop prevention). + +## See Also + +- Adapters: `Coven.Chat.Discord`, `Coven.Chat.Console`. +- Architecture: Abstractions and Branches; Windowing and Shattering. diff --git a/src/Coven.Core.Streaming/README.md b/src/Coven.Core.Streaming/README.md new file mode 100644 index 0000000..5a78148 --- /dev/null +++ b/src/Coven.Core.Streaming/README.md @@ -0,0 +1,168 @@ +# Coven.Core.Streaming + +Windowing and shattering primitives for streaming journals. Turn a flow of chunks into well‑formed outputs under explicit policies, with a daemon that handles buffering, emit timing, and final flush. + +## What’s Inside + +- IWindowPolicy: decide when a buffered window should emit +- StreamWindow: snapshot passed to window policies +- CompositeWindowPolicy: OR‑composition of policies +- LambdaWindowPolicy: delegate‑based policy +- IShatterPolicy: split one entry into zero or more entries +- LambdaShatterPolicy: delegate‑based shatter +- ChainedShatterPolicy: sequential shatter pipeline +- StreamWindowingDaemon: generic daemon that windows a journal and emits outputs + +Depends on Coven.Transmutation for batch transmutation: + +- IBatchTransmuter — converts a batch of chunks into an output (+ optional remainder) +- BatchTransmuteResult — output + remainder contract + +## Why use it? + +- Stream control: shape how incremental chunks become user‑visible outputs. +- Deterministic flush: completion markers drain remaining buffered data. +- Composable: combine policies via `CompositeWindowPolicy` to meet UX needs. +- Extensible: adapt any entry types; not tied to chat or agents. + +## Key Concepts + +- Chunk vs Output: + - Chunks (`TChunk`) are the granular pieces appended to a journal during streaming. + - Outputs (`TOutput`) are finalized entries emitted when a window policy decides to emit. +- Directionality in practice: + - Policies typically operate on afferent chunks (incoming toward the spine) and emit efferent outputs (outbound to users/adapters), but the primitives are direction‑agnostic. +- Window Policy (`IWindowPolicy`): + - `MinChunkLookback`: ensures policy decisions see enough recent context. + - `ShouldEmit(StreamWindow)`: returns true to emit at the current point. +- Shatter Policy (`IShatterPolicy`): + - Optionally breaks an output into zero or more entries (e.g., paragraphs). + - If no shards are produced, the original output is forwarded as‑is. +- Completion (`TCompleted`): + - A special entry that triggers a full drain of the buffer. + - The daemon emits as many outputs as needed to flush remaining chunks. + +### Semantic Windowing + +Policies model readiness, not fixed “turns.” A window is emitted when it’s semantically ready (e.g., paragraph boundary, safe length cap, debounce, or explicit marker). See architecture: `architecture/Windowing-and-Shattering.md`. + +## How StreamWindowingDaemon Works + +Given a journal of `TEntry` (where `TChunk`, `TOutput`, and `TCompleted` are subtypes of `TEntry`): + +1) On Start, the daemon tails the journal after the latest position and sets `Status.Running`. +2) It buffers `TChunk` entries as they arrive. For each new chunk: + - It constructs a `StreamWindow` consisting of the last `MinChunkLookback` chunks, total chunk count, start time, and last emit time. + - If the window policy returns true, it batch‑transmutes the buffer via `IBatchTransmuter`. + - If a shatter policy is provided, it writes each shard; otherwise it writes the transmuted output. + - If the transmuter returns a remainder chunk, the buffer becomes only the remainder; else it clears. +3) On a `TCompleted` entry, it drains the buffer completely: + - Repeatedly batch‑transmute and write outputs until the buffer is empty (guarded against infinite loops). +4) On Shutdown, it cancels, awaits the pump, and sets `Status.Completed`. +5) On unexpected exceptions, it calls `Fail(ex)` so orchestration can react. + +## Usage Examples + +### Minimal policy with final‑only emit + +```csharp +using Coven.Core; +using Coven.Core.Streaming; +using Coven.Transmutation; + +// Emit only when a completion marker arrives (final‑only) +IWindowPolicy policy = new LambdaWindowPolicy(1, _ => false); + +// Simple batch transmuter: concatenate chunk text +public sealed class MyBatchTransmuter : IBatchTransmuter +{ + public Task> Transmute(IEnumerable input, CancellationToken ct = default) + { + string text = string.Concat(input.Select(c => c.Text)); + return Task.FromResult(new BatchTransmuteResult( + new MyOutput(text), + HasRemainder: false, + Remainder: default)); + } +} + +// Wire daemon (e.g., in DI factory) +var daemon = new StreamWindowingDaemon( + daemonEvents: myDaemonEventScrivener, + journal: myJournal, + windowPolicy: policy, + batchTransmuter: new MyBatchTransmuter(), + shatterPolicy: null); +``` + +### Composite policies (paragraphs OR max length) + +```csharp +// Combine multiple emit rules via OR +IWindowPolicy policy = new CompositeWindowPolicy( + new LambdaWindowPolicy(minLookback: 2, window => + { + // Example: emit on blank‑line paragraph boundary + return window.PendingChunks.Any(c => string.IsNullOrWhiteSpace(c.Text)); + }), + new LambdaWindowPolicy(minLookback: 1, window => + { + // Example: emit when total buffered text exceeds N characters + return window.PendingChunks.Sum(c => c.Text.Length) >= 1000; + })); +``` + +### Shattering outputs + +```csharp +// Split an output into multiple entries (e.g., paragraphs) +IShatterPolicy shatter = new LambdaShatterPolicy(entry => +{ + if (entry is MyOutput o) + { + return o.Text + .Split("\n\n") + .Select(p => new MyOutputParagraph(p)); + } + return Array.Empty(); +}); +``` + +### Chat Example (built‑in) + +`Coven.Chat` wires a windowing daemon for chat journals: + +```csharp +services.AddChatWindowing(); + +// Internally registers: +// new StreamWindowingDaemon(...) +// Policy defaults to final‑only (emit on completion) unless overridden via DI +``` + +You can override the chat window policy by registering your own `IWindowPolicy` (or chain policies via `CompositeWindowPolicy`). + +### OpenAI Example (policy ideas) + +`Coven.Agents.OpenAI` includes ready‑made policies (e.g., paragraph, max‑length, thought windowing) that you can mix and match using `CompositeWindowPolicy` to tune when agent responses and thoughts are emitted. + +## Tips + +- Choose `MinChunkLookback` to balance responsiveness and context. +- Use remainders when your batch transmuter only consumes part of the last chunk. +- Provide a `TCompleted` entry to guarantee all buffered content is emitted. +- Prefer pure policies (no side‑effects) for predictability and testability. +- Ensure a single `IScrivener` instance is used for a given flow. + - Be mindful of overhead: windowing/shattering daemons introduce buffering and journaling work. For hot paths, apply window/shatter inline (mid‑process) without a daemon where it makes the most performance sense. + +## Testing + +- Use `InMemoryScrivener` (from `Coven.Core`) to unit‑test daemon behavior. +- Assert emission timing by appending `TChunk` entries and awaiting journal tails. +- Verify full flush by appending `TCompleted` and observing outputs. + +## See Also + +- Coven.Transmutation: `IBatchTransmuter`, `BatchTransmuteResult`, `ITransmuter` +- Coven.Chat: wiring example and default batch transmuter for chat +- Root README: window/shatter overview and end‑to‑end samples diff --git a/src/Coven.Core/README.md b/src/Coven.Core/README.md new file mode 100644 index 0000000..af28c26 --- /dev/null +++ b/src/Coven.Core/README.md @@ -0,0 +1,57 @@ +# Coven.Core + +Engine primitives for composing and running a Coven: MagikBlocks, Scriveners (journals), orchestration, and DI helpers to build a spine (ritual) of work. + +## What’s Inside + +- IMagikBlock: unit of work with `DoMagik`. +- IScrivener: append‑only, typed journal with tailing; includes `InMemoryScrivener`. +- Builder: `BuildCoven`, `CovenServiceBuilder` to register blocks and finalize runtime. +- Orchestration: `ICoven` to invoke rituals; `IBoard` and pull mode options. +- Utilities: `Empty` marker type; capability tags and selection strategy hooks. + +## Why use it? + +- Deterministic pipelines: execute a chain of typed blocks with explicit I/O. +- Decoupled comms: components communicate via journals instead of callbacks. +- Testable by design: swap `InMemoryScrivener` and run blocks in isolation. + +## Usage + +```csharp +using Coven.Core; +using Coven.Core.Builder; +using Microsoft.Extensions.DependencyInjection; + +// A minimal block that writes/reads journals or orchestrates other services +internal sealed class HelloBlock : IMagikBlock +{ + public Task DoMagik(Empty input, CancellationToken cancellationToken = default) + { + // do work; start daemons; read/write journals + return Task.FromResult(input); + } +} + +ServiceCollection services = new(); + +services.BuildCoven(c => +{ + c.MagikBlock(); + c.Done(); +}); + +ServiceProvider provider = services.BuildServiceProvider(); +ICoven coven = provider.GetRequiredService(); +await coven.Ritual(new Empty()); +``` + +## Testing + +- Prefer `InMemoryScrivener` for journals and invoke blocks directly. +- Treat `OperationCanceledException` as cooperative shutdown when using tokens. + +## See Also + +- Architecture: Journaling and Scriveners; Windowing and Shattering. +- Samples: `src/samples/01.DiscordAgent` for end‑to‑end orchestration. diff --git a/src/Coven.Daemonology/README.md b/src/Coven.Daemonology/README.md new file mode 100644 index 0000000..7deff5a --- /dev/null +++ b/src/Coven.Daemonology/README.md @@ -0,0 +1,163 @@ +# Coven.Daemonology + +Long‑running background services (daemons) with a small, testable surface. Provides a status contract so orchestration code can deterministically start, monitor, and shut down components. + +## What’s Inside + +- IDaemon: minimal lifecycle contract (Start/Shutdown, Status) +- ContractDaemon: base class that fulfills status/failure promises via a journal +- Status: lifecycle states (Stopped, Running, Completed) +- DaemonEvent: internal event records written for status/failure changes + +## Why use it? + +- Deterministic startup/shutdown: orchestration can wait for a specific status. +- Failure propagation: surface first failure for coordinated recovery. +- Testable: status/failure are journaled; behavior is unit‑testable. +- Side‑effect‑aware: follows repo guidelines for cancellation and disposal. + +## Key Types + +- IDaemon + - `Status Status { get; }` + - `Task Start(CancellationToken cancellationToken = default)` + - `Task Shutdown(CancellationToken cancellationToken = default)` + +- ContractDaemon + - Derive to implement your own daemon. + - Requires an `IScrivener` to journal lifecycle events. + - Protected helpers for correctness: + - `Transition(Status newStatus, CancellationToken)` — write a status change and update `Status` atomically. + - `Fail(Exception error, CancellationToken)` — write first failure for observers. + - Public observers for orchestration: + - `Task WaitFor(Status target, CancellationToken)` — completes on first matching status change. + - `Task WaitForFailure(CancellationToken)` — completes on first failure. + +- Status + - `Stopped`: not yet started or fully stopped. + - `Running`: actively processing. + - `Completed`: shut down successfully and cannot be restarted. + +## Lifecycle Contract + +Implementors should: + +- Call `Transition(Status.Running)` near the end of Start once work is ready. +- Call `Transition(Status.Completed)` in Shutdown after cooperative cancellation and cleanup. +- Call `Fail(ex)` from catch blocks when unrecoverable errors occur. +- Honor the provided cancellation token and prefer linked tokens. +- Never restart after `Completed` (enforced by base class). + +The base class guarantees thread‑safe state changes and fulfills any outstanding waits by writing events to the daemon event journal. + +## Example: Minimal Custom Daemon + +```csharp +using Coven.Core; +using Coven.Daemonology; + +internal sealed class MyDaemon(IScrivener events) : ContractDaemon(events) +{ + private CancellationTokenSource? _linked; + private Task? _pump; + + public override async Task Start(CancellationToken cancellationToken = default) + { + _linked = CancellationTokenSource.CreateLinkedTokenSource(cancellationToken); + _pump = Task.Run(() => RunAsync(_linked.Token), _linked.Token); + await Transition(Status.Running, cancellationToken); + } + + public override async Task Shutdown(CancellationToken cancellationToken = default) + { + _linked?.Cancel(); + if (_pump is not null) + { + try { await _pump.ConfigureAwait(false); } catch (OperationCanceledException) { } + _pump = null; + } + await Transition(Status.Completed, cancellationToken); + } + + private async Task RunAsync(CancellationToken ct) + { + try + { + while (!ct.IsCancellationRequested) + { + // do work + await Task.Delay(50, ct); + } + } + catch (OperationCanceledException) { /* cooperative */ } + catch (Exception ex) { await Fail(ex, ct); } + } +} +``` + +## DI and Orchestration + +- Provide a journal for daemon events (most integrations call `TryAddSingleton, InMemoryScrivener>`). +- Register your daemon(s) as `ContractDaemon` so orchestration can enumerate them. +- Start daemons inside a MagikBlock and optionally wait for `Running` before proceeding. + +Sample 01 pattern (Discord Agent) starts all daemons injected via DI: + +```csharp +using Coven.Agents; +using Coven.Chat; +using Coven.Core; +using Coven.Daemonology; + +internal sealed class RouterBlock( + IEnumerable daemons, + IScrivener chat, + IScrivener agents) : IMagikBlock +{ + private readonly IEnumerable _daemons = daemons ?? throw new ArgumentNullException(nameof(daemons)); + private readonly IScrivener _chat = chat ?? throw new ArgumentNullException(nameof(chat)); + private readonly IScrivener _agents = agents ?? throw new ArgumentNullException(nameof(agents)); + + public async Task DoMagik(Empty input, CancellationToken cancellationToken = default) + { + foreach (ContractDaemon d in _daemons) + { + await d.Start(cancellationToken).ConfigureAwait(false); + // Optionally: await d.WaitFor(Status.Running, cancellationToken); + } + + // bridge chat ↔ agent work here ... + return input; + } +} +``` + +Integrations such as `Coven.Chat.Discord`, `Coven.Chat.Console`, `Coven.Agents.OpenAI`, and `Coven.Core.Streaming` register their own `ContractDaemon` implementations in DI. Consuming apps typically don’t construct daemons directly. + +## Common Patterns and Tips + +- Cancellation: always use linked tokens inside Start; pass tokens to awaited calls. +- Failure: surface first failure via `Fail(ex)`; orchestration can `await WaitForFailure()` if desired. +- Completion: use a unified Shutdown path; dispose managed resources there, then `Transition(Completed)`. +- No restarts: `Completed` is terminal; attempting to Start again throws. +- Journaling: ensure a single `IScrivener` instance is available for the current scope. + +## Referenced Samples + +- Sample 01 — Discord Agent: `src/samples/01.DiscordAgent` uses multiple daemons (Discord chat, OpenAI agent, stream windowing) and starts them from a RouterBlock. + +## Testing + +Because status and failures are journaled, behaviors are easy to test with `InMemoryScrivener`. + +See tests under `src/Coven.Daemonology.Tests` for examples like: + +- waiting for `Status.Running` after `Start()` +- waiting for `Status.Completed` after `Shutdown()` +- asserting that a completed daemon cannot restart +- propagating the first failure via `WaitForFailure()` + +## See Also + +- Root README: high‑level concepts (MagikBlocks, Scriveners, Window/Shatter) +- Architecture Guide: cancellation token guidance and cross‑cutting standards diff --git a/src/Coven.Spellcasting/README.md b/src/Coven.Spellcasting/README.md new file mode 100644 index 0000000..3b59832 --- /dev/null +++ b/src/Coven.Spellcasting/README.md @@ -0,0 +1,45 @@ +# Coven.Spellcasting + +Structured “tool” actions for agents and apps. Define spells with typed inputs/outputs, compose them into a spellbook, and optionally generate schemas. + +## What’s Inside + +- Contracts: `ISpell`, `ISpell`, `ISpell`, `ISpellContract`. +- Composition: `Spellbook`, `SpellbookBuilder`. +- Schema: `SchemaGen` for describing spell inputs/outputs. +- Definition: `SpellDefinition` (name, description, input/output types). + +## Why use it? + +- Make capabilities explicit and typed for agents/tools. +- Generate machine‑readable schemas for planning and validation. + +## Minimal Example + +```csharp +using Coven.Spellcasting; + +public sealed class AddSpell : ISpell +{ + public SpellDefinition Definition => new( + Name: "add", + Description: "Add two integers", + InputType: typeof(AddInput), + OutputType: typeof(AddResult)); + + public Task Cast(AddInput input, CancellationToken ct = default) + => Task.FromResult(new AddResult(input.A + input.B)); +} + +var book = new SpellbookBuilder() + .AddSpell(new AddSpell()) + .Build(); + +// Optional: schema for tool wiring +string jsonSchema = SchemaGen.Generate(book); +``` + +## See Also + +- Architecture: Abstractions and Branches. +- Packages: `Coven.Agents` for agent orchestration. diff --git a/src/Coven.Transmutation/README.md b/src/Coven.Transmutation/README.md new file mode 100644 index 0000000..fe111a1 --- /dev/null +++ b/src/Coven.Transmutation/README.md @@ -0,0 +1,85 @@ +# Coven.Transmutation + +Pure transformations between types used across Coven. Transmuters describe how one type becomes another, without side‑effects. + +## What’s Inside + +- ITransmuter: one‑way, pure transformation. +- IBiDirectionalTransmuter: two‑way transformation (afferent/efferent directions). +- IBatchTransmuter: many‑to‑one transformation over a window of chunks. +- BatchTransmuteResult: output plus optional remainder chunk. +- LambdaTransmuter: adapter to build a transmuter from a delegate. + +## Principles + +- Pure: no observable side‑effects (no I/O, logging, mutation of external state). +- Deterministic: same inputs → same outputs. +- Cancel‑aware: accept and honor `CancellationToken` where work may be long‑running. +- Exception‑transparent: throw upstream; do not swallow or log. + +## One‑Way Transmutation + +```csharp +using Coven.Transmutation; + +// Simple pure mapping (e.g., DTO → domain) +public sealed class UserDtoToModel : ITransmuter +{ + public Task Transmute(UserDto Input, CancellationToken ct = default) + => Task.FromResult(new User(Input.Id, Input.Name.Trim())); +} +``` + +## Bidirectional Transmutation + +```csharp +using Coven.Transmutation; + +// Two pure mappings in opposite directions (afferent/efferent) +public sealed class UserBiMap : IBiDirectionalTransmuter +{ + public Task TransmuteAfferent(UserDto Input, CancellationToken ct = default) + => Task.FromResult(new User(Input.Id, Input.Name.Trim())); + + public Task TransmuteEfferent(User Output, CancellationToken ct = default) + => Task.FromResult(new UserDto { Id = Output.Id, Name = Output.Name }); +} +``` + +## Batch Transmutation (Many→One) + +```csharp +using Coven.Transmutation; + +// Concatenate chunk text into a single output +public sealed class TextBatch : IBatchTransmuter +{ + public Task> Transmute(IEnumerable input, CancellationToken ct = default) + { + string text = string.Concat(input.Select(c => c.Text)); + return Task.FromResult(new BatchTransmuteResult( + new MyOutput(text), + HasRemainder: false, + Remainder: default)); + } +} +``` + +Remainders are useful when only part of the last chunk is consumed; the unused tail returns as `Remainder` to seed the next window. + +## Delegate Adapter + +```csharp +var t = new LambdaTransmuter((i, ct) => Task.FromResult(i.ToString())); +``` + +## Tips + +- Keep transmuters small and testable; stitch them together via DI. +- Prefer immutable inputs/outputs to reinforce purity. +- Separate policy decisions (window/shatter) from transmutation logic. + +## See Also + +- Architecture: Windowing and Shattering; Journaling and Scriveners. +- Packages using transmuters: `Coven.Chat`, `Coven.Agents`, `Coven.Agents.OpenAI`. diff --git a/src/samples/01.DiscordAgent/README.md b/src/samples/01.DiscordAgent/README.md index 5e793d2..b97ece6 100644 --- a/src/samples/01.DiscordAgent/README.md +++ b/src/samples/01.DiscordAgent/README.md @@ -8,7 +8,7 @@ Run a Discord-backed chat agent powered by OpenAI, wired together using Coven’ - Agent Integration: Uses `Coven.Agents.OpenAI` to map chat to `AgentEntry` prompts/thoughts/responses. Streaming is enabled for responsive output. - Router (MagikBlock): `RouterBlock` is a simple `IMagikBlock` that bridges chat ↔ agents by reading/writing via `IScrivener` logs (journal-first design). - Daemons: Discord and OpenAI run as `ContractDaemon`s managed by the host lifecycle (start/shutdown cooperatively). -- Windowing: Output chunking is governed by `IWindowPolicy` for paragraph-first aggregation and max-length capping. +- Semantic windowing: Output chunking is governed by `IWindowPolicy` for paragraph-first aggregation and max-length capping. - Transmutation: `DiscordOpenAITemplatingTransmuter` customizes how OpenAI request/response items are templated (e.g., decorating with Discord username/model markers). Key files: