Repository navigation
feat(gateway): add deterministic watchdog alerts - #112
Conversation
2678f31 to
519e876
Compare
thefullgc
left a comment
There was a problem hiding this comment.
Thanks — the shape is right (alert-only, outside core, opt-out, injectable thresholds/clock, per-turn dedupe, safe onAlert). But as wired, it won't fire in production. Details below.
Must fix
1. Event names don't match what the gateway actually receives.
The watchdog switches on pi-style AgentEvent names (turn_start, message_start, tool_execution_start, ...). The gateway forwarder gets the RPC event stream, and rpc-mode.ts emits raw store events via facade.subscribe((event) => output(event)) — i.e. AGENT_TURN_START, AGENT_MESSAGE_START, AGENT_MESSAGE_CHUNK, TOOL_EXECUTION_START/UPDATE/END, AGENT_TURN_END, AGENT_TURN_COMPLETED, with data under payload. packages/channels/runtime.ts, the other consumer of this same stream, matches AGENT_TURN_COMPLETED / USER_MESSAGE accordingly. The tests pass only because they feed hand-written lowercase events. Please match the store event types and add an integration test driven by a real reactor event sequence (or a recorded RPC trace).
2. Repeated-call counting is scoped to a single LLM round.
In Pizza, AGENT_TURN_START/AGENT_TURN_END wrap every LLM call / tool-call round (see the isIdleEvent comment in channels/runtime.ts). Resetting toolCounts per turn means a model re-issuing the same command across rounds never reaches the threshold; it only fires for 5 identical parallel calls in one message. Count from USER_MESSAGE to AGENT_TURN_COMPLETED instead.
Please clarify vs. existing core behavior
3. The reactor already has (a) a stream-idle watchdog (_streamIdleTimeoutMs, 10 min, aborts the LLM call and retries) and (b) loop detection (MAX_CONSECUTIVE_IDENTICAL_TOOL_ROUNDS = 6 → AGENT_TURN_COMPLETED { reason: "loop_detected" }). This PR uses 120s / 5 calls. Please document the relationship (gateway = earlier alert only, core = enforcement) and align thresholds so the two don't contradict each other. Also worth considering: the failure modes core can't see — tell not delivered, agent process unresponsive, queued work never starting — are arguably the most valuable thing a gateway-side watchdog can cover.
Smaller
4. toolActive is a single boolean; with parallel tool calls the first tool_execution_end clears it while others still run. Track a set of in-flight toolCallIds.
5. stableStringify on full args for every tool call is costly for large _write payloads and makes big map keys. Store a hash like the reactor's hashArguments.
6. Alerts go to stderr and an in-process watchdog event only; nothing is broadcast to clients yet. Fine for tier 1, but the "a later panel can consume them" path needs a broadcast frame.
7. The test/scheduler/engine.test.ts flake fix is unrelated — please split it into its own PR; happy to merge that one quickly.
b8ab0ca to
0c2f7ce
Compare
|
Review feedback addressed:\n\n- Must fix 1: watchdog now consumes the raw store events the gateway forwarder actually emits (USER_MESSAGE, AGENT_TURN_START, AGENT_MESSAGE_START/CHUNK/END, TOOL_EXECUTION_START/UPDATE/END, AGENT_TURN_END/COMPLETED) with payload fields. Tests drive that real shape now.\n- Must fix 2: repeated calls are counted from USER_MESSAGE to AGENT_TURN_COMPLETED, spanning LLM/tool rounds. Added an integration sequence that repeats across AGENT_TURN_START/END and still alerts.\n- 3: documented gateway vs reactor behavior and aligned thresholds: 120s alert vs 10min core abort; 6 calls vs 6 identical rounds loop guard; gateway stays alert-only.\n- 4: in-flight tools are now a map keyed by ool_call_id, with a parallel-tool test.\n- 5: arguments use a reactor-style FNV-1a hash; no full args retained as keys.\n- 6: alerts are broadcast as WATCHDOG_ALERT on the channel event stream, with a fan-out test.\n- 7: scheduler fix split into #115 and removed from this branch.\n\n |
Observe the event stream the gateway already receives from pooled agents and emit alerts for idle model streams, idle tools, and repeated tool calls. This is the gateway-side, alert-only first tier: no model calls, no automatic cancellation, and no core-runtime dependency for standalone CLI sessions. Alerts are logged and exposed on the GatewayServer watchdog event; tests cover the detector and the gateway event-stream wiring.
The gateway forwarder emits raw store events from rpc-mode, not pi AgentEvent names. Switch the watchdog to AGENT_*/TOOL_* payloads, scope repeated-tool counting from USER_MESSAGE to AGENT_TURN_COMPLETED so it spans LLM rounds, track parallel tools by call id, hash arguments with the reactor-style FNV hash, and document the relationship to the reactor stream-idle/loop guards. Also broadcast WATCHDOG_ALERT on the existing channel event stream and replace the hand-written lowercase-event tests with a real store-event sequence plus a channel fan-out assertion.
0c2f7ce to
34b65ae
Compare
Summary
First, deliberately small tier of the watchdog discussed in #110, following the maintainer guidance to keep it out of core and start with deterministic rules + alerts only.
The gateway already receives every pooled agent's raw store-event stream for channel fan-out. This adds a gateway-side watchdog that observes that same stream and reports:
There are no model calls and no automatic cancellation in this PR. Alerts are logged, emitted on the
GatewayServerwatchdogevent, and broadcast as aWATCHDOG_ALERTframe on the existing channel event stream.Review follow-ups
AgentEventnames to the raw store event names the gateway actually receives:USER_MESSAGE,AGENT_TURN_START,AGENT_MESSAGE_START/CHUNK/END,TOOL_EXECUTION_START/UPDATE/END,AGENT_TURN_END/COMPLETED.USER_MESSAGE→AGENT_TURN_COMPLETED, so a model re-issuing the same call across LLM/tool rounds is detected.tool_call_ids instead of onetoolActiveboolean.WATCHDOG_ALERTframes for panels/channels.Design
packages/gateway/watchdog.tsis a small state machine with injectablenowfor tests.ensureForwarder()path, so it only sees agents the gateway already owns. Standalone CLI sessions do not pay for it.watchdog: falsedisables it; thresholds andonAlertare injectable throughGatewayServerOptions.Testing
WATCHDOG_ALERTreaches an attached client.npm run buildpasses.Related: #110. This is alert-only infrastructure; abort/retry, agent-unresponsive/tell-not-delivered checks, and an independent watchdog session are follow-ups.