Skip to content

feat(gateway): add deterministic watchdog alerts - #112

Merged
thefullgc merged 3 commits into
pizza-agent:mainfrom
Lumea-art:feat/gateway-watchdog
Nov 15, 2026
Merged

thefullgc merged 3 commits into
pizza-agent:mainfrom
Lumea-art:feat/gateway-watchdog

Conversation

@Lumea-art

@Lumea-art Lumea-art commented Nov 14, 2026 •

Copy link
Copy Markdown
Contributor

Summary

First, deliberately small tier of the watchdog discussed in #110, following the maintainer guidance to keep it out of core and start with deterministic rules + alerts only.

The gateway already receives every pooled agent's raw store-event stream for channel fan-out. This adds a gateway-side watchdog that observes that same stream and reports:

  • a model turn has produced no output for too long
  • a running tool has produced no progress for too long
  • the same tool call (same name and arguments) repeats within one user task

There are no model calls and no automatic cancellation in this PR. Alerts are logged, emitted on the GatewayServer watchdog event, and broadcast as a WATCHDOG_ALERT frame on the existing channel event stream.

Review follow-ups

  • Switched from pi AgentEvent names to the raw store event names the gateway actually receives: USER_MESSAGE, AGENT_TURN_START, AGENT_MESSAGE_START/CHUNK/END, TOOL_EXECUTION_START/UPDATE/END, AGENT_TURN_END/COMPLETED.
  • Repeated-call counting now spans USER_MESSAGE → AGENT_TURN_COMPLETED, so a model re-issuing the same call across LLM/tool rounds is detected.
  • Parallel tools are tracked by a map of in-flight tool_call_ids instead of one toolActive boolean.
  • Tool arguments are hash-signed with the reactor's FNV-1a style hash instead of storing full argument JSON as map keys.
  • Thresholds and docs now align with core: gateway stream alert at 120s (core aborts at 10min); repeated-call alert at 6 calls in a task (core loop guard is 6 identical rounds). Gateway remains alert-only.
  • Alerts are broadcast as WATCHDOG_ALERT frames for panels/channels.
  • The scheduler flake fix was split into test(scheduler): make lock-migration race deterministic #115.

Design

  • packages/gateway/watchdog.ts is a small state machine with injectable now for tests.
  • Wired into the existing ensureForwarder() path, so it only sees agents the gateway already owns. Standalone CLI sessions do not pay for it.
  • State is forgotten when the agent is torn down.
  • Alerts are deduplicated per task/rule, so a long stall does not log every sweep.
  • watchdog: false disables it; thresholds and onAlert are injectable through GatewayServerOptions.
  • Defaults: 5s sweep, 120s model idle, 10min tool idle, 6 repeated calls.

Testing

  • Unit tests use the raw store event sequence and cover stream idle, tool idle, repeated calls across rounds, parallel tools, task reset, and task completion.
  • Gateway integration test emits a store-event sequence and verifies both stream-idle and cross-round repeated-call alerts.
  • Channel fan-out test verifies WATCHDOG_ALERT reaches an attached client.
  • npm run build passes.
  • Related gateway tests: 23 passed.

Related: #110. This is alert-only infrastructure; abort/retry, agent-unresponsive/tell-not-delivered checks, and an independent watchdog session are follow-ups.

@Lumea-art
Lumea-art force-pushed the feat/gateway-watchdog branch 2 times, most recently from 2678f31 to 519e876 Compare November 14, 2026 11:55

@thefullgc thefullgc left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks — the shape is right (alert-only, outside core, opt-out, injectable thresholds/clock, per-turn dedupe, safe onAlert). But as wired, it won't fire in production. Details below.

Must fix

1. Event names don't match what the gateway actually receives.
The watchdog switches on pi-style AgentEvent names (turn_start, message_start, tool_execution_start, ...). The gateway forwarder gets the RPC event stream, and rpc-mode.ts emits raw store events via facade.subscribe((event) => output(event)) — i.e. AGENT_TURN_START, AGENT_MESSAGE_START, AGENT_MESSAGE_CHUNK, TOOL_EXECUTION_START/UPDATE/END, AGENT_TURN_END, AGENT_TURN_COMPLETED, with data under payload. packages/channels/runtime.ts, the other consumer of this same stream, matches AGENT_TURN_COMPLETED / USER_MESSAGE accordingly. The tests pass only because they feed hand-written lowercase events. Please match the store event types and add an integration test driven by a real reactor event sequence (or a recorded RPC trace).

2. Repeated-call counting is scoped to a single LLM round.
In Pizza, AGENT_TURN_START/AGENT_TURN_END wrap every LLM call / tool-call round (see the isIdleEvent comment in channels/runtime.ts). Resetting toolCounts per turn means a model re-issuing the same command across rounds never reaches the threshold; it only fires for 5 identical parallel calls in one message. Count from USER_MESSAGE to AGENT_TURN_COMPLETED instead.

Please clarify vs. existing core behavior

3. The reactor already has (a) a stream-idle watchdog (_streamIdleTimeoutMs, 10 min, aborts the LLM call and retries) and (b) loop detection (MAX_CONSECUTIVE_IDENTICAL_TOOL_ROUNDS = 6 → AGENT_TURN_COMPLETED { reason: "loop_detected" }). This PR uses 120s / 5 calls. Please document the relationship (gateway = earlier alert only, core = enforcement) and align thresholds so the two don't contradict each other. Also worth considering: the failure modes core can't see — tell not delivered, agent process unresponsive, queued work never starting — are arguably the most valuable thing a gateway-side watchdog can cover.

Smaller

4. toolActive is a single boolean; with parallel tool calls the first tool_execution_end clears it while others still run. Track a set of in-flight toolCallIds.

5. stableStringify on full args for every tool call is costly for large _write payloads and makes big map keys. Store a hash like the reactor's hashArguments.

6. Alerts go to stderr and an in-process watchdog event only; nothing is broadcast to clients yet. Fine for tier 1, but the "a later panel can consume them" path needs a broadcast frame.

7. The test/scheduler/engine.test.ts flake fix is unrelated — please split it into its own PR; happy to merge that one quickly.

@Lumea-art

Copy link
Copy Markdown
Contributor Author

Review feedback addressed:\n\n- Must fix 1: watchdog now consumes the raw store events the gateway forwarder actually emits (USER_MESSAGE, AGENT_TURN_START, AGENT_MESSAGE_START/CHUNK/END, TOOL_EXECUTION_START/UPDATE/END, AGENT_TURN_END/COMPLETED) with payload fields. Tests drive that real shape now.\n- Must fix 2: repeated calls are counted from USER_MESSAGE to AGENT_TURN_COMPLETED, spanning LLM/tool rounds. Added an integration sequence that repeats across AGENT_TURN_START/END and still alerts.\n- 3: documented gateway vs reactor behavior and aligned thresholds: 120s alert vs 10min core abort; 6 calls vs 6 identical rounds loop guard; gateway stays alert-only.\n- 4: in-flight tools are now a map keyed by ool_call_id, with a parallel-tool test.\n- 5: arguments use a reactor-style FNV-1a hash; no full args retained as keys.\n- 6: alerts are broadcast as WATCHDOG_ALERT on the channel event stream, with a fan-out test.\n- 7: scheduler fix split into #115 and removed from this branch.\n\n
pm run build and the 23 related gateway tests pass; CI is running again.

Observe the event stream the gateway already receives from pooled agents and emit alerts for idle model streams, idle tools, and repeated tool calls. This is the gateway-side, alert-only first tier: no model calls, no automatic cancellation, and no core-runtime dependency for standalone CLI sessions.

Alerts are logged and exposed on the GatewayServer watchdog event; tests cover the detector and the gateway event-stream wiring.
The gateway forwarder emits raw store events from rpc-mode, not pi AgentEvent names. Switch the watchdog to AGENT_*/TOOL_* payloads, scope repeated-tool counting from USER_MESSAGE to AGENT_TURN_COMPLETED so it spans LLM rounds, track parallel tools by call id, hash arguments with the reactor-style FNV hash, and document the relationship to the reactor stream-idle/loop guards.

Also broadcast WATCHDOG_ALERT on the existing channel event stream and replace the hand-written lowercase-event tests with a real store-event sequence plus a channel fan-out assertion.
@Lumea-art
Lumea-art force-pushed the feat/gateway-watchdog branch from 0c2f7ce to 34b65ae Compare November 15, 2026 01:34
@Lumea-art

Copy link
Copy Markdown
Contributor Author

Rebased onto latest main after #113/#114 landed. All CI checks are green again; ready for re-review.

@thefullgc thefullgc left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM!

@thefullgc
thefullgc merged commit 01be3ee into pizza-agent:main Nov 15, 2026
7 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants