feat(provider): Wave 1.5/2/3/4 obs, identity, quota, CLI probes - #239
Open
agnitum2009 wants to merge 12 commits into
Open
feat(provider): Wave 1.5/2/3/4 obs, identity, quota, CLI probes#239agnitum2009 wants to merge 12 commits into
agnitum2009 wants to merge 12 commits into
Conversation
…essage_bureau Introduces Rust-backed Python extension modules to reduce Python orchestration memory for stable subsystems: - ccb-heartbeat, ccb-mailbox, ccb-message-bureau (Rust implementations) - ccb-py-heartbeat (heartbeat Python binding) - ccb-py-mailbox (mailbox_kernel + message_bureau Python binding) Also brings in the supporting crates they depend on: - ccb-project, ccb-storage, ccb-storage-classification, ccb-jobs - ccb-types, ccb-runtime-env, ccb-ui-text Validation: - cargo test -p ccb-py-heartbeat -p ccb-py-mailbox -p ccb-jobs -p ccb-storage passes - cargo clippy -p ccb-py-heartbeat -p ccb-py-mailbox -p ccb-jobs -p ccb-storage clean - Python smoke tests for both extension modules pass
- Route evaluate_heartbeat through ccb_py_heartbeat when the extension is installed and CCB_HEARTBEAT_RUST is not explicitly disabled. - Fall back to the Python implementation automatically in source checkouts that do not have the Rust extension built. - Convert Rust results back to Python dataclasses so existing consumers (job_heartbeat_runtime/tick.py) need no changes. - Add tests for the env gate and backend selection.
…ility - Add record_codec compatibility layer so Rust-written mailbox, inbox, and lease records carry schema_version/record_type headers expected by Python readers. - Wire ccb-mailbox stores through record_codec for cross-language round-trips. - Extend ccb-py-mailbox bindings with apply_incremental_summary_update, upsert_mailbox_summary, rewrite_head, and Ellipsis-aware optional active_inbound_event_id mapping. - Add env-gated mailbox_kernel/__init__.py shim that delegates to the Rust extension when available and falls back to Python otherwise. Relates to PR SeemSeam#237.
Add an opt-in Rust backend for message_bureau via CCB_MESSAGE_BUREAU_RUST=1. The proven Python implementation remains the default, so production is unaffected. Rust side: - Extend ccb-py-mailbox PyO3 bindings for MessageBureauFacade and MessageBureauControlService (recording, query, ack_reply, callback edges). - Add record_codec wrappers for Message/Attempt/Reply/CallbackEdge stores so Python readers see schema_version/record_type compatible JSONL. - Serialize DeliveryScope as single/group/broadcast strings for Python parity. - Add serde aliases for DeliveryScope in ccb-jobs. - Validate target mailbox exists in queue_summary before building detail. Python side: - lib/message_bureau/__init__.py now imports the Rust extension only when CCB_MESSAGE_BUREAU_RUST=1; otherwise it re-exports the existing Python classes. Results from Rust are converted back to Python dataclasses. Validation: - cargo test -p ccb-mailbox -p ccb-py-mailbox: pass - cargo clippy -p ccb-mailbox -p ccb-py-mailbox -p ccb-jobs: clean - test_message_bureau_*.py + test_v2_message_bureau_*.py (85 tests): pass with both Python and Rust backends.
…p RSS Reduce the amount of Python code loaded at ccbd startup by only loading provider backends and request handlers that are actually needed. Provider backend filtering: - Add optional ``providers`` filter to provider_core registry/catalog helpers and provider_execution registry so only configured providers are imported. - ``builtin_backends.build_builtin_backends`` now conditionally imports each backend module instead of importing all 18 providers up front. - ``ccbd.app_runtime.bootstrap`` computes configured providers from ``config.agents`` and passes the set to ``build_default_provider_catalog`` and ``build_default_execution_registry``. - ``ccbd.app_runtime.service_graph`` builds session bindings once for the configured providers and passes them to ``RuntimeService`` and ``HealthMonitor`` instead of letting each service rebuild the full map. Local imports in hot paths: - Move provider_backends imports out of ``cli.services.provider_hooks`` top level and into the per-provider branches of ``_materialize_provider_home``, ``provider_hook_home_root``, and ``_route_claude_binary_cache_if_possible``. - Move ``provider_backends.codex...resolve_codex_home_layout`` and ``provider_hooks.activity`` imports into the functions that use them in ``ccbd.project_view``. Lazy handlers: - ``ccbd.app_runtime.handlers`` no longer imports ``ccbd.handlers`` at the top level. ``register_handlers`` now registers lightweight wrappers that import and construct the real handler on the first request. This defers loading all handler modules (and their heavy dependencies such as reload_apply_graph and provider_backends.pane_log_support) until first use. Measured improvement (PYTHONPATH=lib, import ccbd.app): - RSS delta: ~36 MB -> ~30.4 MB - provider_backends modules loaded: 454 -> 237 - ``ccbd.handlers`` no longer loaded at import time Validation: - test_ccbd*.py + test_provider*.py: 639 passed, 2 skipped - test_message_bureau_*.py + test_v2_message_bureau_*.py: 85 passed - test_ccbd_service_graph.py, test_ccbd_project_view.py, test_v2_provider_core_registry.py, test_v2_provider_catalog.py, test_provider_hook_settings.py: all passed
…ng, doctor & reply diagnostics Wave 1 of provider-error observability. The Codex account usage-limit banner previously only appeared on the tmux pane; CCB recorded empty model_empty_output and reported the agent "healthy". Now the pane content is parsed and surfaced. - codex_pane.py: USAGE_LIMIT_MARKERS, usage_limit status, PaneStatus.retry_after, parse_retry_after() - health_assessment: content-aware ProviderPaneAssessment (pane_signal_state/reason/retry_after/pane_tail) + health_from_pane_signal() - ping_runtime: provider_health in build_agent_payload diagnostics + handler wiring via health_monitor - ops_views_doctor: render provider_health line - codex execution + provider-finish-hook: attach pane_tail/error_kind/retry_after to diagnostics on empty/error turns Additive and backward-compatible; non-codex providers remain content-blind until a later wave. 21 new tests; full suite green (only pre-existing mobile-gateway network failures remain). Co-Authored-By: Claude <noreply@anthropic.com>
- Kimi completion detection: turn_end is not terminal; pane idle/stable marks real completion - Provider/model/account quota buckets with degraded state and retry-after - Identity injection helpers and CLI identity/probe services - MCP delegation schema/tool updates for roster/peer_status - Provider pane health assessment and ping payload extensions
…eout is_pane_alive and get_pane_content called tmux_run_fn directly with no timeout, so a hung tmux server could block the calling thread indefinitely. Route both through run_tmux_capture(timeout=2.0) — matching pane_exists and describe_pane in the same module — so a stuck tmux returns "unavailable" after 2s instead of hanging. Latent bug surfaced while diagnosing codex no-reply stalls (not the root cause, but correct hygiene). Co-Authored-By: Claude <noreply@anthropic.com>
When an ask gets no reply, distinct failure modes (unreachable /
reached-but-quota-full / reached-but-errored / answered-but-CCB-lost-it)
all surfaced identically as status=running + reply=empty, forcing senders
to guess. Add a 17-code NoReplyReason taxonomy so each non-success terminal
job carries an attributable reason the sender can read.
Foundation:
- lib/ccbd/no_reply_reason.py: NoReplyReason enum + descriptions + triage.
- JobRecord.no_reply_reason/no_reply_detail, persisted to jobs.jsonl;
finalization complete_job attaches/requires it on non-success terminals.
- `ccb why <job_id>` CLI surfaces reason + description + triage hint;
non-success CCB_REPLY body carries a [no_reply_reason=...] prefix.
Provider can't-answer (B):
- pane-signal -> terminal on the normal poll path (codex/kimi/claude) plus
a health->terminalize bridge, using TWO-TIER markers: high-confidence
specific banners (e.g. "hit your usage limit") drive terminalization;
broad keywords ("quota", "usage limit", "rate limit", ...) stay
diagnostics-only, so a healthy agent whose output merely discusses these
topics is not false-positively killed (parse_codex_pane_status strict=,
high_confidence_signal()).
- codex crash detection (provider_crashed) via pane_dead delegation.
Unreachable (A):
- queue fail-fast stranded degraded-bucket jobs after a grace
(dispatch_handoff_skipped_degraded) + agent_busy_queue_blocked tag.
Answered-but-not-delivered (C):
- reply_delivery_stalled flag (terminal job with reply unconsumed >120s).
- kimi premature-completion: PANE_FALLBACK_STABLE_SECS 10 -> 45 (heuristic;
noted insufficient alone — kimi's slow deliberation can still exceed it;
a stronger finish signal is a follow-up).
Tests: provider_pane_strict_terminalization, false-positive regression,
kimi completion, native CLI completion, execution service.
Co-Authored-By: Claude <noreply@anthropic.com>
kimi-code emitted no authoritative task-complete signal: native TurnEnd is per-message and the pane idle prompt (`│ >`/`K2.7 Code`/`context:`) shows BOTH at true completion and during inter-step thinking pauses, so the pane-idle-stable heuristic closed jobs prematurely and captured only a deliberation preamble. Adopt the existing CCB_DONE:<anchor> sentinel protocol (droid uses it) so kimi itself declares completion. - wrap_kimi_prompt injects the CCB_DONE:<anchor> completion instruction. - _kimi_sentinel_buffer joins native TurnEnd reply + pane observation reply + the carved pane reply region; done_seen = is_done_text(buffer, anchor). Authoritative fast completion (reason=kimi_sentinel_complete, completion_source=sentinel), bypassing the 45s pane-idle fallback, which is kept (kimi_pane_idle_complete) along with native_turn_timeout for the no-marker case. - _extract_kimi_reply_region carves the assistant reply region (after the last CCB_REQ_ID prompt anchor, up to the idle input box). This excludes the prompt's inline CCB_DONE example (avoids false-fire) and the trailing input-box line (so is_done_text's tail scan matches kimi's emitted marker, not the box). _strip_kimi_reply_bullet drops the `●`/`•` decoration. - When done_seen, the reply is re-extracted from the sentinel buffer (content before the marker) instead of trusting a possibly-stale partial observation.reply (kimi emits multiple bullets; an early poll may have captured only the first). Prior reply kept only as a fallback. Verified end-to-end on a real multi-file task: kimi Reads, answers, emits CCB_DONE; completion_reason=kimi_sentinel_complete; reply = full answer before the marker (not the first bullet). Tests: test_kimi_sentinel_completion.py (sentinel hit, no-marker fallback + timeout bound, wrong-anchor ignored, reply extraction, prompt-echo exclusion). Co-Authored-By: Claude <noreply@anthropic.com>
Codex protocol logs remain the success authority, but a missing terminal event must not leave the active job holding the queue indefinitely. Add a bounded degraded timeout, classify the timeout state by anchor/reply evidence, and preserve provider no-reply detail so operators can choose resend, harvest, or repair actions from evidence instead of guessing. Constraint: Fix must use daemon-side reliability/no-reply mechanisms and avoid treating pane idle as successful completion. Rejected: Pane-idle success terminalization | Codex panes can be idle during non-terminal states and would create Kimi-style premature completion risk. Confidence: high Scope-risk: moderate Directive: Keep protocol_log as Codex completion authority; only use reliability timeout for degraded/incomplete queue release with explicit no_reply_detail. Tested: PYTHONPATH=.test-deps:lib python -m pytest test/test_v2_execution_service.py test/test_provider_execution_service_runtime.py test/test_no_reply_reason.py -q Tested: PYTHONPATH=.test-deps:lib python -m pytest test -q -> 3508 passed, 2 skipped, 8 known unrelated failures Not-tested: Live post-timeout Codex job aging for 900 seconds in production.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This PR rolls up the Wave 1.5/2/3/4 Python-side fixes for provider observation, identity awareness, quota handling, and CLI diagnostics.
What's included
Notes