Skip to content

Fix awaiting an event handler from a background task: caught exceptions and finished superseding runs - #7907

Open
FarhanAliRaza wants to merge 2 commits into
mainfrom
claude/awaited-handler-from-background-7839
Open

FarhanAliRaza wants to merge 2 commits into
mainfrom
claude/awaited-handler-from-background-7839

Conversation

@FarhanAliRaza

@FarhanAliRaza FarhanAliRaza commented Nov 15, 2026 •

Copy link
Copy Markdown
Contributor

All Submissions:

  • Have you followed the guidelines stated in CONTRIBUTING.md file?
  • Have you checked to ensure there aren't any other open Pull Requests for the desired changed?

Type of change

  • Bug fix (non-breaking change which fixes an issue)

Changes To Core Features:

  • Have you added an explanation of what your changes do and why you'd like us to include them?
  • Have you written new tests for your core changes, as applicable?
  • Have you successfully ran tests with your changes locally?

Problem

Awaiting an event handler from a background task with no state lock held enqueues the event and awaits its future. Two things differed from the inline await (#7839), both reproduced on main:

  1. A caught exception was still reported. _resolve_future set the exception on the future (so the awaiting task received it) and then also ran the backend exception handler, so the user got the error toast even though the task caught the exception. If the task did not catch it, it was reported twice: once for the awaited event and once for the background task.
  2. A finished supersedes=True run was reported as cancelled. _cancel_older_chains cancelled the previous invocation's future whenever it was not done. A handler task that has returned resolves its future in a done callback, one loop iteration later. A newer invocation enqueued in that window cancelled the future of a run that had completed and committed its writes. The in-memory state manager hits the window every time: the second background task is waiting for the state lock the first search holds, and once it gets the lock it reaches enqueue without suspending, ahead of the first task's done callback. Redis suspends on the way, so the future is already resolved.

Fix

  • EventFuture gets an awaited flag, set by execute_event_inline on the future it is about to await. When _resolve_future delivers an exception to an awaited future, it stops there: no telemetry, no backend exception handler. An exception the caller does not catch fails the background task, whose own failure path reports it once.
  • _cancel_older_chains leaves alone a future whose task is already done and only cancels the events that run chained; the done callback then resolves the future with the run's result (or exception). A run that is still in progress is cancelled as before.

Not changed

  • The inline (lock held) await path, and other code that awaits an event future directly (e.g. upload streams): those futures are not marked, so their exceptions are reported as before.
  • If the awaiting caller is already gone (future cancelled) when the event fails, the exception is still reported to the backend exception handler.
  • Other external cancellations (disconnect, processor stop) are untouched; only supersession checked for a finished run.
  • No state manager change was needed: the memory manager only exposed the ordering.

Tests

  • tests/units/reflex_base/event/processor/test_base_state_processor.py
    • test_exception_of_handler_awaited_from_background_is_the_callers: caught → backend handler not called; uncaught → called exactly once.
    • test_superseding_handler_awaited_from_background_returns_finished_result (memory and Redis managers): both tasks get their results.
  • tests/units/reflex_base/event/processor/test_event_processor.py
    • test_superseding_event_keeps_the_result_of_a_finished_run: deterministic version of the window (enqueue between the handler returning and its done callback); the finished run resolves with its result and its chained event is cancelled.
  • tests/integration/tests_playwright/test_background_task.py
    • test_await_failing_handler_from_background_task in the existing app: no toast when the background task catches the exception.

All four fail on main (except the Redis parametrization, which is the reference behaviour) and pass with the fix.

Run locally:

  • uv run pytest tests/units/reflex_base/event tests/units/test_state.py tests/units/istate -q — passes
  • PLAYWRIGHT_BROWSERS_PATH=... uv run pytest tests/integration/tests_playwright/test_background_task.py -q — 8 passed
  • uv run ruff check ., uv run ruff format . — clean
  • uv run pyright on the changed files — 0 errors

closes #7839

https://claude.ai/code/session_01LwJUCU8RtEco15mQvkxdQb


Generated by Claude Code

View guided diff Turn on auto-fix

An exception raised by a handler awaited from a background task (no state
lock held, so the event is enqueued) was delivered to the awaiting task and
also reported to the backend exception handler, showing the error toast even
when the task caught it. The enqueued event's future is now marked as awaited,
and its exception is left to the caller: if the task does not catch it, the
task's own failure reports it once.

A supersedes=True handler run that had returned, but whose task done callback
had not yet resolved its future, was cancelled by a newer invocation enqueued
in that window, so the awaiting task got CancelledError for a run that
completed. The in-memory state manager hits the window because a task waiting
for the state lock resumes without suspending before it enqueues. Superseding
such a run now only cancels the events it chained.

Closes #7839

Claude-Session: https://claude.ai/code/session_01LwJUCU8RtEco15mQvkxdQb
@FarhanAliRaza
FarhanAliRaza requested a review from a team as a code owner November 15, 2026 09:14

@cubic-dev-ai cubic-dev-ai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

1 issue found across 7 files

Prompt for AI agents (unresolved issues)

Check if these issues are valid — if so, understand the root cause of each and fix them. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. If appropriate, use sub-agents to investigate and fix each issue separately.


<file name="tests/units/reflex_base/event/processor/test_base_state_processor.py">

<violation number="1" location="tests/units/reflex_base/event/processor/test_base_state_processor.py:5704">
P2: This fixed sleep does not guarantee that the second background run has reached the superseding enqueue, so the test can pass without exercising the race it claims to cover. Synchronize on an event set by the second run immediately before it awaits `search`, or use another deterministic barrier instead of a timed sleep.</violation>
</file>

Reply with feedback, questions, or to request a fix.

View guided diff | Turn on auto-fix | Re-trigger cubic

token, Event.from_event_type(BgSearchState.run("second"))[0]
)
# Let the second task start and queue for the lock the search holds.
await asyncio.sleep(0.1)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2: This fixed sleep does not guarantee that the second background run has reached the superseding enqueue, so the test can pass without exercising the race it claims to cover. Synchronize on an event set by the second run immediately before it awaits search, or use another deterministic barrier instead of a timed sleep.

Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At tests/units/reflex_base/event/processor/test_base_state_processor.py, line 5704:

<comment>This fixed sleep does not guarantee that the second background run has reached the superseding enqueue, so the test can pass without exercising the race it claims to cover. Synchronize on an event set by the second run immediately before it awaits `search`, or use another deterministic barrier instead of a timed sleep.</comment>

<file context>
@@ -5582,3 +5582,131 @@ def ping(self):
+            token, Event.from_event_type(BgSearchState.run("second"))[0]
+        )
+        # Let the second task start and queue for the lock the search holds.
+        await asyncio.sleep(0.1)
+        release.set()
+        await asyncio.wait_for(
</file context>

@greptile-apps

greptile-apps Bot commented Nov 15, 2026 •

Copy link
Copy Markdown
Contributor

RetriggerView in GreptileConfidence Score: 5/5

[Medium impact] The PR appears safe to merge; the previous result-loss issue is fixed.

Summary

Awaited handlers now leave caught errors to their callers and keep results from runs that finished before being replaced.

  • Background tasks can catch awaited handler errors without a separate backend report.
  • Finished superseding handlers keep their results when newer work arrives.

Reviews (2) · Last reviewed commit: "Resolve a finished event's future before..." · Reviewed by Greptile

The next queued handler for a token starts eagerly on Python 3.12+, and could
supersede the run that just finished while its future was still unresolved and
its task already forgotten, cancelling a completed run.

Claude-Session: https://claude.ai/code/session_01LwJUCU8RtEco15mQvkxdQb
@codspeed

codspeed Bot commented Nov 15, 2026 •

Copy link
Copy Markdown

Merging this PR will not alter performance

1 warning

⚠️ 15 benchmarks are dominated by system calls

System calls cannot be simulated consistently and are left out of the measure. The reported value understates the real cost. Switch to the Walltime instrument to measure them.

Learn more about system calls

See affected benchmarks

✅ 143 untouched benchmarks
⏩ 8 skipped benchmarks1


Comparing claude/awaited-handler-from-background-7839 (231713f) with main (8e5bb84)

Open in CodSpeed

Footnotes

  1. 8 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

1 participant