Skip to content

Restore token streaming for plain-text REPL answers #78

Description

@GeigerJ2

The aiida-agents REPL used to stream the agent's reply token by token; now it blocks and dumps the whole answer at once after the spinner. On a local Ollama model that's a few seconds of spinner followed by a wall of text, which reads worse than watching it type out.

It regressed in #8 (commit 7b4d333), the human-gated write path, where ask switched from agent.run_stream(...) + stream_text to a blocking await agent.run(question). That was deliberate rather than careless: #8 added output_type=(str, DeferredToolRequests) and _handle_deferred, which need the full AgentRunResult to inspect result.output, and run_stream is geared to streaming text, so blocking was the quick way to get the result object back.

I think we can have both: stream while the model is producing a plain-text answer, and fall back to the blocking path only for the DeferredToolRequests (approval) case, which can't stream anyway. Worth checking first how pydantic-ai's streaming composes with a union output_type that includes deferred tool requests before committing to an approach.

This is independent of the prompt_toolkit REPL rewrite in #7: streaming is output-side (consuming run_stream instead of blocking on agent.run), the input library is input-side, so switching to prompt_toolkit neither fixes nor removes this. The ask change stands either way.

For reference, the streaming version #8 removed:

old ask
async with agent.run_stream(question) as result:
    print("Agent: ", end="", flush=True)
    printed_len = 0
    async for chunk in result.stream_text(debounce_by=None):
        print(chunk[printed_len:], end="", flush=True)
        printed_len = len(chunk)
    print()

Noticed while reviewing #19, which only added history threading to ask, not the streaming change. Ties into the CLI work in #7.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Fields

    Priority

    None yet

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions