Is Retry Possible in SemanticKernelAgent Adapter

[Yuva-raj-18]

Hi Everyone,

I'm using the SemanticKernelAgentAdapter for creating new agents. Occasionally, the agent returns an empty response at random, even though we're maintaining the conversation history correctly. I have two questions:

FYI - I am using dotnet version of autogen.

What could be causing the agent to return an empty response, and how can I fix it?

Is it possible to implement a retry mechanism only when the agent returns an empty response?

[Alex-Wenner-FHR (Alex-Wenner-FHR)]

I think this is actually similar to the discussion I brought up.

#6176

I see the same problem with the SemanticKernelAgentAdapter with AWS Bedrock.

[Yuva-raj-18]

Eric Zhu (@ekzhu) I'm using the Semantic Kernel Agent Adapter with Claude 3.7 Sonnet and occasionally getting empty responses at random. I'm working with the .NET version of AutoGen. Any ideas on how to fix this?"

[Lucas Kim (kimnamu)]

Thanks Yuva-raj-18 for the screenshot and Alex-Wenner-FHR (@Alex-Wenner-FHR) for linking #6176; that is the Python side of the same cause, so here is the .NET side, verified with AutoGen.SemanticKernel 0.2.3 + Microsoft.SemanticKernel.Connectors.Amazon 1.80.1-alpha against Bedrock Claude.

Cause. Bedrock Converse can return an assistant turn with content: [], and nothing on the way to your agent checks for it: the SK connector makes a ChatMessageContent with zero Items, SemanticKernelAgent forwards it, and the message connector maps zero items to a MultiModalMessage with no content (the blank line under from: SupportAgent). One trigger that is easy to hit in a group chat: if the history handed to an agent ends with its own message, it is sent as an assistant turn, Converse treats it as a prefill, and the model has nothing to add.

Retry. Yes, with RegisterMiddleware after RegisterMessageConnector(), re-asking only while the reply is empty:

static bool IsEmpty(IMessage m) => m switch
{
    MultiModalMessage mm => !mm.Content.Any(),   // what the connector emits for a Converse turn with content: []
    _ => string.IsNullOrWhiteSpace(m.GetContent()),
};

var agent = new SemanticKernelAgent(kernel, "SupportAgent",
        settings: new AmazonClaudeExecutionSettings { MaxTokensToSample = 1024 }) // default settings send maxTokens: 0, which Converse rejects
    .RegisterMessageConnector()
    .RegisterMiddleware(async (msgs, opt, next, ct) =>
    {
        var reply = await next.GenerateReplyAsync(msgs, opt, ct);
        for (var attempt = 1; attempt <= 2 && IsEmpty(reply); attempt++)
            reply = await next.GenerateReplyAsync(msgs, opt, ct);
        return reply;
    });

scenario	no retry	with retry
Bedrock Claude haiku-4-5, history ends with the agent's own message	MultiModalMessage, 0 items	still empty after 2 retries (same history, same prefill)
stub IChatCompletionService: empty once, then text	MultiModalMessage, 0 items	TextMessage "hello (call 2)"

live/no-retry: type=MultiModalMessage from=SupportAgent content=null items=0
  [retry 1] empty reply from SupportAgent, calling again
  [retry 2] empty reply from SupportAgent, calling again
live/retry   : type=MultiModalMessage from=SupportAgent content=null items=0
stub/no-retry: type=MultiModalMessage from=SupportAgent content=null items=0
  [retry 1] empty reply from SupportAgent, calling again
stub/retry   : type=TextMessage from=SupportAgent content="hello (call 2)"

So the retry covers the random case; for a trailing self-message it re-sends the same prefill, and dropping or folding that message before the call is what fixes that one.

Verification details

Converse returning zero content blocks (aws cli, us-east-1, us.anthropic.claude-haiku-4-5-20251001-v1:0), and the code path (autogen main 027ecf0a, semantic-kernel main):

# stopSequences hit on the first token
{"stopReason": "stop_sequence", "content": [], "usage": {"inputTokens": 14, "outputTokens": 1, "totalTokens": 15}}
# history ends with a complete assistant message (prefill)
{"stopReason": "end_turn", "content": [], "usage": {"inputTokens": 15, "outputTokens": 3, "totalTokens": 18}}

semantic-kernel  dotnet/src/Connectors/Connectors.Amazon/Bedrock/Core/Clients/BedrockChatCompletionClient.cs
                 ConvertToMessageContent / CreateChatMessageContentItemCollection: one TextContent per block, [] -> zero Items
autogen          dotnet/src/AutoGen.SemanticKernel/SemanticKernelAgent.cs:68-75
                 wraps reply[0] in a MessageEnvelope without inspecting it
autogen          dotnet/src/AutoGen.SemanticKernel/Middleware/SemanticKernelChatMessageContentConnector.cs:80-99
                 items.Count() == 1 -> TextMessage, anything else (including 0) -> MultiModalMessage
                 :165 the agent's own messages are sent as AuthorRole.Assistant

MultiModalMessage does not implement ICanGetTextContent, so GetContent() is null for it; IsEmpty checks the item count instead. Repro: net8.0 console app, kernel from AddBedrockChatCompletionService(modelId), MaxTokensToSample = 64; the stub returns an empty ChatMessageContentItemCollection on its first call, text afterwards. Claude 3.7 Sonnet is no longer served on the account I have, so the live run uses Haiku 4.5; the code path is model-independent.