Skip to content

Conversation

skamenan7
Copy link
Contributor

Bedrock provider was returning None for both streaming and non-streaming inference, causing 'NoneType' object has no attribute 'choices' errors.

Primary fix: Reorder inheritance to put mixin classes before protocol class in BedrockInferenceAdapter so actual implementations are called.

Additional AWS Bedrock API compatibility fixes:

  • Fix non-streaming: use res["body"].read() instead of next(res["body"])
  • Fix streaming: add proper event structure checks and safe access
  • Disable repetition_penalty (not supported by Bedrock Llama models)

Fixes #3621

Test Plan

Non-Streaming Test

curl -X POST http://localhost:8321/v1/openai/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"model": "bedrock-inference/meta.llama3-1-8b-instruct-v1:0", "messages": [{"role": "user", "content": "Hello"}], "stream": false}'

Non-Streaming Success Response

{
  "id": "chatcmpl-4a6c063b-53d2-4286-b81d-34ed9c5671e1",
  "choices": [{
    "message": {
      "role": "assistant",
      "content": "Hello. What would you like to talk about?",
      "name": null,
      "tool_calls": null
    },
    "finish_reason": "stop",
    "index": 0,
    "logprobs": null
  }],
  "object": "chat.completion",
  "created": 1759264305,
  "model": "bedrock-inference/meta.llama3-1-8b-instruct-v1:0"
}

Streaming Test

curl -X POST http://localhost:8321/v1/openai/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"model": "bedrock-inference/meta.llama3-1-8b-instruct-v1:0", "messages": [{"role": "user", "content": "Hello"}], "stream": true}'

Streaming Success Response

data: {"id":"chatcmpl-72161218-0f23-4512-a8bd-042ce5a3dcad","choices":[{"delta":{"content":"","function_call":null,"refusal":null,"role":null,"tool_calls":null},"finish_reason":"stop","index":0,"logprobs":null}],"created":1759264322,"model":"bedrock-inference/meta.llama3-1-8b-instruct-v1:0","object":"chat.completion.chunk","service_tier":null,"system_fingerprint":null,"usage":null}

data: {"id":"chatcmpl-72161218-0f23-4512-a8bd-042ce5a3dcad","choices":[{"delta":{"content":"Hello","function_call":null,"refusal":null,"role":null,"tool_calls":null},"finish_reason":"stop","index":0,"logprobs":null}],"created":1759264322,"model":"bedrock-inference/meta.llama3-1-8b-instruct-v1:0","object":"chat.completion.chunk","service_tier":null,"system_fingerprint":null,"usage":null}

data: {"id":"chatcmpl-72161218-0f23-4512-a8bd-042ce5a3dcad","choices":[{"delta":{"content":". How","function_call":null,"refusal":null,"role":null,"tool_calls":null},"finish_reason":"stop","index":0,"logprobs":null}],"created":1759264322,"model":"bedrock-inference/meta.llama3-1-8b-instruct-v1:0","object":"chat.completion.chunk","service_tier":null,"system_fingerprint":null,"usage":null}

Server Log Snippets - Success

INFO: Found credentials in shared credentials file: ~/.aws/credentials
INFO: BedrockInferenceAdapter.chat_completion [StatusCode.OK] (561.00ms)
INFO: output: {'metrics': None, 'completion_message': {'role': 'assistant', 'content': 'Hello, how can I assist you today?', 'stop_reason': 'end_of_turn', 'tool_calls': []}, 'logprobs': None}
INFO: InferenceRouter.openai_chat_completion [StatusCode.OK] (716.71ms)

Bedrock provider was returning None for both streaming and non-streaming
inference, causing 'NoneType' object has no attribute 'choices' errors.

Primary fix: Reorder inheritance to put mixin classes before protocol class
in BedrockInferenceAdapter so actual implementations are called.

Additional AWS Bedrock API compatibility fixes:
- Fix non-streaming: use res["body"].read() instead of next(res["body"])
- Fix streaming: add proper event structure checks and safe access
- Disable repetition_penalty (not supported by Bedrock Llama models)

Fixes llamastack#3621
@meta-cla meta-cla bot added the CLA Signed This label is managed by the Meta Open Source bot. label Oct 1, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

CLA Signed This label is managed by the Meta Open Source bot.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Bedrock Provider Non-Streaming and Streaming Inference Failures

1 participant