Skip to content

Commit a18a73b

Browse files
authored
feat: bedrock streaming optimisations (#602)
## Summary Implements proper AWS EventStream handling for Bedrock streaming responses, replacing the manual buffer parsing with the official AWS SDK eventstream decoder. Solves #597 ## Changes - Added `aws/aws-sdk-go-v2/aws/protocol/eventstream` dependency to properly decode AWS EventStream format - Refactored `ChatCompletionStream` method in the Bedrock provider to use the official AWS EventStream decoder - Moved the AWS request signing function to improve code organization - Added proper handling of tool calls in streaming responses - Fixed empty braces handling in tool call arguments to avoid duplication - Improved error handling and response processing for streaming responses ## Type of change - [x] Bug fix - [x] Feature - [x] Refactor - [ ] Documentation - [ ] Chore/CI ## Affected areas - [x] Core (Go) - [ ] Transports (HTTP) - [x] Providers/Integrations - [ ] Plugins - [ ] UI (Next.js) - [ ] Docs ## How to test Test Bedrock streaming with various models, especially those that support tool calls: ```sh # Core/Transports go version go test ./core/providers/bedrock_test.go # Test streaming with Claude models curl -X POST http://localhost:8000/v1/chat/completions \ -H "Content-Type: application/json" \ -d '{ "model": "anthropic.claude-3-sonnet-20240229-v1:0", "messages": [{"role": "user", "content": "Tell me a short story"}], "stream": true }' # Test streaming with tool calls curl -X POST http://localhost:8000/v1/chat/completions \ -H "Content-Type: application/json" \ -d '{ "model": "anthropic.claude-3-sonnet-20240229-v1:0", "messages": [{"role": "user", "content": "What is the weather in New York?"}], "stream": true, "tools": [{"type": "function", "function": {"name": "get_weather", "description": "Get the weather in a location", "parameters": {"type": "object", "properties": {"location": {"type": "string"}}, "required": ["location"]}}}] }' ``` ## Breaking changes - [x] No - [ ] Yes ## Related issues Improves Bedrock streaming reliability and performance, especially for tool calls. ## Security considerations No security implications as this change only affects how we parse AWS EventStream responses. ## Checklist - [x] I added/updated tests where appropriate - [x] I verified builds succeed (Go and UI) - [x] I verified the CI pipeline passes locally if applicable
2 parents 298c854 + 056a8cb commit a18a73b

File tree

8 files changed

+422
-507
lines changed

8 files changed

+422
-507
lines changed

core/go.mod

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -6,6 +6,7 @@ toolchain go1.24.3
66

77
require (
88
github.com/aws/aws-sdk-go-v2 v1.38.0
9+
github.com/aws/aws-sdk-go-v2/aws/protocol/eventstream v1.6.10
910
github.com/aws/aws-sdk-go-v2/config v1.31.0
1011
github.com/bytedance/sonic v1.14.0
1112
github.com/google/uuid v1.6.0

core/go.sum

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -4,6 +4,8 @@ github.com/andybalholm/brotli v1.2.0 h1:ukwgCxwYrmACq68yiUqwIWnGY0cTPox/M94sVwTo
44
github.com/andybalholm/brotli v1.2.0/go.mod h1:rzTDkvFWvIrjDXZHkuS16NPggd91W3kUSvPlQ1pLaKY=
55
github.com/aws/aws-sdk-go-v2 v1.38.0 h1:UCRQ5mlqcFk9HJDIqENSLR3wiG1VTWlyUfLDEvY7RxU=
66
github.com/aws/aws-sdk-go-v2 v1.38.0/go.mod h1:9Q0OoGQoboYIAJyslFyF1f5K1Ryddop8gqMhWx/n4Wg=
7+
github.com/aws/aws-sdk-go-v2/aws/protocol/eventstream v1.6.10 h1:zAybnyUQXIZ5mok5Jqwlf58/TFE7uvd3IAsa1aF9cXs=
8+
github.com/aws/aws-sdk-go-v2/aws/protocol/eventstream v1.6.10/go.mod h1:qqvMj6gHLR/EXWZw4ZbqlPbQUyenf4h82UQUlKc+l14=
79
github.com/aws/aws-sdk-go-v2/config v1.31.0 h1:9yH0xiY5fUnVNLRWO0AtayqwU1ndriZdN78LlhruJR4=
810
github.com/aws/aws-sdk-go-v2/config v1.31.0/go.mod h1:VeV3K72nXnhbe4EuxxhzsDc/ByrCSlZwUnWH52Nde/I=
911
github.com/aws/aws-sdk-go-v2/credentials v1.18.4 h1:IPd0Algf1b+Qy9BcDp0sCUcIWdCQPSzDoMK3a8pcbUM=

0 commit comments

Comments
 (0)