Commit a18a73b
authored
feat: bedrock streaming optimisations (#602)
## Summary
Implements proper AWS EventStream handling for Bedrock streaming responses, replacing the manual buffer parsing with the official AWS SDK eventstream decoder.
Solves #597
## Changes
- Added `aws/aws-sdk-go-v2/aws/protocol/eventstream` dependency to properly decode AWS EventStream format
- Refactored `ChatCompletionStream` method in the Bedrock provider to use the official AWS EventStream decoder
- Moved the AWS request signing function to improve code organization
- Added proper handling of tool calls in streaming responses
- Fixed empty braces handling in tool call arguments to avoid duplication
- Improved error handling and response processing for streaming responses
## Type of change
- [x] Bug fix
- [x] Feature
- [x] Refactor
- [ ] Documentation
- [ ] Chore/CI
## Affected areas
- [x] Core (Go)
- [ ] Transports (HTTP)
- [x] Providers/Integrations
- [ ] Plugins
- [ ] UI (Next.js)
- [ ] Docs
## How to test
Test Bedrock streaming with various models, especially those that support tool calls:
```sh
# Core/Transports
go version
go test ./core/providers/bedrock_test.go
# Test streaming with Claude models
curl -X POST http://localhost:8000/v1/chat/completions \
-H "Content-Type: application/json" \
-d '{
"model": "anthropic.claude-3-sonnet-20240229-v1:0",
"messages": [{"role": "user", "content": "Tell me a short story"}],
"stream": true
}'
# Test streaming with tool calls
curl -X POST http://localhost:8000/v1/chat/completions \
-H "Content-Type: application/json" \
-d '{
"model": "anthropic.claude-3-sonnet-20240229-v1:0",
"messages": [{"role": "user", "content": "What is the weather in New York?"}],
"stream": true,
"tools": [{"type": "function", "function": {"name": "get_weather", "description": "Get the weather in a location", "parameters": {"type": "object", "properties": {"location": {"type": "string"}}, "required": ["location"]}}}]
}'
```
## Breaking changes
- [x] No
- [ ] Yes
## Related issues
Improves Bedrock streaming reliability and performance, especially for tool calls.
## Security considerations
No security implications as this change only affects how we parse AWS EventStream responses.
## Checklist
- [x] I added/updated tests where appropriate
- [x] I verified builds succeed (Go and UI)
- [x] I verified the CI pipeline passes locally if applicableFile tree
8 files changed
+422
-507
lines changed- core
- providers
- schemas
- providers/bedrock
- framework/streaming
8 files changed
+422
-507
lines changed| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
6 | 6 | | |
7 | 7 | | |
8 | 8 | | |
| 9 | + | |
9 | 10 | | |
10 | 11 | | |
11 | 12 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
4 | 4 | | |
5 | 5 | | |
6 | 6 | | |
| 7 | + | |
| 8 | + | |
7 | 9 | | |
8 | 10 | | |
9 | 11 | | |
| |||
0 commit comments