-
Notifications
You must be signed in to change notification settings - Fork 5.2k
http: add stream_to_metadata filter for stream parsing #42762
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: main
Are you sure you want to change the base?
Conversation
Signed-off-by: Peter Leng <yleng@pinterest.com>
|
CC @envoyproxy/api-shepherds: Your approval is needed for changes made to |
Signed-off-by: Peter Leng <yleng@pinterest.com>
Signed-off-by: Peter Leng <yleng@pinterest.com>
Signed-off-by: Peter Leng <yleng@pinterest.com>
Signed-off-by: Peter Leng <yleng@pinterest.com>
Signed-off-by: Peter Leng <yleng@pinterest.com>
|
Tagging @JuniorHsu and @KBaichoo for review 🙏 |
happy to help review and be one of the owner :) |
Signed-off-by: Peter Leng <yleng@pinterest.com>
Signed-off-by: Peter Leng <yleng@pinterest.com>
Signed-off-by: Peter Leng <yleng@pinterest.com>
Signed-off-by: Peter Leng <yleng@pinterest.com>
|
/retest |
Signed-off-by: Peter Leng <yleng@pinterest.com>
Signed-off-by: Peter Leng <yleng@pinterest.com>
Signed-off-by: Peter Leng <yleng@pinterest.com>
Signed-off-by: Peter Leng <yleng@pinterest.com>
|
@KBaichoo added you as maintainer for now but let me know if we should change that 😁 |
|
/assign @KBaichoo |
|
PeterL328 is not allowed to assign users. |
JuniorHsu
left a comment
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
i did a first pass but like we want api review at the moment as the patch is somewhat mature
docs/root/configuration/http/http_filters/stream_to_metadata_filter.rst
Outdated
Show resolved
Hide resolved
docs/root/configuration/http/http_filters/stream_to_metadata_filter.rst
Outdated
Show resolved
Hide resolved
docs/root/configuration/http/http_filters/stream_to_metadata_filter.rst
Outdated
Show resolved
Hide resolved
docs/root/configuration/http/http_filters/stream_to_metadata_filter.rst
Outdated
Show resolved
Hide resolved
docs/root/configuration/http/http_filters/stream_to_metadata_filter.rst
Outdated
Show resolved
Hide resolved
api/envoy/extensions/filters/http/stream_to_metadata/v3/stream_to_metadata.proto
Outdated
Show resolved
Hide resolved
Signed-off-by: Peter Leng <yleng@pinterest.com>
Signed-off-by: Peter Leng <yleng@pinterest.com>
Commit Message: Add stream_to_metadata HTTP filter for stream parsing
Additional Description:
This PR introduces a new HTTP filter for extracting values from response flow streams and writes them into dynamic metadata. We start with supporting SSE (server-sent events) and JSON extraction from data events but keep interface generic so we can support other types of streams and formats later.
Primary use-case for this filter will be to extract token usage information from LLM responses and use in places like rate-limit or obersevability.
At Pinterest, we have an internal version of this filter that is more specific but have large part of the logic in this filter has been battle tested in our egress proxy.
Per-route configuration override and request stream parsing can be added in later PR.
Key Features
max_event_size(default: 8KB, max: 10MB) to prevent unbounded buffer growth.stop_processing_on_matchallows for picking first occurrence or last occurance.Risk Level: Low (new filter)
Testing: Unit/Integration tests
Docs Changes: Added doc for stream_to_metadata
Release Notes: Added stream_to_metadata HTTP filter for extracting values from Server-Sent Events (SSE) streams and writing them to dynamic metadata. Useful for token-based rate limiting of LLM APIs.
Platform Specific Features: N/A
Fixes #29758