-
Notifications
You must be signed in to change notification settings - Fork 1.7k
Description
Component(s)
receiver/otlp
receiver/kafka
What happened?
We are using exporter helper configuration retry_on_failure and sending_queue to avoid request failure at open telemetry when backend is unavailable. We tested block_on_overflow
in sending_queue configuration which is seen to be useful to blocks the request until the queue has space. It works good with otlp
receiver but with kafka
receiver it pulls the requests though queue is full. currently, we are making use of combination of Kafka and otlp as recivers. we are utilizing kafka as failover. It is explained with below figures.
fig 1 shows, we receiving on kafka failover receiver.

fig. 2 shows, exporter pulls the request though queue is full as shown in fig. 3.


As we have tested with otlp, block_on_overflow
blocking request if queue is full. but with kafka receiver it pulls the requests though queue is full and hence seen failure.
Expected behavior:
with , block_on_overflow
it should not pull requests if queue is full.
Collector version
0.135.0
Environment information
Collector version: 0.135.0
OpenTelemetry Collector configuration
mode: "deployment"
namespaceOverride: "traces"
config:
connectors:
forward/traces:
exporters:
debug:
verbosity: basic
loadbalancing/traces:
routing_key: "**********"
protocol:
otlp:
# all options from the OTLP exporter are supported
# except the endpoint
timeout: 30s
tls:
insecure: true
resolver:
k8s:
service: "********************.********************"
ports:
- ********
extensions:
health_check:
endpoint: ************
processors:
batch: {}
receivers:
kafka/traces-failover:
auth:
sasl:
mechanism: ***********
password: **************
username: ************
tls:
insecure: false
insecure_skip_verify: true
brokers:
- **************.*********:9093
encoding: otlp_proto
group_id: traces-group
initial_offset: earliest
message_marking:
after: true
metadata:
full: false
protocol_version: 3.7.0
topic: traces
otlp/traces:
protocols:
http:
cors:
allowed_origins:
endpoint: *********
include_metadata: true
max_request_body_size: **********
grpc:
endpoint: *************
include_metadata: true
service:
extensions:
- health_check
pipelines:
traces:
exporters:
- forward/traces
receivers:
- kafka/traces-failover
- otlp/traces
traces/traces:
exporters:
- loadbalancing/traces
processors:
- batch
receivers:
- forward/traces
Log output
Additional context
No response
Tip
React with 👍 to help prioritize this issue. Please use comments to provide useful context, avoiding +1
or me too
, to help us triage it. Learn more here.