Skip to content

[expoterhelper] block_on_overflow not works with kafkareceiver #13901

@VijayPatil872

Description

@VijayPatil872

Component(s)

receiver/otlp
receiver/kafka

What happened?

We are using exporter helper configuration retry_on_failure and sending_queue to avoid request failure at open telemetry when backend is unavailable. We tested block_on_overflow in sending_queue configuration which is seen to be useful to blocks the request until the queue has space. It works good with otlp receiver but with kafka receiver it pulls the requests though queue is full. currently, we are making use of combination of Kafka and otlp as recivers. we are utilizing kafka as failover. It is explained with below figures.
fig 1 shows, we receiving on kafka failover receiver.

Image

fig. 2 shows, exporter pulls the request though queue is full as shown in fig. 3.

Image Image

As we have tested with otlp, block_on_overflow blocking request if queue is full. but with kafka receiver it pulls the requests though queue is full and hence seen failure.

Expected behavior:
with , block_on_overflow it should not pull requests if queue is full.

Collector version

0.135.0

Environment information

Collector version: 0.135.0

OpenTelemetry Collector configuration

mode: "deployment"
namespaceOverride: "traces"
config:
  connectors:
    forward/traces:

  exporters:
    debug:
      verbosity: basic

    loadbalancing/traces:
      routing_key: "**********"
      protocol:
        otlp:
          # all options from the OTLP exporter are supported
          # except the endpoint
          timeout: 30s
          tls:
            insecure: true
      resolver:
        k8s:
          service: "********************.********************"
          ports:
          - ********
     
  extensions:
    health_check: 
      endpoint: ************

  processors:
    batch: {}
    
  receivers:
    kafka/traces-failover:
      auth:
        sasl:
          mechanism: ***********
          password: **************
          username: ************
        tls:
          insecure: false
          insecure_skip_verify: true
      brokers:
      - **************.*********:9093
      encoding: otlp_proto
      group_id: traces-group
      initial_offset: earliest
      message_marking:
        after: true
      metadata:
        full: false
      protocol_version: 3.7.0
      topic: traces

    otlp/traces:
      protocols:
        http:
          cors:
            allowed_origins:
          endpoint: *********
          include_metadata: true
          max_request_body_size: **********
        grpc:
          endpoint: *************
          include_metadata: true      
  service:
    extensions:
      - health_check
    pipelines:
      traces:
        exporters:
          - forward/traces
        receivers:
          - kafka/traces-failover
          - otlp/traces
      traces/traces:
        exporters:
          - loadbalancing/traces
        processors:
          - batch
        receivers:
          - forward/traces

Log output

Additional context

No response

Tip

React with 👍 to help prioritize this issue. Please use comments to provide useful context, avoiding +1 or me too, to help us triage it. Learn more here.

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions