Skip to content

Implement average lifetime long batch's size and document count metric #17892

@andsel

Description

@andsel

Create the metrics partial subtree related to batch.event_count.average.lifetime and batch.byte_size.average.lifetime.
This initial implementation must also expose a setting named pipeline.batch.metrics.sampling_mode which defines 3 behaviours:

  • disabled: no collection of any batch data, so the response of HTTP API shouldn't change respect to actual shape.
  • minimal: collect the metrics for a subset of all the batches (1% of the total).
  • full: collect the metrics data from each batch.

Initially this setting is none by default and is defined per pipeline.

Consider the work study PR #17758 for the core part of memory occupation estimation.

  • expose the event_count for memory and persistent
  • expose the byte_size for memory and persistent
  • implement the flag

Metadata

Metadata

Assignees

Labels

No labels
No labels

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions