Skip to content

MySQL -> ES, 自定义的 ES output,执行时 Execute 函数接受到 Msgs 数量为1 #254

@Stanley3

Description

@Stanley3

MySQL Input 的配置

[input]
    type = "mysql"
    mode = "batch"

[input.config]
    nr-scanner = 10
    table-scan-batch = 10000
    batch-per-second-limit = 10
    max-full-dump-count = 10000

ES Output 的配置

[output]
    type="custom-elasticsearch"

[output.config]
    ignore-bad-request = true

Scheduler 的配置

[scheduler]
    type = "batch-table-scheduler"

[scheduler.config]
    nr-worker = 10
    batch-size = 1000
    queue-size = 1024
    sliding-window-size = 10240
    nr-retries = 3
    retry-sleep = "2s"

按照这个配置,在执行到 func (output *customESOutput) Execute(msgs []*core.Msg) error 函数时,msgs 数量总是为1,导致 ES 的 bulk request 形同虚设。
请问,应该如何调整配置,解决这个问题。

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions