-
Notifications
You must be signed in to change notification settings - Fork 174
Open
Labels
enhancementNew feature or requestNew feature or request
Description
There are some use cases where parsing binlog is the bottleneck, such as syncing from severals hours/days ago after dumping big uninterested table. Throughput is around 40 to 50 thousands records per second in production for us and exhausted one cpu core. If we could parse binlog in parallel, much higher throughput in this scenario could be reached I think.
To make this possible, we have to break the sequential assumption(within one input stream) from input to sliding window. One possible solution is add a prepare
step before submit
in scheduler. Sequence sensitive logic such as id allocating should be done before prepare
, then start parallel parsing, and finally submit it as before.
Metadata
Metadata
Assignees
Labels
enhancementNew feature or requestNew feature or request