Clarification on relationship between offset.flush.interval.ms and MQ batch commit timing #168
Description
Hello,
We have a simple local Docker Compose setup using the IBM MQ Source Connector together with Kafka Connect to send messages from MQ to a Kafka topic.
In our testing, we noticed that increasing the offset.flush.interval.ms value (the standard Kafka Connect worker property) appears to affect MQ commit behavior.
Specifically, higher values cause the connector to eventually throw: ConnectException: Missing commits for message batch.
Now, our current understanding is that this happens because the connector waits for the Connect worker’s offset flush callback before committing MQ messages, and that callback frequency is controlled by offset.flush.interval.ms.
Could you confirm if we are on the right track on this?
We’d also like to know if there are any trade-offs of reducing offset.flush.interval.ms. From what we found out until now, the main drawback seems to be increased overhead due to more frequent offset writes (slightly higher network and CPU usage).
Is there any additional MQ-specific disadvantage we should be aware of?
We would really appreciate any help.