Optimize commit_index calculation #225
akiradeveloper
started this conversation in
Ideas
Replies: 0 comments
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
-
Calculating commit_index is done by leader and later propagated to followers. Simply saying, the algorithm is list the replicated index of all the servers and find the center value.
Let n be the number of servers in the cluster. Naively, the computation is the order of O(n logn) and it is trivial when n is small. It can be O(n) optimally but it will be practically slower because the coefficients are larger as the code is complicated.
In the worst case, this calculation is done n time to increment one commit_index. So in total we consume O(n^2 logn) calculation per one log entry. If n is big (say 1000) this will be non-trivial. (I don't know someone to manage n=1000 Raft cluster)
If we have bitmask where bit 1 is "replication index is updated since last commit_index update", n is replaced with
popcnt
which is interpreted as CPU instruction because if number of 1s in the bitmask is smaller than the majority number, the computation can be skipped.Disadvantage: N will be limited to at most 64 if we are adhere to call popcnt only once. This is practically okay I believe.
Beta Was this translation helpful? Give feedback.
All reactions