-
Notifications
You must be signed in to change notification settings - Fork 867
feat(shard distributor): use smoothed load for initial ephemeral shard placement #7457
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Open
AndreasHolt
wants to merge
67
commits into
cadence-workflow:master
Choose a base branch
from
AndreasHolt:initial-placement-with-statistics
base: master
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
Open
feat(shard distributor): use smoothed load for initial ephemeral shard placement #7457
AndreasHolt
wants to merge
67
commits into
cadence-workflow:master
from
AndreasHolt:initial-placement-with-statistics
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Signed-off-by: Andreas Holt <6665487+AndreasHolt@users.noreply.github.com>
Signed-off-by: Andreas Holt <6665487+AndreasHolt@users.noreply.github.com>
… is being reassigned in AssignShard Signed-off-by: Andreas Holt <6665487+AndreasHolt@users.noreply.github.com>
Signed-off-by: Andreas Holt <6665487+AndreasHolt@users.noreply.github.com>
Signed-off-by: Andreas Holt <6665487+AndreasHolt@users.noreply.github.com>
Signed-off-by: Andreas Holt <6665487+AndreasHolt@users.noreply.github.com>
…to not overload etcd's 128 max ops per txn Signed-off-by: Andreas Holt <6665487+AndreasHolt@users.noreply.github.com>
…s txn and retry monotonically Signed-off-by: Andreas Holt <6665487+AndreasHolt@users.noreply.github.com>
…ents Signed-off-by: Andreas Holt <6665487+AndreasHolt@users.noreply.github.com>
…shard metrics, move out to staging to separate function Signed-off-by: Andreas Holt <6665487+AndreasHolt@users.noreply.github.com>
Signed-off-by: Andreas Holt <6665487+AndreasHolt@users.noreply.github.com>
Signed-off-by: Andreas Holt <6665487+AndreasHolt@users.noreply.github.com>
… And more idiomatic naming of collection vs singular type Signed-off-by: Andreas Holt <6665487+AndreasHolt@users.noreply.github.com>
…ook more like executor key tests Signed-off-by: Andreas Holt <6665487+AndreasHolt@users.noreply.github.com>
…ey in BuildShardKey, as we don't use it Signed-off-by: Andreas Holt <6665487+AndreasHolt@users.noreply.github.com>
…o "statistics" Signed-off-by: Andreas Holt <6665487+AndreasHolt@users.noreply.github.com>
…ollow conventions Signed-off-by: Andreas Holt <6665487+AndreasHolt@users.noreply.github.com>
…o ewma) Signed-off-by: Andreas Holt <6665487+AndreasHolt@users.noreply.github.com>
…t heartbeat Signed-off-by: Andreas Holt <6665487+AndreasHolt@users.noreply.github.com>
…rdStatistics Signed-off-by: Andreas Holt <6665487+AndreasHolt@users.noreply.github.com>
Signed-off-by: Andreas Holt <6665487+AndreasHolt@users.noreply.github.com>
…unt, for logging Signed-off-by: Andreas Holt <6665487+AndreasHolt@users.noreply.github.com>
… for ephemeral shards Signed-off-by: Andreas Holt <6665487+AndreasHolt@users.noreply.github.com>
…xecutor properly Signed-off-by: Andreas Holt <6665487+AndreasHolt@users.noreply.github.com>
…e with new load based selection Signed-off-by: Andreas Holt <6665487+AndreasHolt@users.noreply.github.com>
Signed-off-by: Andreas Holt <6665487+AndreasHolt@users.noreply.github.com>
Signed-off-by: Andreas Holt <6665487+AndreasHolt@users.noreply.github.com>
Signed-off-by: Andreas Holt <6665487+AndreasHolt@users.noreply.github.com>
…eartbeat TTL Signed-off-by: Andreas Holt <6665487+AndreasHolt@users.noreply.github.com>
…o ewma) Signed-off-by: Andreas Holt <6665487+AndreasHolt@users.noreply.github.com>
…eartbeat TTL Signed-off-by: Andreas Holt <6665487+AndreasHolt@users.noreply.github.com> Signed-off-by: Theis Randeris Mathiassen <theis@tmathiassen.dk>
…o ewma) Signed-off-by: Andreas Holt <6665487+AndreasHolt@users.noreply.github.com> Signed-off-by: Theis Randeris Mathiassen <theis@tmathiassen.dk>
…t heartbeat Signed-off-by: Andreas Holt <6665487+AndreasHolt@users.noreply.github.com> Signed-off-by: Theis Randeris Mathiassen <theis@tmathiassen.dk>
…rdStatistics Signed-off-by: Andreas Holt <6665487+AndreasHolt@users.noreply.github.com> Signed-off-by: Theis Randeris Mathiassen <theis@tmathiassen.dk>
Signed-off-by: Andreas Holt <6665487+AndreasHolt@users.noreply.github.com> Signed-off-by: Theis Randeris Mathiassen <theis@tmathiassen.dk>
Signed-off-by: Theis Randeris Mathiassen <theis@tmathiassen.dk>
Signed-off-by: Andreas Holt <6665487+AndreasHolt@users.noreply.github.com> Signed-off-by: Theis Randeris Mathiassen <theis@tmathiassen.dk>
Signed-off-by: Theis Randeris Mathiassen <theis@tmathiassen.dk>
Signed-off-by: Theis Randeris Mathiassen <theis@tmathiassen.dk>
Signed-off-by: Theis Randeris Mathiassen <theis@tmathiassen.dk>
Signed-off-by: Andreas Holt <6665487+AndreasHolt@users.noreply.github.com>
…adencefork into heartbeat-shard-statistics Signed-off-by: Andreas Holt <6665487+AndreasHolt@users.noreply.github.com>
Signed-off-by: Andreas Holt <6665487+AndreasHolt@users.noreply.github.com>
…on from heartbeat TTL Signed-off-by: Andreas Holt <6665487+AndreasHolt@users.noreply.github.com>
…-statistics Signed-off-by: Andreas Holt <6665487+AndreasHolt@users.noreply.github.com>
Signed-off-by: Andreas Holt <6665487+AndreasHolt@users.noreply.github.com>
…dd+modify tests Modified pickLeastLoadedExecutor to skip executrs with non ACTIVE status when selecting an executor for shard assignment. Updated error message to reflect when no active executors are available. Add tests to cover this behavior. Adjust previous tests to also mock executors s.t. we get executors status Signed-off-by: Andreas Holt <6665487+AndreasHolt@users.noreply.github.com>
Signed-off-by: Theis Mathiassen <theis.mathiassen@gmail.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What changed?
assignEphemeralShardto call a newpickLeastLoadedExecutorhelper that sums smoothed shard load per executor (falling back to shard count) and log the selected targetAggregateLoadandAssignedCounttags so handler logs can show load totals when placing ephemeral shardsWhy?
How did you test it?
TestPickLeastLoadedExecutorandShardNotFound_Ephemeral_LoadBasedin handler_test.go to verify logic.Potential risks
If shard stats are missing or stale, the aggregated load will calculate as zero. In this case, the logic degrades to the previous behavior (selecting based on shard count), minimizing the risk of bad placement.
Release notes
Ephemeral shard placement now favors the executor with the lowest smoothed load (with shard-count tie breaker) and logs the inputs for each decision.
Documentation Changes