[PERF] Akka.Cluster.Sharding.Delivery throughput

[Arkatufus]

Version Information
Version of Akka.NET? dev
Which Akka.NET Modules? Akka.Cluster.Sharding.Delivery

Describe the performance issue

Akka.Cluster.Sharding.Delivery message throughput drops when a burst of messages were being sent

Data and Specs

Data taken using the ClusterDeliveryBenchmark benchmark in #7732, message timing logs were taken from the modified ReliableDeliveryShardingSpec unit test in the same PR.

Burst	Op/Sec
200	3146.93
400	2430.84
800	1530.29
1500	828.15
3000	389.93

As can be seen, message throughput drops linearly when more and more messages were being sent in burst.

Expected behavior

Message throughput should stay constant regardless of message traffic

Actual behavior

Message throughput drops

[Arkatufus]

Timing log from ReliableDeliveryShardingSpec:

[user/producer-1] TRACK:[ProAct.Rcv-RequestNext,msg-adapter,46:36.1956]
[user/shardingController-1] TRACK:[ProducerController.OnMsg-Send,Job(msg-1497),entity-0,46:36.1956]
[user/shardingController-1] TRACK:[ProducerController.ReceiveWrappedRequestNext-RequestNext,entity-0,46:36.1957]
[system/sharding/TestConsumer-1/7/entity-0/consumerController-p-1-entity-0] TRACK:[ConsumerController.Deliver-Delivery,,46:36.1957]
[system/sharding/TestConsumer-1/7/entity-0/consumer] processed [500] [msg: msg-1497] from [p-1-entity-0]
[user/shardingController-1] TRACK:[ProducerController.OnAck,entity-0,46:36.1958]

[user/producer-1] TRACK:[ProAct.Rcv-RequestNext,msg-adapter,46:36.1957]
[user/shardingController-1] TRACK:[ProducerController.OnMsg-Send,Job(msg-1498),entity-1,46:36.1958]
[system/sharding/TestConsumer-1/6/entity-1/consumerController-p-1-entity-1] TRACK:[ConsumerController.Deliver-Delivery,,46:36.1958]
[user/shardingController-1] TRACK:[ProducerController.ReceiveWrappedRequestNext-RequestNext,entity-1,46:36.1958]
[system/sharding/TestConsumer-1/6/entity-1/consumer] processed [500] [msg: msg-1498] from [p-1-entity-1]
[user/shardingController-1] TRACK:[ProducerController.OnAck,entity-1,46:36.1959]

[user/producer-1] TRACK:[ProAct.Rcv-RequestNext,msg-adapter,46:36.1958]
[user/shardingController-1] TRACK:[ProducerController.OnMsg-Send,Job(msg-1499),entity-2,46:36.1959]
[user/shardingController-1] TRACK:[ProducerController.ReceiveWrappedRequestNext-RequestNext,entity-2,46:36.1959]
[system/sharding/TestConsumer-1/4/entity-2/consumerController-p-1-entity-2] TRACK:[ConsumerController.Deliver-Delivery,,46:36.1959]
[system/sharding/TestConsumer-1/4/entity-2/consumer] processed [500] [msg: msg-1499] from [p-1-entity-2]
[user/shardingController-1] TRACK:[ProducerController.OnAck,entity-2,46:36.1960]

The timing between RequestNext between 2 input messages are between 0.1 to 0.2 ms, which means a theoretical message throughput between 10,000 to 5,000 messages.

[Arkatufus]

The first thought that came to mind is back-pressure due to buffer limitation. If
There are 3 place where things are buffered in the system, all 3 can be controlled via HOCON settings.

• ShardingProducerController
• ShardingConsumerController
• ConsumerController stash size

Benchmark results

Increased sharding consumer and producer buffer size from 1000 to 10000

Burst	Op/Sec
200	3190.19
400	2568.07
800	1534.68

Increased consumer stash size from 50 to 1000

Burst	Op/Sec
200	3220.52
400	2463.29
800	1538.27

It is obvious that the buffers were not the limiting factor as the benchmark numbers are exactly the same as the control numbers

[Arkatufus]

As suggested in #7729 (comment) this could be caused by actor state switching. Benchmarking this reveals that this is not true, there are no difference in switching and non switching.

Burst	UseSingleState	Mean	Error	StdDev	Median	Req/sec
200	False	203.2 us	14.43 us	75.23 us	160.5 us	4,921.44
200	True	198.4 us	14.19 us	73.96 us	153.9 us	5,041.51
400	False	309.4 us	12.35 us	64.39 us	293.0 us	3,232.36
400	True	302.8 us	12.63 us	65.85 us	281.5 us	3,302.86
800	False	576.0 us	5.08 us	26.50 us	570.2 us	1,736.05
800	True	585.8 us	2.41 us	12.56 us	583.4 us	1,707.15