HDDS-14213. Fix DiskBalancer VolumeChoosingPolicy to balance when dest volume exists but no src volume exceeds threshold

[Gargi-jais11]

What changes were proposed in this pull request?

This PR handles the below case to be fixed:

one volume under threshold and no volumes beyond threshold

Disk1, 30, 100
Disk2, 30, 100
Disk3, 20, 10080 / 300 = 26.7%

Disk1: 30%
Disk2: 30%
Disk3: 20%
Threshold: 10
Disk utilization range (16.7, 36.7)
Out range volume list: NULL

Threshold: 5
Disk utilization range (21.7, 31.7)
Out range volume list: Disk3 

What is the link to the Apache JIRA

https://issues.apache.org/jira/browse/HDDS-14213

How was this patch tested?

Added a new unit Test TestDefaultVolumeChoosingPolicy which covers below test scenarios:

• One Volume Beyond Threshold, No VolumesUnderThreshold
• Volumes with two Above And one BelowThreshold
• All Volumes Within Threshold
• One Volume Under Threshold No Volumes Above
• Extreme Imbalance
• SmallThresholdModerateImbalance
• Volumes At Threshold Boundaries
• Multiple volumes above threshold, one below
• Best destination has low utilization but insufficient space

[Gargi-jais11]

@ChenSammi and @szetszwo Please review the patch.

[szetszwo]

@Gargi-jais11 , thanks for working on this!

Please see the comments inlined.

[Gargi-jais11]

I have completed a comprehensive performance analysis comparing the Old Policy (O(N)Log(N)) vs. the New Policy (O(N^2)) across different cluster sizes (20, 100, and 200 volumes).
Results:
The change drastically improves balancing effectiveness in constrained environments without introducing any performance regression at scale.

Before optimisation:

Created 20 volumes in 16 ms
Performance results for DefaultVolumeChoosingPolicy
Total volumes: 20
Total threads: 10
Threshold(%): 10.0
Total operations: 100000
Volume Pair Chosen operations: 1252
Volume Pair Not Chosen operations: 98748
Failed operations: 0
Total time (ms): 1897
Average time per operation (ns): 18978.46474
Operations per second: 52691.30109836271

Created 100 volumes in 58 ms
Performance results for DefaultVolumeChoosingPolicy
Total volumes: 100
Total threads: 50
Threshold(%): 10.0
Total operations: 500000
Volume Pair Chosen operations: 25249
Volume Pair Not Chosen operations: 474751
Failed operations: 0
Total time (ms): 167566
Average time per operation (ns): 335133.097966
Operations per second: 2983.889105758967

Created 200 volumes in 75 ms
Performance results for DefaultVolumeChoosingPolicy
Total volumes: 200
Total threads: 50
Threshold(%): 10.0
Total operations: 500000
Volume Pair Chosen operations: 53555
Volume Pair Not Chosen operations: 446445
Failed operations: 0
Total time (ms): 360269
Average time per operation (ns): 720539.560926
Operations per second: 1387.8488485973648

After optimisation:

Created 20 volumes in 16 ms
Performance results for DefaultVolumeChoosingPolicy
Total volumes: 20
Total threads: 10
Threshold(%): 10.0
Total operations: 100000
Volume Pair Chosen operations: 6168
Volume Pair Not Chosen operations: 93832
Failed operations: 0
Total time (ms): 1958
Average time per operation (ns): 19583.56348
Operations per second: 51063.229683467194

Created 100 volumes in 59 ms
Performance results for DefaultVolumeChoosingPolicy
Total volumes: 100
Total threads: 50
Threshold(%): 10.0
Total operations: 500000
Volume Pair Chosen operations: 19065
Volume Pair Not Chosen operations: 480935
Failed operations: 0
Total time (ms): 169903
Average time per operation (ns): 339807.85983
Operations per second: 2942.839522606342

Created 200 volumes in 102 ms
Performance results for DefaultVolumeChoosingPolicy
Total volumes: 200
Total threads: 50
Threshold(%): 10.0
Total operations: 500000
Volume Pair Chosen operations: 49133
Volume Pair Not Chosen operations: 450867
Failed operations: 0
Total time (ms): 319351
Average time per operation (ns): 638703.620642
Operations per second: 1565.6714126574684

[szetszwo]

@Gargi-jais11 , thanks for the update!

+1 the change looks good.

[ChenSammi]

Thanks @Gargi-jais11 , and @szetszwo for review.