Add BN metric alert configuration suggestions

[Nana-EC]

Reviewer Notes

Provide metric alerting value suggestions to allow operators to get going quickly on observing their BN

Related Issue(s)

Fixes #1615

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.

@@             Coverage Diff              @@
##               main    #1851      +/-   ##
============================================
+ Coverage     80.57%   80.58%   +0.01%     
+ Complexity     1178     1177       -1     
============================================
  Files           127      127              
  Lines          5550     5553       +3     
  Branches        591      591              
============================================
+ Hits           4472     4475       +3     
- Misses          805      806       +1     
+ Partials        273      272       -1     

see 3 files with indirect coverage changes

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[mustafauzunn]

conventional-pr-title check is not passing.
Beside that looking good

[jsync-swirlds]

Just some optional suggestions for alert conditions.

[jsync-swirlds]

These are a bit too tight; we'd actually prefer to see utilization higher than 60% under load; over 80% might be worth alerting, depending on how large the queues are.

[AlfredoG87]

I think that 60% utilization makes sense here, specially since is a L severity.

[jsync-swirlds]

I'm more concerned that under moderate load we're planning to alert when the node is at its most healthy. That seems backwards.
Utilization between 40% and 80% should be "green" and only alert outside that range (on either end).

[jsync-swirlds]

This is repeated below with a different value.

[jsync-swirlds]

These are almost certainly far too long.
In order, "healthy" numbers would be (<=) .5, .02, .5, and .05 (in seconds)

[AlfredoG87]

I agree that these are too large values:
I would say that these numbers would vary on the network, mainnet/testnet vs sphere, however if we want to give recommendations for the current public networks we should state it at the top.

and then my recommendation is as follows:

publisher_receive_latency_ns --> 3 or 2.5 secs (here we are expecting at least 2 secs since that is the time the CN takes to wrap-up a block, and then some more due to transmission and block_proof generation.

hashing_block_time --> 250 ms

verification_block_time --> same as above 250 ms.

files_recent_persistence_time_latency_ns --> 100 ms.

[AlfredoG87]

LGTM. just a couple of nit suggestions.

[AlfredoG87]

I agree that these are too large values:
I would say that these numbers would vary on the network, mainnet/testnet vs sphere, however if we want to give recommendations for the current public networks we should state it at the top.

and then my recommendation is as follows:

publisher_receive_latency_ns --> 3 or 2.5 secs (here we are expecting at least 2 secs since that is the time the CN takes to wrap-up a block, and then some more due to transmission and block_proof generation.

hashing_block_time --> 250 ms

verification_block_time --> same as above 250 ms.

files_recent_persistence_time_latency_ns --> 100 ms.

[AlfredoG87]

I think that 60% utilization makes sense here, specially since is a L severity.

[AlfredoG87]

I think 40 is too much. I would recommend 20 or even 10. are we expecting to have that many publishers connected to a single node in Mainnet or even spheres?

[jsync-swirlds]

In Hedera mainnet we may have all 30-ish nodes connected at once; that is definitely not ideal, but with only 3 LFH nodes (out of 5 total), it could reasonably happen.