Automatic statistical calculation precision degradation based on used memory compared to ceiling #3
Description
The variable structures required to capture the data for statistical calculation could lead to excessive memory usage as each of the durations are stored in order to calculate the percentile values. With a very large amount of requests, across a very large time range, this could lead to significant size.
I was initially leaning towards percentile approximation using a set of duration buckets similar to how Prometheus calculates percentiles as this is far more memory efficient and adapted to the use case of running this tool in a production environment. However, then this takes away from the tools ability to be precise where other tools cannot (ie; Prometheus) as we are operating with the exact data and if run on a system with lots of memory this memory usage is tiny in comparison for after the fact analysis.
I've decided to take the middle way, and default to high precision, yet monitor memory usage throughout, and then when a memory threshold is hit (say 95% of ceiling --max-memory), then set a flag to use approximated statistics, and go through the existing data structures and delete the duration arrays. Considering that max memory is detected, and can be manually set, this should still provide high precision statistics for smaller files on a production system, and use bucket-based approximations for very large files or constrained pods.