[FR] Report cross-thread statistics for multithreaded runs

[shuralnik-sc]

Is your feature request related to a problem? Please describe.
When running multithreaded benchmarks - in certain cases there is a need to understand a "fairness" of the measured code, i.e. if work is equally distributed among all threads and/or if all threads were able to acquire some resource in an equal (or intentionally unequal) manner.
Currently counters for all threads are summed and normalized which in fact leads to a loss of information about how work of each thread compares to its peers.

Describe the solution you'd like
Is it possible to output the same set of statistics for multithreaded runs on a cross-thread basis as it exists today for repeated runs? I.e. to see median and stddev between N threads of the same multithreaded run?

Describe alternatives you've considered
Tried to leverage existing API, like custom counters but it appears that it also undergoes the same averaging for all threads so basically info about individual thread results is lost already. So there is no alternative as I understand it.

[dmah42]

you're right that this isn't offered. the context here is that multi-threading something to benchmark it is anticipated to be to see what the impact of the threading (and thread counts) has on the overall performance (and throughput, etc). separating out the thread reports would run counter to that expectation.

that being said, i understand the desire and i don't think the library currently supports it, so if someone wants to come up with a design for how we can offer this without complicating the reporting or main loop too much, i'm open to hearing about it.

[jlecavelier]

Just for the record I raised that issue in a previous ticket :
#1849
Wouldn't you agree that it is reasonable for someone who registers a benchmark to run in a multithreaded fashion, explicitly overriding the memory manager, to expect gathering of memory statistics happening as they set it up, in a multi threaded fashion, and not in serial?

I will do some digging and check out how I got around that issue.
I do think that this is a reasonable issue to tackle