Remove Idle time from weight calculations

[JackCA]

I've been trying to debug some long response times with my Node.js web server, but it's almost impossible because the heaviest "Weight" filters take into account the Idle time.

For example:

This is the 1% Weight which took 8.54sec, but with only 80ms of it being non-idle time.

I am able to hide the idle time using the filters, but that doesn't change how weight is calculated.

Is there any way to completely hide idle time or have Weights calculated without idle time?

[nolanmar511]

Right now, Stackdriver Profiler's weight filtering feature is largely language and profile-type agnostic right now, and the weight is always determined. So, weight filtering right now isn't really interesting for wall time profiles collected against a single thread (Python wall profiles, in addition to the Node.js wall profiles would be impacted); for multi-threaded profiles, the profile's total (or weight) would be proportional to the number of threads (so would be useful).

I can think of two options to make wall time profiling useful for the Node.js agent: we can either completely discard "(idle)" time from Node.js wall profiles (I do worry that this would make it less clear that wall profiles are wall profiles, not CPU profiles), or we can add a tag to the profile including the total excluding the "(idle)" portion and use that as a weight.

I am a bit concerned, though, that changing the meaning of the weight filter for Node.js wall profiles, because it's a one-off which will make it more difficult to document and because at some point we may enable worker thread profiling.

@aalexand and @kalyanac -- Any thoughts?

[aalexand]

The problem is real and exists for "wall" and "thread" profiles. E.g. in Go agent's "thread" profiles the weight filter is also often not interesting since there are many parked goroutines which do not represent any work in flight per se. Similar with Java wall profiles - threads often represent parked thread pool worker threads and filtering by the number of those is not useful.

Ideally, as a user I would love to have the focus filter (or a separate filter like that) affect the weight filter. Being able to express filtering like "show me the data from profiles where the number of threads having foo() in the stack is at top 5%" would be a killer.

[nolanmar511]

A weight-filter-specific focus filter would definitely be the dream. It would solve this issue, and we wouldn't have to document any language-type / profile type specific issue.

Do we want to consider a shorter-term solution or aim for the ideal solution?