Support for Query Insights to enable performance and cost analysis

[MarcBridner]

Is your feature request related to a problem? Please describe.
We recently saw a $3k spike in our Timestream bill during a single day. We only run dashboards off of our Timestream instance and we suspect that a user forgot the WHERE clause on a query and/or had a really low refresh interval. We have 1000's of panels, so tracking the cause down is difficult to say the least.

Timestream does not expose any useful metrics on a per-query level in a centralized place, but there is a Query Insights feature that can be used to do performance and cost optimizations.

Query Insights is enabled per query and adds execution path metadata to the response.

1. https://docs.aws.amazon.com/timestream/latest/developerguide/API_query_QueryInsights.html
2. https://docs.aws.amazon.com/timestream/latest/developerguide/API_query_QueryInsightsResponse.html

Describe the solution you'd like
I would like an option to turn on Query Insights for all queries, and log the resulting query insight responses to a separate file on the Grafana host(s).

This would allow us to ship logs elsewhere for analysis.

Describe alternatives you've considered
No alternative that does not increase complexity.

[idastambuk]

Hi @MarcBridner, adding query insights is certainly a good improvement. The way I imagine it is that it could be in the form of an additional response frame (named "query insights") for each query response, would that work for your use case? I will add it to our backlog to implement in the future.

However, if you'd like to see it sooner than later, we encourage external PR submissions! The only blocker for this is the Timestream migration to aws-sdk-go-v2 cause it seems like this API isn't available in v1. We're hoping to get that done soon

[MarcBridner]

Hi

I think having it available as a separate frame would be useful when analyzing the cost of an individual panel, but what I'm requesting is to offline analyze it for ALL queries sent by Grafana - so writing it to disk (or I guess the Grafana DB?) seems to be the only reasonably easy thing to do ... until this gets built in Timestream by AWS.

[MarcBridner]

Just to give other people an idea of the severity of what a few run-away timestream dashboards with auto-refresh can do, we just got hit with a 6 digit bill from Timestream.

[MarcBridner]

It turns out enabling query insights will reduce the TPS rate by a lot, so its not actually very useful.

What ended up being useful for us is output.QueryStatus.CumulativeBytesScanned and output.QueryStatus.CumulativeBytesMetered.

We modified a local copy of the plugin to emit AWS EMF metrics per dashboard/panel with these values. Dashboard name and panel ID was taken from the HTTP headers, if present.

Since we run on Fargate, CW will pick these metrics up for us.
This is unfortunately a hack solution that probably only works for us, and isn't worth contributing back to this plugin.

A better solution would be to emit Grafana metrics for the plugin with these values. That would require Prometheus or Graphite to be useful, of which we have neither

This issue can probably be resolved