Observability with Istio 1.9

[StuartHarris]

• update kube-state-metrics to work with the recently updated Kubernetes version (1.17)
• separate out oms-agent manifests
• group prometheus, oms-agent and kube-state-metrics under observability folder
• update Makefile in cluster-init
• manifests for prometheus install
• update prometheus config to use sidecar (without intercepting scrapes)
• test config and remove old configmap
• validate that metrics are being captured for Istio, K8s and our installed services

[StuartHarris]

Small point, wondering if this line is valuable. Probs won't restart prometheus pods unless there's an update. Maybe we should update the readme to do the make instead? What do you think?

[TimboTambo]

This command will be used when istio has been upgraded, in which case I think this line will cause the prometheus pod to restart due to istioctl kube-inject, which is what we want - is that right?

[StuartHarris]

Yep, that's true. 👍