@@ -25,7 +25,7 @@ Prometheus
25
25
26
26
You need Prometheus to scrape and store metrics from applications in its
27
27
time series database. These metrics will be used by the HPA to decide if it
28
- has to scale the application. You can use an already running Prometheus in
28
+ has to scale the application. You can use an already running Prometheus in
29
29
your environment or opt to set one up with the following steps.
30
30
31
31
.. warning ::
@@ -48,7 +48,7 @@ Prometheus, Alertmanager and configure ServiceMonitors.
48
48
helm install coreos/prometheus-operator --name prometheus-operator --namespace application-monitoring
49
49
50
50
After installing the Prometheus operator you can install Prometheus. To accomplish
51
- this you use an yaml values file. This yaml file defines you want 2 Prometheus
51
+ this you use a yaml values file. This yaml file defines you want 2 Prometheus
52
52
pods running, keep data for 14 days and want to have a persistent volume
53
53
of 20 GB per Prometheus pod. You can tweak the Prometheus install further
54
54
by following the official `documentation <https://github.com/coreos/prometheus-operator/tree/master/helm/prometheus >`__.
@@ -88,22 +88,27 @@ You need to create a CA and SSL cert to validate your APIService with Kubernetes
88
88
openssl req -x509 -sha256 -new -nodes -days 365 -newkey rsa:2048 -keyout ${PURPOSE} -ca.key -out ${PURPOSE} -ca.crt -subj " /CN=ca"
89
89
echo ' {"signing":{"default":{"expiry":"43800h","usages":["signing","key encipherment","' ${PURPOSE} ' "]}}}' > " ${PURPOSE} -ca-config.json"
90
90
91
- export SERVICE_NAME=prometheus-adapter-prometheus-adapter
92
- export ALT_NAMES=' "prometheus-adapter-prometheus-adapter .application-monitoring","prometheus-adapter- prometheus-adapter.application-monitoring.svc"'
91
+ export SERVICE_NAME=prometheus-adapter
92
+ export ALT_NAMES=' "prometheus-adapter.application-monitoring","prometheus-adapter.application-monitoring.svc"'
93
93
echo ' {"CN":"' ${SERVICE_NAME} ' ","hosts":[' ${ALT_NAMES} ' ],"key":{"algo":"rsa","size":2048}}' | cfssl gencert -ca=server-ca.crt -ca-key=server-ca.key -config=server-ca-config.json - | cfssljson -bare apiserver
94
94
95
+
96
+ .. warning ::
97
+ Make sure the ``SERVICE_NAME `` and ``ALT_NAMES `` match your application release
98
+ name and namespace where it is deployed.
99
+
95
100
Now create an ``prometheus-adapater.yaml `` with the following content:
96
101
97
102
.. code-block :: yaml
98
103
99
104
tls :
100
105
enable : true
101
106
ca : |-
102
- <content of server-ca.crt>
107
+ <replace with content of server-ca.crt>
103
108
key : |-
104
- <content of apiserver-key.pem>
109
+ <replace with content of apiserver-key.pem>
105
110
certificate : |-
106
- <content of apiserver.pem>
111
+ <replace with content of apiserver.pem>
107
112
108
113
# Change URL and port if you setup your own Prometheus server.
109
114
prometheus :
@@ -117,15 +122,16 @@ Install the Prometheus Adapter:
117
122
.. code-block :: bash
118
123
119
124
helm repo add incubator https://kubernetes-charts-incubator.storage.googleapis.com/
120
- helm install incubator /prometheus-adapter --name prometheus-adapter --namespace application-monitoring -f prometheus-adapter.yaml
125
+ helm install stable /prometheus-adapter --name prometheus-adapter --namespace application-monitoring -f prometheus-adapter.yaml
121
126
122
127
123
- Now you can test if it works by running the following command against your
128
+ You can test if HPA works by running the following command against your
124
129
Kubernetes cluster.
125
130
126
131
.. code-block :: bash
127
132
128
133
kubectl get --raw /apis/custom.metrics.k8s.io/v1beta1
134
+
129
135
{" kind" :" APIResourceList" ," apiVersion" :" v1" ," groupVersion" :" custom.metrics.k8s.io/v1beta1" ," resources" :[]}
130
136
131
137
Usage
@@ -144,7 +150,7 @@ your application with Prometheus.
144
150
kind : ServiceMonitor
145
151
metadata :
146
152
name : <example>
147
- namespace : <example_namespace>
153
+ namespace : application-monitoring
148
154
labels :
149
155
prometheus : prometheus-applications
150
156
spec :
@@ -159,32 +165,31 @@ your application with Prometheus.
159
165
matchLabels :
160
166
<key> : <value that matches your application>
161
167
162
- When adding the following ``ServiceMonitor ``, make sure to keep ``prometheus ``
163
- as an key in labels, this is how Prometheus discovers the different
164
- ServiceMonitors .
168
+ When adding the ``ServiceMonitor ``, make sure to keep ``prometheus `` as an key
169
+ in labels, that is how Prometheus discovers the different ServiceMonitors.
170
+ The `` ServiceMonitor `` has to be deployed in the same namespace as your Prometheus .
165
171
166
172
After applying the ``ServiceMonitor ``, Prometheus should start discovering
167
173
all your application pods and start to monitor them.
168
174
169
- Now you can find the correct metric and you can get it out of the custom.metrics
170
- API endpoint.
175
+ You can find the correct metric by querying the custom.metrics API endpoint.
171
176
172
177
.. code-block :: bash
173
178
174
179
kubectl get --raw /apis/custom.metrics.k8s.io/v1beta1 | jq
175
- kubectl get --raw " /apis/custom.metrics.k8s.io/v1beta1/namespaces/default /pods/*/requestcount " | jq
176
- kubectl get --raw /apis/custom.metrics.k8s.io/v1beta1/namespaces/default /service/example/requestcount | jq
180
+ kubectl get --raw " /apis/custom.metrics.k8s.io/v1beta1/namespaces/<aplication_namespace> /pods/*/<metric_name> " | jq
181
+ kubectl get --raw /apis/custom.metrics.k8s.io/v1beta1/namespaces/< aplication_namespace > /service/< service_name > / < metric_name > | jq
177
182
178
- When you found the correct metric to scale on, you can create your
183
+ When you found the correct metric to scale on, you can create your
179
184
``HorizontalPodAutoscaler ``.
180
185
181
186
.. code-block :: yaml
182
187
183
188
kind : HorizontalPodAutoscaler
184
189
apiVersion : autoscaling/v2beta1
185
190
metadata :
186
- name : example
187
- namespace : default
191
+ name : < example>
192
+ namespace : <example_namespace>
188
193
spec :
189
194
scaleTargetRef :
190
195
apiVersion : apps/v1beta2
@@ -193,15 +198,12 @@ When you found the correct metric to scale on, you can create your
193
198
minReplicas : 2
194
199
maxReplicas : 4
195
200
metrics :
196
- - type : Object
197
- object :
198
- target :
199
- kind : Service
200
- name : example
201
- metricName : requestcount
202
- targetValue : 30
201
+ - type : Pods
202
+ pods :
203
+ metricName : <metric_name>
204
+ targetAverageValue : <metric_value>
203
205
204
- Now you watch the horizontal pod autoscaler:
206
+ Watch the horizontal pod autoscaler:
205
207
206
208
.. code-block :: bash
207
209
@@ -212,3 +214,6 @@ More examples can be found in the kubernetes `documentation <https://kubernetes.
212
214
213
215
.. warning ::
214
216
Certainly take a look a types ``Object `` and ``Pods `` for HPA based on custom-metrics.
217
+
218
+ A fully worked out example with a example application that has metrics can be found in
219
+ `luxas repo <https://github.com/luxas/kubeadm-workshop/blob/master/demos/monitoring/sample-metrics-app.yaml#L51 >`__.
0 commit comments