Skip to content

Commit 1da38c7

Browse files
Merge pull request #15 from santhoshkvuda/070223
Release v2.0.1
2 parents 1a44e24 + 8e59c52 commit 1da38c7

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

41 files changed

+11985
-213
lines changed

.gitignore

Lines changed: 57 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,57 @@
1+
.DS_Store
2+
3+
####
4+
## Ignore PEM files
5+
####
6+
7+
**.pem
8+
9+
####
10+
## gitignore for terraform artifacts
11+
####
12+
13+
# Local .terraform directories
14+
**/.terraform/*
15+
16+
## Terraform Locck files
17+
*.terraform.lock.hcl
18+
19+
# .tfstate filesdas
20+
*.tfstate
21+
*.tfstate.*
22+
23+
# Crash log files
24+
crash.log
25+
crash.*.log
26+
27+
# Exclude all .tfvars files, which are likely to contain sensitive data, such as
28+
# password, private keys, and other secrets. These should not be part of version
29+
# control as they are data points which are potentially sensitive and subject
30+
# to change depending on the environment.
31+
*.tfvars
32+
*.tfvars.json
33+
34+
# Include sample tfvars
35+
!terraform-sample.tfvars
36+
37+
# Ignore override files as they are usually used to override resources locally and so
38+
# are not checked in
39+
override.tf
40+
override.tf.json
41+
*_override.tf
42+
*_override.tf.json
43+
44+
# Include override files you do wish to add to version control using negated pattern
45+
# !example_override.tf
46+
47+
# Include tfplan files to ignore the plan output of command: terraform plan -out=tfplan
48+
# example: *tfplan*
49+
50+
# Ignore CLI configuration files
51+
.terraformrc
52+
terraform.rc
53+
54+
# Ignore util dir
55+
logan/util/*
56+
57+

CHANGELOG.md

Lines changed: 11 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,16 @@
11
# Change Log
22

3+
## 2022-02-07
4+
### Added
5+
- Create a new mount (rw) using the value provided for baseDir.
6+
- Expose "encoding" parameter of Fluentd's tail plugin as part of values.yaml, which allows users to override default encoding (ASCII-8BIT) for applicable logs/log types.
7+
- Partial CRI logs handling.
8+
- Oracle Resource Manager / Terraform support for deploying the solution.
9+
### Changed
10+
- Modified /var/log to mount as readonly by default, except when /var/log is set as baseDir (to store Fluentd state, buffer etc.,).
11+
### Breaking Changes
12+
- Logging Analytics Fluentd Output plugin log location will be derived using baseDir instead using value of fluentd:ociLoggingAnalyticsOutputPlugin:plugin_log_location. The default value still remains unchanged and is a non breaking change except if it was modified to a different value.
13+
314
## 2022-08-30
415
### Added
516
- Helm chart templatisation/parameterisation to provide granular level control on the chart and its values.

README.md

Lines changed: 127 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -4,10 +4,13 @@
44

55
This provides an end-to-end monitoring solution for Oracle Container Engine for Kubernetes (OKE) and other forms of Kubernetes Clusters using Logging Analytics, Monitoring and other Oracle Cloud Infrastructure (OCI) Services.
66

7-
![Sample Services Dashboard](https://user-images.githubusercontent.com/80283985/153080889-62b30482-5a9c-4244-92e3-e7a4df5ba33e.png)
7+
![Kubernetes Cluster Summary Dashboard](logan/images/kubernetes-cluster-summary-dashboard.png)
88

9+
![Kubernetes Nodes Dashboard](logan/images/kubernetes-nodes-dashboard.png)
910

10-
![Topology Based Exploration](https://user-images.githubusercontent.com/80283985/153081174-f22dcf71-d994-4dc5-ad42-9f424c3f1573.png)
11+
![Kubernetes Workloads Dashboard](logan/images/kubernetes-workloads-dashboard.png)
12+
13+
![Kubernetes Pods Dashboard](logan/images/kubernetes-pods-dashboard.png)
1114

1215
## Logs
1316

@@ -75,6 +78,26 @@ The following are the list of objects supported at present:
7578

7679
## Installation Instructions
7780

81+
### Deploy using Oracle Resource Manager
82+
83+
> **_NOTE:_** If you aren't already signed in, when prompted, enter the tenancy and user credentials. Review and accept the terms and conditions. If you aren't on-boarded to OCI Logging Analytics, refer to [Pre-requisites](#pre-requisites) section to enable Logging Analytics in the region where you want to deploy the stack. The default container image available through the deployment is only for demo/non-production use-cases, we recommend you to refer [Docker Image](#docker-image) section to build your own image.
84+
85+
- Click to deploy the stack
86+
87+
[![Deploy to Oracle Cloud][orm_button]][oci_kubernetes_monitoring_stack]
88+
89+
- Select the region and compartment where you want to deploy the stack.
90+
91+
- Follow the on-screen prompts and instructions to create the stack.
92+
93+
- After creating the stack, click Terraform Actions, and select Plan.
94+
95+
- Wait for the job to be completed, and review the plan.
96+
97+
- To make any changes, return to the Stack Details page, click Edit Stack, and make the required changes. Then, run the Plan action again.
98+
99+
- If no further changes are necessary, return to the Stack Details page, click Terraform Actions, and select Apply.
100+
78101
### Pre-requisites
79102

80103
- Logging Analytics Service must be enabled in the given OCI region before trying out the following Solution. Refer [Logging Analytics Quick Start](https://docs.oracle.com/en-us/iaas/logging-analytics/doc/quick-start.html) for details.
@@ -393,3 +416,105 @@ subjects:
393416
name: <serviceaccount>
394417
namespace: <namespace>
395418
```
419+
420+
### How to set encoding for logs ?
421+
422+
**Note**: This is supported only through the helm chart based deployment.
423+
424+
By default Fluentd tail plugin that is being used to collect various logs has default encoding set to ASCII-8BIT. To overrided the default encoding, use one of the following approaches.
425+
426+
#### Global level
427+
428+
Set value for encoding under fluentd:tailPlugin section of values.yaml, which applies to all the logs being collected from the cluster.
429+
430+
```
431+
fluentd:
432+
...
433+
...
434+
tailPlugin:
435+
...
436+
...
437+
encoding: <ENCODING-VALUE>
438+
```
439+
440+
#### Specific log type level
441+
442+
The encoding can be set at invidivual log types like kubernetesSystem, linuxSystem, genericContainerLogs, which applies to all the logs under the specific log type.
443+
444+
```
445+
fluentd:
446+
...
447+
...
448+
kubernetesSystem:
449+
...
450+
...
451+
encoding: <ENCODING-VALUE>
452+
```
453+
454+
```
455+
fluentd:
456+
...
457+
...
458+
genericContainerLogs:
459+
...
460+
...
461+
encoding: <ENCODING-VALUE>
462+
```
463+
464+
#### Specific log level
465+
466+
The encoding can be set at individual log level too, which takes precedence over all others.
467+
468+
```
469+
fluentd:
470+
...
471+
...
472+
kubernetesSystem:
473+
...
474+
...
475+
logs:
476+
kube-proxy:
477+
encoding: <ENCODING-VALUE>
478+
```
479+
480+
```
481+
fluentd:
482+
...
483+
...
484+
customLogs:
485+
custom-log1:
486+
...
487+
...
488+
encoding: <ENCODING-VALUE>
489+
...
490+
...
491+
```
492+
493+
## Importing Logging Analytics Kubernetes Dashboards
494+
495+
The Dashboards are imported as part of deploying the Kubernetes solution using [Oracle Resource Manager stack](#deploy-using-oracle-resource-manager). The following steps can be used to import the Dashboards manually to your tenancy.
496+
497+
1. Download and configure [OCI CLI](https://docs.oracle.com/en-us/iaas/Content/API/SDKDocs/cliinstall.htm) or open cloud-shell where OCI CLI is pre-installed. Alternative methods like REST API, SDK, Terraform etc can also be used.
498+
1. Find the **OCID** of compartment, where the dashboards need to be imported.
499+
1. Download the dashboard JSONs from [here](logan/terraform/oke/modules/dashboards/dashboards_json/).
500+
1. **Replace** all the instances of the keyword - "`${compartment_ocid}`" in the JSONs with the **Compartment OCID** identified in STEP 2.
501+
- Following are the set of commands for quick reference that can be used in a linux/cloud-shell envirnment :
502+
503+
```
504+
sed -i "s/\${compartment_ocid}/<Replace-with-Compartment-OCID>/g" file://cluster.json
505+
sed -i "s/\${compartment_ocid}/<Replace-with-Compartment-OCID>/g" file://node.json
506+
sed -i "s/\${compartment_ocid}/<Replace-with-Compartment-OCID>/g" file://workload.json
507+
sed -i "s/\${compartment_ocid}/<Replace-with-Compartment-OCID>/g" file://pod.json
508+
```
509+
1. Run the following commands to import the dashboards.
510+
511+
```
512+
oci management-dashboard dashboard import --from-json file://cluster.json
513+
oci management-dashboard dashboard import --from-json file://node.json
514+
oci management-dashboard dashboard import --from-json file://workload.json
515+
oci management-dashboard dashboard import --from-json file://pod.json
516+
```
517+
518+
[orm_button]: https://oci-resourcemanager-plugin.plugins.oci.oraclecloud.com/latest/deploy-to-oracle-cloud.svg
519+
520+
[oci_kubernetes_monitoring_stack]: https://cloud.oracle.com/resourcemanager/stacks/create?zipUrl=https://github.com/oracle-quickstart/oci-kubernetes-monitoring/releases/latest/download/oci-kubernetes-monitoring-stack.zip

logan/helm-chart/Chart.yaml

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -15,7 +15,7 @@ type: application
1515
# This is the chart version. This version number should be incremented each time you make changes
1616
# to the chart and its templates, including the app version.
1717
# Versions are expected to follow Semantic Versioning (https://semver.org/)
18-
version: 2.0.0
18+
version: 2.0.1
1919

2020
# This is the version number of the application being deployed. This version number should be
2121
# incremented each time you make changes to the application. Versions are not expected to

logan/helm-chart/templates/configmap-logs.yaml

Lines changed: 75 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -46,7 +46,7 @@ data:
4646
config_file_location {{ .Values.oci.path }}/{{ .Values.oci.file }}
4747
profile_name "{{ .Values.fluentd.ociLoggingAnalyticsOutputPlugin.profile_name }}"
4848
{{- end }}
49-
plugin_log_location "{{ .Values.fluentd.ociLoggingAnalyticsOutputPlugin.plugin_log_location }}"
49+
plugin_log_location "{{ .Values.fluentd.baseDir }}"
5050
plugin_log_level "{{ .Values.fluentd.ociLoggingAnalyticsOutputPlugin.plugin_log_level }}"
5151
plugin_log_file_size "{{ .Values.fluentd.ociLoggingAnalyticsOutputPlugin.plugin_log_file_size }}"
5252
plugin_log_file_count "{{ .Values.fluentd.ociLoggingAnalyticsOutputPlugin.plugin_log_file_count }}"
@@ -90,6 +90,13 @@ data:
9090
pos_file {{ $.Values.fluentd.baseDir }}/oci_la_fluentd_outplugin/pos/{{ $name }}.logs.pos
9191
tag oci.oke.{{ $name }}.*
9292
read_from_head "{{ $.Values.fluentd.tailPlugin.readFromHead }}"
93+
{{- if $logDefinition.encoding }}
94+
encoding {{ $logDefinition.encoding }}
95+
{{- else if $.Values.fluentd.kubernetesSystem.encoding }}
96+
encoding {{ $.Values.fluentd.kubernetesSystem.encoding }}
97+
{{- else if $.Values.fluentd.tailPlugin.encoding }}
98+
encoding {{ $.Values.fluentd.tailPlugin.encoding }}
99+
{{- end }}
93100
<parse>
94101
{{- if eq $runtime "docker" }}
95102
@type json
@@ -134,6 +141,20 @@ data:
134141
tag ${tag}
135142
</record>
136143
</filter>
144+
# Concat filter to handle partial logs in CRI/ContainerD
145+
# Docker can also have partial logs but handling is different for different docker versions. Considering Kubernetes/OKE moved to ContainerD/CRI since last 4-5 releases, ignoring docker handling.
146+
# This filter can not be clubbed with concat filter for multiline as both are mutually exclusive.
147+
{{- if eq $runtime "cri" }}
148+
<filter oci.oke.{{ $name }}.**>
149+
@type concat
150+
key message
151+
use_partial_cri_logtag true
152+
partial_cri_logtag_key logtag
153+
partial_cri_stream_key stream
154+
# timeout scenario should not occur in general for partial logs handling
155+
timeout_label "@NORMAL"
156+
</filter>
157+
{{- end }}
137158
{{- if $logDefinition.multilineStartRegExp }}
138159
# Concat filter to handle multi-line log records.
139160
<filter oci.oke.{{ $name }}.**>
@@ -159,6 +180,13 @@ data:
159180
pos_file {{ $.Values.fluentd.baseDir }}/oci_la_fluentd_outplugin/pos/{{ $name }}.logs.pos
160181
tag oci.oke.{{ $name }}.*
161182
read_from_head "{{ $.Values.fluentd.tailPlugin.readFromHead }}"
183+
{{- if $logDefinition.encoding }}
184+
encoding {{ $logDefinition.encoding }}
185+
{{- else if $.Values.fluentd.linuxSystem.encoding }}
186+
encoding {{ $.Values.fluentd.linuxSystem.encoding }}
187+
{{- else if $.Values.fluentd.tailPlugin.encoding }}
188+
encoding {{ $.Values.fluentd.tailPlugin.encoding }}
189+
{{- end }}
162190
<parse>
163191
{{- if $logDefinition.multilineStartRegExp }}
164192
@type multiline
@@ -217,6 +245,13 @@ data:
217245
pos_file {{ .Values.fluentd.baseDir }}/oci_la_fluentd_outplugin/pos/syslog.logs.pos
218246
tag oci.oke.syslog.messages.**
219247
read_from_head "{{ .Values.fluentd.tailPlugin.readFromHead }}"
248+
{{- if .Values.fluentd.linuxSystem.logs.syslog.encoding }}
249+
encoding {{ .Values.fluentd.linuxSystem.logs.syslog.encoding }}
250+
{{- else if .Values.fluentd.linuxSystem.encoding }}
251+
encoding {{ .Values.fluentd.linuxSystem.encoding }}
252+
{{- else if .Values.fluentd.tailPlugin.encoding }}
253+
encoding {{ .Values.fluentd.tailPlugin.encoding }}
254+
{{- end }}
220255
<parse>
221256
@type multiline
222257
format_firstline {{ .Values.fluentd.linuxSystem.logs.syslog.multilineStartRegExp }}
@@ -325,6 +360,11 @@ data:
325360
pos_file {{ $.Values.fluentd.baseDir }}/oci_la_fluentd_outplugin/pos/{{ $name }}.logs.pos
326361
tag oci.oke.{{ $name }}.*
327362
read_from_head "{{ $.Values.fluentd.tailPlugin.readFromHead }}"
363+
{{- if $logDefinition.encoding }}
364+
encoding {{ $logDefinition.encoding }}
365+
{{- else if $.Values.fluentd.tailPlugin.encoding }}
366+
encoding {{ $.Values.fluentd.tailPlugin.encoding }}
367+
{{- end }}
328368
<parse>
329369
{{- if eq "false" ($logDefinition.isContainerLog | toString) }}
330370
{{- if $logDefinition.multilineStartRegExp }}
@@ -368,7 +408,20 @@ data:
368408
tag ${tag}
369409
</record>
370410
</filter>
371-
411+
# Concat filter to handle partial logs in CRI/ContainerD
412+
# Docker can also have partial logs but handling is different for different docker versions. Considering Kubernetes/OKE moved to ContainerD/CRI since last 4-5 releases, ignoring docker handling.
413+
# This filter can not be clubbed with concat filter for multiline as both are mutually exclusive.
414+
{{- if and (ne "false" ($logDefinition.isContainerLog | toString)) (eq $runtime "cri") }}
415+
<filter oci.oke.{{ $name }}.**>
416+
@type concat
417+
key message
418+
use_partial_cri_logtag true
419+
partial_cri_logtag_key logtag
420+
partial_cri_stream_key stream
421+
# timeout scenario should not occur in general for partial logs handling
422+
timeout_label "@NORMAL"
423+
</filter>
424+
{{- end }}
372425
{{- if and (ne "false" ($logDefinition.isContainerLog | toString)) ($logDefinition.multilineStartRegExp) }}
373426
# Concat filter to handle multi-line log records.
374427
<filter oci.oke.{{ $name }}.**>
@@ -397,6 +450,11 @@ data:
397450
read_from_head "{{ .Values.fluentd.tailPlugin.readFromHead }}"
398451
# Modify the exclude path once a specific container log config is explictly defined to avoid duplicate collection.
399452
exclude_path [{{ $excludePath }}]
453+
{{- if .Values.fluentd.genericContainerLogs.encoding }}
454+
encoding {{ .Values.fluentd.genericContainerLogs.encoding }}
455+
{{- else if .Values.fluentd.tailPlugin.encoding }}
456+
encoding {{ .Values.fluentd.tailPlugin.encoding }}
457+
{{- end }}
400458
<parse>
401459
{{- if eq $runtime "docker" }}
402460
@type json
@@ -449,6 +507,21 @@ data:
449507
</filter>
450508
{{- end }}
451509
510+
# Concat filter to handle partial logs in CRI/ContainerD
511+
# Docker can also have partial logs but handling is different for different docker versions. Considering Kubernetes/OKE moved to ContainerD/CRI since last 4-5 releases, ignoring docker handling.
512+
# This filter can not be clubbed with concat filter for multiline as both are mutually exclusive.
513+
{{- if eq $runtime "cri" }}
514+
<filter oci.oke.containerlogs.**>
515+
@type concat
516+
key message
517+
use_partial_cri_logtag true
518+
partial_cri_logtag_key logtag
519+
partial_cri_stream_key stream
520+
# timeout scenario should not occur in general for partial logs handling
521+
timeout_label "@NORMAL"
522+
</filter>
523+
{{- end }}
524+
452525
#customFluentd config
453526
{{- if .Values.fluentd.customFluentdConf }}
454527
{{- include "common.tplvalues.render" (dict "value" .Values.fluentd.customFluentdConf "context" $) | nindent 4 }}

logan/helm-chart/templates/configmap-objects.yaml

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -18,7 +18,7 @@ data:
1818
config_file_location {{ .Values.oci.path }}/{{ .Values.oci.file }}
1919
profile_name "{{ .Values.fluentd.ociLoggingAnalyticsOutputPlugin.profile_name }}"
2020
{{- end }}
21-
plugin_log_location "{{ .Values.fluentd.ociLoggingAnalyticsOutputPlugin.plugin_log_location }}"
21+
plugin_log_location "{{ .Values.fluentd.baseDir }}"
2222
plugin_log_level "{{ .Values.fluentd.ociLoggingAnalyticsOutputPlugin.plugin_log_level }}"
2323
plugin_log_file_size "{{ .Values.fluentd.ociLoggingAnalyticsOutputPlugin.plugin_log_file_size }}"
2424
plugin_log_file_count "{{ .Values.fluentd.ociLoggingAnalyticsOutputPlugin.plugin_log_file_count }}"
@@ -103,4 +103,4 @@ data:
103103
tag ${tag}
104104
</record>
105105
</filter>
106-
{{- end }}
106+
{{- end }}

0 commit comments

Comments
 (0)