generated from amazon-archives/__template_Apache-2.0
-
Notifications
You must be signed in to change notification settings - Fork 39
New Automode Observability Patterns #189
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Open
zjaco13
wants to merge
11
commits into
main
Choose a base branch
from
feat/automode
base: main
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
Open
Changes from all commits
Commits
Show all changes
11 commits
Select commit
Hold shift + click to select a range
96733d9
new automode patterns
zjaco13 46734dd
version upgrade + docs for opensource pattern
zjaco13 51db11e
Merge branch 'main' into feat/automode
zjaco13 d7eac6c
fix lint
zjaco13 00d3f9a
fix lint
zjaco13 0ee81cc
update version to try and fix workflow
zjaco13 a72b5b3
remove xray since unsupported currently
zjaco13 86579ec
fix wf
zjaco13 c915d42
rename + mkdocs
zjaco13 47581af
Merge branch 'main' into feat/automode
zjaco13 1de69fb
add deployment length note
zjaco13 File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,6 @@ | ||
| import SingleNewEksAutoModeAWSNativeObservabilityPattern from '../lib/single-new-eks-automode-awsnative-observability-pattern'; | ||
| import { configureApp } from '../lib/common/construct-utils'; | ||
|
|
||
| const app = configureApp(); | ||
|
|
||
| new SingleNewEksAutoModeAWSNativeObservabilityPattern(app, 'single-new-eks-automode-awsnative'); |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,6 @@ | ||
| import { configureApp } from '../lib/common/construct-utils'; | ||
| import SingleNewEksAutoModeOpenSourcePattern from '../lib/single-new-eks-opensource-observability-pattern/automode'; | ||
|
|
||
| const app = configureApp(); | ||
|
|
||
| new SingleNewEksAutoModeOpenSourcePattern(app, 'single-new-eks-automode-opensource'); | ||
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
190 changes: 190 additions & 0 deletions
190
...s-observability-accelerators/single-new-eks-automode-awsnative-observability.md
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,190 @@ | ||
| # Single Cluster AWS Native Observability - Auto Mode | ||
|
|
||
| ## Architecture | ||
|
|
||
| The following figure illustrates the architecture of the pattern we will be deploying for Single EKS Auto Mode Cluster Native Observability pattern using AWS native tools such as CloudWatch Logs and Container Insights. | ||
|
|
||
|  | ||
|
|
||
| This example makes use of CloudWatch Container Insights as a vizualization and metric-aggregation layer. | ||
| Amazon CloudWatch Container Insights helps customers collect, aggregate, and summarize metrics and logs from containerized applications and microservices. Metrics data is collected as performance log events using the embedded metric format. These performance log events use a structured JSON schema that enables high-cardinality data to be ingested and stored at scale. From this data, CloudWatch creates aggregated metrics at the cluster, node, pod, task, and service level as CloudWatch metrics. The metrics that Container Insights collects are available in CloudWatch automatic dashboards. | ||
|
|
||
| By combining Container Insights and CloudWatch logs, we are able to provide a foundation for EKS (Amazon Elastic Kubernetes Service) Observability. Monitoring EKS for metrics has two categories: | ||
| the control plane and the Amazon EKS nodes (with Kubernetes objects). | ||
| The Amazon EKS control plane consists of control plane nodes that run the Kubernetes software, | ||
| such as etcd and the Kubernetes API server. To read more on the components of an Amazon EKS cluster, | ||
| please read the [service documentation](https://docs.aws.amazon.com/eks/latest/userguide/clusters.html). | ||
|
|
||
| ## Objective | ||
|
|
||
| - Deploys one production grade Amazon EKS Auto Mode cluster. | ||
| - Enables Control Plane Logging. | ||
| - AWS Distro For OpenTelemetry Operator and Collector | ||
| - Logs with [AWS for FluentBit](https://github.com/aws/aws-for-fluent-bit) and CloudWatch Logs | ||
| - Enables CloudWatch Container Insights. | ||
| - Installs Prometheus Node Exporter for infrastructure metrics. | ||
|
|
||
| ## Prerequisites | ||
|
|
||
| Ensure that you have installed the following tools on your machine. | ||
|
|
||
| 1. [aws cli](https://docs.aws.amazon.com/cli/latest/userguide/install-cliv2.html) | ||
| 2. [kubectl](https://Kubernetes.io/docs/tasks/tools/) | ||
| 3. [cdk](https://docs.aws.amazon.com/cdk/v2/guide/getting_started.html#getting_started_install) | ||
| 4. [npm](https://docs.npmjs.com/cli/v8/commands/npm-install) | ||
|
|
||
| ## Deploying | ||
|
|
||
| 1. Clone your forked repository | ||
|
|
||
| ```sh | ||
| git clone https://github.com/aws-observability/cdk-aws-observability-accelerator.git | ||
| ``` | ||
|
|
||
| 2. Install the AWS CDK Toolkit globally on your machine using | ||
|
|
||
| ```bash | ||
| npm install -g aws-cdk | ||
| ``` | ||
|
|
||
| 3. Install project dependencies by running `npm install` in the main folder of this cloned repository | ||
|
|
||
| 4. Once all pre-requisites are set you are ready to deploy the pipeline. Run the following command from the root of this repository to deploy the pipeline stack: | ||
|
|
||
| ```bash | ||
| make build | ||
| make pattern single-new-eks-automode-awsnative-observability deploy | ||
| ``` | ||
|
|
||
| ## Verify the resources | ||
|
|
||
| Run update-kubeconfig command. You should be able to get the command from CDK output message. | ||
|
|
||
| ```bash | ||
| aws eks update-kubeconfig --name single-new-eks-automode-awsnative-observability-accelerator --region <your region> --role-arn arn:aws:iam::xxxxxxxxx:role/single-new-eks-awsnative--singleneweksawsnativeauto-JN3QM2KMBNCO | ||
| ``` | ||
|
|
||
| Let’s verify the resources created by steps above. | ||
|
|
||
| ```bash | ||
| kubectl get nodes -o wide | ||
| ``` | ||
|
|
||
| Output: | ||
|
|
||
| ```console | ||
| NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME | ||
| i-08431fdb27179b448 Ready <none> 26m v1.33.4-eks-e386d34 10.0.18.245 3.17.164.102 Bottlerocket (EKS Auto, Standard) 2025.11.8 (aws-k8s-1.33-standard) 6.12.53 containerd://1.7.28+bottlerocket | ||
| ``` | ||
|
|
||
| Next, lets verify the namespaces in the cluster: | ||
|
|
||
| ```bash | ||
| kubectl get ns # Output shows all namespace | ||
| ``` | ||
|
|
||
| Output: | ||
|
|
||
| ```console | ||
| NAME STATUS AGE | ||
| amazon-cloudwatch Active 5h36m | ||
| cert-manager Active 5h36m | ||
| default Active 5h46m | ||
| kube-node-lease Active 5h46m | ||
| kube-public Active 5h46m | ||
| kube-system Active 5h46m | ||
| prometheus-node-exporter Active 5h36m | ||
| ``` | ||
|
|
||
| ## Visualization | ||
|
|
||
| Navigate to CloudWatch and go to "Container Insights". | ||
|
|
||
| View the Container Map: | ||
|
|
||
|  | ||
|
|
||
| View the Performance Monitoring Dashboard: | ||
|
|
||
|  | ||
|
|
||
| ## Viewing Logs | ||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Same feedback. If nothing specific to AM or new, point to old doc link |
||
|
|
||
| Refer to "Using CloudWatch Logs Insights to Query Logs in [Logging](../../logs.md). | ||
|
|
||
| ## Enabling Application Signals for your services | ||
|
|
||
| Amazon CloudWatch Application Signals is a new integrated native APM experience | ||
| in AWS. CloudWatch Application Signals supports **Java**, **Python**, **.NET**, and **Node.js** applications | ||
| running on your Amazon EKS Auto Mode cluster. | ||
|
|
||
| If you haven't enabled Application Signals in this account yet, follow steps 1 - 4 in our [AWS documentation](https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/CloudWatch-Application-Monitoring-Sections.html). | ||
|
|
||
| Next, you have to update your Application to | ||
| `Configure application metrics and trace sampling`. For this, you must add an | ||
| annotation to a manifest YAML in your cluster. Adding this annotation | ||
| auto-instruments the application to send metrics, traces, and logs to | ||
| Application Signals. You have two options for the annotation: | ||
|
|
||
| 1. **Annotate Workload** auto-instruments a single workload in the cluster. | ||
| - Paste the below line into the PodTemplate section of the workload manifest. | ||
|
|
||
| ``` | ||
| apiVersion: apps/v1 | ||
| kind: Deployment | ||
| spec: | ||
| template: | ||
| metadata: | ||
| # add this annotation under the pod template metadata of the services deployment YAML you want to monitor | ||
| annotations: | ||
| instrumentation.opentelemetry.io/inject-java: "true" | ||
| instrumentation.opentelemetry.io/inject-python: "true" | ||
| instrumentation.opentelemetry.io/inject-dotnet: "true"; | ||
| instrumentation.opentelemetry.io/otel-dotnet-auto-runtime: "linux-x64" # for generic Linux glibc based images, this is default value and can be omitted | ||
| instrumentation.opentelemetry.io/otel-dotnet-auto-runtime: "linux-musl-x64" # for Alpine Linux (linux-musl-x64) based images | ||
| instrumentation.opentelemetry.io/inject-nodejs: "true" | ||
| ... | ||
| ``` | ||
|
|
||
| - In your terminal, enter `kubectl apply -f your_deployment_yaml` to apply the change. | ||
|
|
||
| 2. **Annotate Namespace** auto-instruments all workloads deployed in the selected namespace. | ||
| - Paste the below line into the metadata section of the namespace manifest. | ||
|
|
||
| ``` | ||
| annotations: instrumentation.opentelemetry.io/inject-java: "true" | ||
| apiVersion: apps/v1 | ||
| kind: Namespace | ||
| metadata: | ||
| name: <your_namespace> | ||
| # add this annotation under metadata of the namespace manifest you want to monitor | ||
| annotations: | ||
| instrumentation.opentelemetry.io/inject-java: "true" | ||
| instrumentation.opentelemetry.io/inject-python: "true" | ||
| instrumentation.opentelemetry.io/inject-dotnet: "true" | ||
| instrumentation.opentelemetry.io/inject-nodejs: "true" | ||
| ... | ||
| ``` | ||
|
|
||
| - In your terminal, enter `kubectl apply -f your_namespace_yaml` to apply the change. | ||
| - In your terminal, enter a command to restart all pods in the namespace. An example command to restart deployment workloads is `kubectl rollout restart deployment -n namespace_name` | ||
|
|
||
| ## Visualization of CloudWatch Application Signals data | ||
|
|
||
| After enabling your Application to pass metrics and traces by following | ||
| [the steps provided above](#enabling-application-signals-for-your-services), | ||
| open your Amazon CloudWatch console in the same region as your EKS cluster, | ||
| then from the left hand side choose `Application Signals -> Services` and you | ||
| will see the metrics shown on the sample dashboard below: | ||
|
|
||
|  | ||
|
|
||
|  | ||
|
|
||
| ## Teardown | ||
|
|
||
| You can teardown the whole CDK stack with the following command: | ||
|
|
||
| ```bash | ||
| make pattern single-new-eks-automode-awsnative-observability destroy | ||
| ``` | ||
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I see name conflict. The naming scheme of patterns is single-new-eks-automode-opensource-observability.ts and single-new-eks-automode-awsnative-observability.ts. You are mixing it among your patterns. Please correct this uniformly
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
should be resolved throughout