Skip to content
40 changes: 21 additions & 19 deletions knowledge-content/opentelemetry-demo/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -90,24 +90,24 @@ Log in to your Oracle Cloud Infrastructure console and navigate to the Applicati
### Visualising Traces and Spans
Click on the "Traces" tab in the APM console / Trace Explorer. You will see a list of traces generated by the OTel demo application. Each trace represents a user request or an operation within your application:

![Trace-Explorer-05](https://github.com/user-attachments/assets/81e98ec3-cac9-4498-9c58-4a204e361dc0)
![Trace-Explorer-05](https://github.com/user-attachments/assets/2046b3bd-fea5-4ec4-aa80-decec7e2e7ec)

Select a specific trace to view its details, including the topology of the trace and its spans:

![Trace-Explorer-04](https://github.com/user-attachments/assets/e864abb7-3db0-4491-8bde-a3a7e7141c30)
![Trace-Explorer-04](https://github.com/user-attachments/assets/6b7a7c54-2ea0-4af4-a180-40c0f127e5a1)

Click on any of the spans to see the span details, which include kubernetes data, SpanId, TraceID, etc.:

![Trace-Explorer-03](https://github.com/user-attachments/assets/c6244f5c-c8ba-418b-a5dd-eac696e9cea9)
![Trace-Explorer-03](https://github.com/user-attachments/assets/526b6e3f-b5b8-48d6-bcac-2baaa250f67d)

Services Topology view allows to visualize associations between services, several parameters can be used for the arrow width:

![Service-Topology-07](https://github.com/user-attachments/assets/78e97dde-f569-4e18-b9e2-ebc5c95077f5)
![Service-Topology-07](https://github.com/user-attachments/assets/4b25dbe7-004f-46b1-80ad-fa436310aeec)

![Service-Topology-08](https://github.com/user-attachments/assets/be3f58ec-7875-4514-a759-36e8297e2488)
![Service-Topology-08](https://github.com/user-attachments/assets/6e05eca8-96b8-402b-a4e2-594d1a972d70)

## Using OCI Logging Analytics service to collect and analyze Logs from Kubernetes Infrastructure and Pods
OCI Logging Analytics provides a [complete solution](https://docs.oracle.com/en-us/iaas/logging-analytics/doc/kubernetes-solution.html) for monitoring Kubernetes (K8s) cluster deployed in OCI, third party public clouds, private clouds, or on-premises including managed Kubernetes deployments. We will start with discovering the K8s cluster running the OpenTelemetry Demo App.
OCI Logging Analytics (LA) provides a [complete solution](https://docs.oracle.com/en-us/iaas/logging-analytics/doc/kubernetes-solution.html) for monitoring Kubernetes (K8s) cluster deployed in OCI, third party public clouds, private clouds, or on-premises including managed Kubernetes deployments. We will start with discovering the K8s cluster running the OpenTelemetry Demo App.

### Getting Started with monitoring a K8s Cluster using OCI Logging Analytics
Log in to your Oracle Cloud Infrastructure console and navigate to Logging Analytics Administration. Select Solutions -> Kubernetes -> Connect Clusters -> Monitor Kubernetes -> Oracle OKE, here we are assuming that the K8s cluster is running in OCI. Select the Cluster, press *Next* and select the right compartment to be used for telemetry data and related monitoring resources, usually it will be the same like the one used for collecting the logs. Click on *Configure log collection*, this will create all neded dynamic groups and policies to allow collecting logs, metrics, and object information from related Kubernetes components, compute nodes, subnets, and load balancers. The deployed solution will create these statefulsets, daemonsets and cronjobs in namespace *oci-onm*:
Expand All @@ -124,28 +124,27 @@ oci-onm-mgmt-agent-0 1/1 Running 2 80d
```
### Screenshots from the Solution Dashboard
Cluster overview:
![K8s-Solution-01](https://github.com/user-attachments/assets/3d767718-60aa-4b47-b84a-93fedf5f9db4)
![K8s-Solution-01](https://github.com/user-attachments/assets/95c10df5-ce67-485d-a5b8-fc6530e0f835)

Workload overview:
![K8s-Solution-02](https://github.com/user-attachments/assets/e8bad199-f6c0-4155-8140-0758cdffab12)
![K8s-Solution-02](https://github.com/user-attachments/assets/010717eb-e42a-4644-9e03-b7b8fe2b3e74)

Node overview:
![K8s-Solution-03](https://github.com/user-attachments/assets/ce809518-70e9-4d09-8cbe-0f3ff4662172)
![K8s-Solution-03](https://github.com/user-attachments/assets/4ead23d9-5d04-412b-8a24-a6e9c7d5a7cd)

Pod overview:
![K8s-Solution-04](https://github.com/user-attachments/assets/a0d7dbe1-9cc9-4219-886e-fa8f4691a382)
![K8s-Solution-04](https://github.com/user-attachments/assets/700f4a9f-d92e-483e-acc7-422165eb0006)

### Screenshots from Log Explorer

List of TOP Log Sources:
![K8s-Log-Sources](https://github.com/user-attachments/assets/a4637601-d932-436a-9921-2f333f0d5450)
![K8s-Log-Sources](https://github.com/user-attachments/assets/18a3d344-6b7c-43dc-99a3-647b13d97ddf)

Log Records by Pod name:
![K8s-Log-by-Pods](https://github.com/user-attachments/assets/47934bca-3084-47e7-87e6-4df986cbe42a)
![K8s-Log-by-Pods](https://github.com/user-attachments/assets/3e3499ef-06ea-4dcf-bfee-a6300015edb1)

Compare Cluster results:
![K8s-Log-Sources_compare-cluster](https://github.com/user-attachments/assets/27d78650-85fe-4c73-9566-08e115e3ebc4)

![K8s-Log-Sources_compare-cluster](https://github.com/user-attachments/assets/59ab42d7-6ffb-4eaf-ba92-a1a928bbddc7)

## Using OCI Monitoring service to inspect the collected metrics

Expand All @@ -155,17 +154,20 @@ The collected metrics can be inspected using the Metrics Explorer from the OCI M

Metrics collected by the Management Agent can be found in metric namespace *mgmtagent_kubernetes_metrics*:

![Metrics-Explorer-06](https://github.com/user-attachments/assets/9863f36e-7097-4b34-ae7e-c11454335118)

![Metrics-Explorer-06](https://github.com/user-attachments/assets/05192ea6-091f-4ad2-ba68-2fabc37d3bff)

Metrics sent by the OpenTelemetry Collector can be found in metric namespace *oracle_apm_monitoring*:

![Metrics-Explorer-03](https://github.com/user-attachments/assets/1aa3bea0-4bd1-4bba-81e1-eff9a2d474ea)
![Metrics-Explorer-03](https://github.com/user-attachments/assets/3dfcff66-b2e5-40fc-8d06-50a8e9c311e6)

Finally, out-of-the-box or custom dashboards can be used to visualize the collected metrics:

![k8s-metrics-dashboard-01](https://github.com/user-attachments/assets/5b93d412-4974-4049-a22b-5c8d5be89fc1)
![k8s-metrics-dashboard-01](https://github.com/user-attachments/assets/7d93b237-2b63-40ae-b837-e33269a94fcd)

![k8s-metrics-dashboard-02](https://github.com/user-attachments/assets/1c7173fa-9354-436d-9640-48b04669b235)

![k8s-metrics-dashboard-02](https://github.com/user-attachments/assets/8b6ff3d8-f168-4581-b68e-0baf21b5b986)
## Further Logging Analytics and APM Resources

Further resources for Logging Analytics can be found in folders [log-sources](log-sources/README.md), [logan-lookups](logan-lookups/README.md) and [dashboards](dashboards/README.md). We are providing there an enhanced log source definition to allow queries used in a dashboard with Business Analytics charts.

To maximize the usage value of APM, you can find in folder [apm-configs](apm-configs/README.md) all needed info to enrich the OTel spans with attributes matching the expected APM naming schema.
36 changes: 36 additions & 0 deletions knowledge-content/opentelemetry-demo/apm-configs/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,36 @@
# Enriching OTel Span Attributes to maximize OCI APM usage

Attribues of spans generated by OTel libraries and further processed later on by the OTel Collector are following a distinct naming schema. To get the best experience from APM for OTel based spans, we will use the APM feature "Span enrichment" to adjust and add span attribute names.

Prereqs:
- Have a terminal available on a system with OCI CLI configured, e.g. Cloud Shell
- Ensure the provided three JSON files from this folder are available in the terminal/shell
- Create a shell parameter APM_DOMAIN_ID containing the OCID of your APM Domain (available from APM/Administration):
APM_DOMAIN_ID=ocid1.apmdomain.oc1....

First, we will use file "otel_spans.json" to create a "Span filter" with name "OpenTelemetry Spans". By default we have assumed that you used "otel-demo-app" as namespace for the demo shop. If this is not the case you have to adjust "otel_spans.json" to your needs. When done run the following OCI CLI command:
```bash
oci apm-config config create-span-filter --from-json file://otel_spans.json --apm-domain-id $APM_DOMAIN_ID
```
From APM Administration, select your APM Domain and switch on the left to "Span filters", you should see now the following span filter:

![OTel-Span-Filter](https://github.com/user-attachments/assets/7908e14d-fa99-41d2-bd19-4c7b0d02e48d)

Click on the three dots at the right end of the line with the filter and copy its OCID to the clipboard.

Edit now JSON file "otel-span-options.json" at the top and replace the placeholder with the saved OCID:
```bash
{
"filterId": "ocid1.apmconfig.oc1........",
"disabled": false,
...
```
Finally, run the following command:
```bash
oci apm-config config create-options --from-json file://create-span-group.json --apm-domain-id $APM_DOMAIN_ID --options file://otel-span-options.json
```
In the OCI Console select now "Span enrichment" on the left, you should see the following:

![otel-spans-enrichment-group](https://github.com/user-attachments/assets/1f36d454-e0b0-411a-99c5-dc9882676d44)

We are all set :-)
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
{
"displayName": "OTel Spans Enrichment Group",
"description": "Rules to modify and enhance OTel spans to be in sync with APM schema",
"group": "data-manipulation.spans"
}
Original file line number Diff line number Diff line change
@@ -0,0 +1,252 @@
{
"filterId": "<replace with OpenTelemetry Span Filter OCID>",
"disabled": false,
"rules": [
{
"disabled": false,
"displayName": "Apply Oracle provided OTel rule",
"systemRuleId": 500
},
{
"actions": [
{
"config": {
"output": "WebApplicationName",
"value": "OTel Demo App"
},
"disabled": false,
"name": "basic.Set"
},
{
"config": {
"output": "ApmrumPageUpdateType",
"value": "Page Load"
},
"disabled": false,
"name": "basic.Set"
},
{
"config": {
"output": "ApmrumType",
"value": "Page"
},
"disabled": false,
"name": "basic.Set"
},
{
"config": {
"input": "Component",
"output": "OtelComponent"
},
"disabled": false,
"name": "basic.Rename"
},
{
"config": {
"output": "Component",
"value": "BROWSER"
},
"disabled": false,
"name": "basic.Set"
},
{
"config": {
"output": "PageViews",
"value": "1"
},
"disabled": false,
"name": "basic.Set"
},
{
"config": {
"input": "PageViews",
"output": "PageViews"
},
"disabled": false,
"name": "string.ToNumber"
},
{
"config": {
"input": "SpanDuration",
"output": "PageResponseTime",
"pattern": "(.*)",
"valueOnFail": "0"
},
"disabled": false,
"name": "string.Extract"
},
{
"config": {
"input": "PageResponseTime",
"output": "PageResponseTime"
},
"disabled": false,
"name": "string.ToNumber"
},
{
"config": {
"input": "ProcessRuntimeVersion",
"output": "UserName",
"pattern": "^.*Chrome/116.*$",
"replaceWith": "Synthetic_User"
},
"disabled": false,
"name": "string.Replace"
},
{
"config": {
"input": "HttpUserAgent",
"output": "UserName",
"pattern": "^.*HeadlessChrome.*$",
"replaceWith": "Loadgen_User"
},
"disabled": false,
"name": "string.Replace"
},
{
"config": {
"input": "ProcessRuntimeVersion",
"output": "UserName",
"pattern": "^.*Mac OS X.*$",
"replaceWith": "Demo_User"
},
"disabled": false,
"name": "string.Replace"
},
{
"config": {
"input": "ProcessRuntimeVersion",
"output": "UserName",
"pattern": "^.*iPhone.*$",
"replaceWith": "iOS_User"
},
"disabled": false,
"name": "string.Replace"
},
{
"config": {
"input": "ProcessRuntimeVersion",
"output": "UserName",
"pattern": "^.*Android.*$",
"replaceWith": "Android_User"
},
"disabled": false,
"name": "string.Replace"
}
],
"disabled": false,
"displayName": "Add attributes to browser agent spans",
"filterText": "ServiceName = 'frontend-web'"
},
{
"actions": [
{
"config": {
"input": "HttpUrl",
"output": "HttpUrl",
"pattern": "/_next/static.*$",
"replaceWith": "/_next/static"
},
"disabled": false,
"name": "string.Replace"
},
{
"config": {
"input": "HttpUrl",
"output": "HttpUrl",
"pattern": "/api/cart.*$",
"replaceWith": "/api/cart"
},
"disabled": false,
"name": "string.Replace"
},
{
"config": {
"input": "HttpUrl",
"output": "HttpUrl",
"pattern": "/api/recommendations.*$",
"replaceWith": "/api/recommendations"
},
"disabled": false,
"name": "string.Replace"
},
{
"config": {
"input": "HttpUrl",
"output": "HttpUrl",
"pattern": "/images.*$",
"replaceWith": "/images"
},
"disabled": false,
"name": "string.Replace"
},
{
"config": {
"input": "HttpUrl",
"output": "HttpUrl",
"pattern": "/api/data.*$",
"replaceWith": "/api/data"
},
"disabled": false,
"name": "string.Replace"
},
{
"config": {
"input": "HttpUrl",
"output": "HttpUrl",
"pattern": "/api/products.*$",
"replaceWith": "/api/products"
},
"disabled": false,
"name": "string.Replace"
},
{
"config": {
"input": "HttpUrl",
"output": "HttpUrl",
"pattern": "/icons.*$",
"replaceWith": "/icons"
},
"disabled": false,
"name": "string.Replace"
}
],
"disabled": false,
"displayName": "Create short unique HttpUrl patterns"
},
{
"actions": [
{
"config": {
"input": "K8sDeploymentName",
"output": "DisplayName",
"pattern": "otel-demo-app-(.*$)"
},
"disabled": false,
"name": "string.Extract"
}
],
"disabled": false,
"displayName": "Add attributes to application services spans",
"filterText": "ServiceName like '%service' or ServiceName = 'imageprovider'"
},
{
"actions": [
{
"config": {
"input": "ProcessRuntimeVersion",
"output": "UserAgent",
"pattern": "(.*)"
},
"disabled": false,
"name": "string.Extract"
}
],
"description": "Recognize load generation scripts to capture userAgent for load data.",
"disabled": false,
"displayName": "Mark browser",
"filterText": "ProcessRuntimeName = 'browser'"
}
],
"type": "data-manipulation.group"
}
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
{
"displayName": "OpenTelemetry Spans",
"description": "All spans coming from OTel Collector",
"filterText": "\"k8s.namespace.name\" = 'otel-demo-app'"
}
Loading