diff --git a/examples/ecs-ec2-windows/README.md b/examples/ecs-ec2-windows/README.md new file mode 100644 index 00000000..04754c88 --- /dev/null +++ b/examples/ecs-ec2-windows/README.md @@ -0,0 +1,27 @@ +# Demonstration/Example deployment of ECS EC2 Open Telemetry Agent for Windows + +## Usage + +To run this example you need to save this code in Terraform file, and change the values according to your settings. + +For parameter details, see [ECS EC2 Windows demo module README](../../modules/ecs-ec2-windows/README.md) + +```hcl +module "ecs_ec2_windows_demo" { + source = "../../modules/ecs-ec2-windows" + ecs_cluster_name = var.ecs_cluster_name + coralogix_region = "Singapore" + api_key = var.api_key + security_group_id = var.security_group_id + subnet_ids = var.subnet_ids +} +``` + +now execute: +```bash +$ terraform init +$ terraform plan +$ terraform apply +``` + +Run `terraform destroy` when you don't need these resources. diff --git a/examples/ecs-ec2-windows/ecs-ec2-windows.tf b/examples/ecs-ec2-windows/ecs-ec2-windows.tf new file mode 100644 index 00000000..2f642350 --- /dev/null +++ b/examples/ecs-ec2-windows/ecs-ec2-windows.tf @@ -0,0 +1,25 @@ +variable "api_key" { + type = string + sensitive = true +} + +variable "ecs_cluster_name" { + type = string +} + +variable "security_group_id" { + type = string +} + +variable "subnet_ids" { + type = list +} + +module "ecs_ec2_windows_demo" { + source = "../../modules/ecs-ec2-windows" + ecs_cluster_name = var.ecs_cluster_name + coralogix_region = "Singapore" + api_key = var.api_key + security_group_id = var.security_group_id + subnet_ids = var.subnet_ids +} diff --git a/modules/ecs-ec2-windows/README.md b/modules/ecs-ec2-windows/README.md new file mode 100644 index 00000000..60413113 --- /dev/null +++ b/modules/ecs-ec2-windows/README.md @@ -0,0 +1,83 @@ +# Demonstration/Example deployment of ECS EC2 Open Telemetry Agent for Windows + +Terraform module to launch an example ECS Service illustrating the deployment of a demo Windows container application alongside an [Coralogix Opentelemetry Collector](https://hub.docker.com/r/coralogixrepo/coralogix-otel-collector) running in a sidecar Windows container to collect logs and metrics from applications within the ECS Task, and forwarding to Coralogix. The defatult sample application logs to STDOUT every second, and you can replace this example your own container application image. This example requires deployment into existing AWS ECS Cluster on EC2 Windows container instances. + +This example is intended for demonstration and instructional purposes only. It should not be deployed directly into production. The example may be customized to suit the user's requirements. + +## Usage + +Provisions a demonstration ECS Service having 1 Task consisting of 1 example Windows application and 1 Windows OTEL Collector as a sidecar. + +```hcl +module "ecs_ec2_windows_demo" { + source = "../../modules/ecs-ec2-windows" + ecs_cluster_name = "ecs-cluster-name" + coralogix_region = "Europe"|"Europe2"|"India"|"Singapore"|"US"|"US2" + api_key = var.api_key + security_group_id = var.security_group_id + subnet_ids = var.subnet_ids + application_name = "Coralogix Application Name" + subsystem_name = "Coralogix Subsystem Name" + custom_domain = "[optional] custom Coralogix domain" + app_image = "[optional] User-provided demo App as a Windows container image, to demonstrate collection of console logs and metrics. If omitted, defaults to a provided sample Windows logging app." + otel_image = "[optional] Coralogix Open Telemetry distribution Windows image name and tag." + otel_config_file = "[optional] file path to custom OTEL collector config file" +} +``` + +#### Windows Sample Application +The default Windows sample application logs the following text every 1 second: +> "Hello from console writer. __N__" + +Where __N__ is an incrementing number counter. + +#### Verification +To verify successful deployment: +* Verify the logs are captured on your Coralogix logs console. +* Verify the metrics for the containers can be displayed on Coralogix Grafana. + +#### Integrating the OTEL Collector into your application. + +For logging to work, a Windows application should do either of the following: +1. Log to STDOUT. The OTEL Collector sidecar has been configured to collect docker container logs from the host. +2. Mount the log file volume to the OTEL Collector sidecar, and include the log location into the OTEL [filelog receiver configuration](./otel_ecs_ec2_win.config.yaml). +3. Instrument your application for OTEL, configured to export to the sidecar OTLP endpoints, which for 'awsvpc' networking, are at http://localhost:4317 and http://localhost:4318 + +## Requirements + +| Name | Version | +|------|---------| +| terraform | ~> 1.6.0 | +| aws | >= 5.0.0 | + +## Providers + +| Name | Version | +|------|---------| +| aws | >= 5.0.0 | + +## Resources + +| Name | Type | +|------|------| +| [aws_ecs_service.demo_service](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/ecs_service) | resource | +| [aws_ecs_task_definition.demo_task_definition](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/ecs_task_definition) | resource | +| [aws_iam_role.ecsTaskExecutionRole](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/iam_role) | resource | +| [aws_iam_role_policy.ecs_awslogs_policy](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/iam_role_policy) | resource | +| [aws_region.current](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/data-sources/region) | data source | + +## Inputs + +| Name | Description | Type | Default | Required | +|------|-------------|------|---------|:--------:| +| api\_key | The Send-Your-Data API key for your Coralogix account. See: https://coralogix.com/docs/send-your-data-api-key/ | `string` | n/a | yes | +| app\_image | Optional user-provided demo App as a Windows container image, to demonstrate collection of console logs and metrics. If omitted, defaults to a provided sample Windows logging app. | `string` | `""` | no | +| application\_name | Optional Application name as Coralogix metadata. | `string` | `"ECS-Windows-Demo"` | no | +| coralogix\_region | The region of the Coralogix endpoint domain: [Europe, Europe2, India, Singapore, US, US2, Custom]. If "Custom" then __custom\_domain__ parameter must be specified. | `string` | n/a | yes | +| custom\_domain | Optional Coralogix custom domain, e.g. "private.coralogix.com" Private Link domain. If specified, overrides the public domain corresponding to the __coralogix\_region__ parameter. | `string` | `null` | no | +| ecs\_cluster\_name | Name of the AWS ECS Cluster to deploy the demonstration ECS Service, consisting of 1 Coralogix OTEL Collector and 1 sample app as Windows containers in the task. Supports EC2 Windows instances only, not Fargate. | `string` | n/a | yes | +| otel\_config\_file | Optional file path to a custom opentelemetry configuration file. Defaults to an embedded configuration. | `string` | `null` | no | +| otel\_image | Optional Coralogix Open Telemetry distribution Windows image name and tag. | `string` | `"coralogixrepo/coralogix-otel-collector:0.1.0-windowsserver-1809"` | no | +| security\_group\_id | Security Group ID to deploy the ECS Service into. Must be in the same VPC as the ECS Cluster. | `string` | n/a | yes | +| subnet\_ids | List of subnet IDs to deploy the ECS Service into. Must be in the same VPC as the ECS Cluster. | `list` | n/a | yes | +| subsystem\_name | Optional Subsystem name as Coralogix metadata. | `string` | `"ECS-Windows-Demo"` | no | diff --git a/modules/ecs-ec2-windows/container_definitions.tftpl.json b/modules/ecs-ec2-windows/container_definitions.tftpl.json new file mode 100644 index 00000000..80fad053 --- /dev/null +++ b/modules/ecs-ec2-windows/container_definitions.tftpl.json @@ -0,0 +1,114 @@ +[ + %{ if app_image != "" } + { + "name" : "custom_demo_app", + "image" : "${app_image}", + "cpu" : 512, + "memory" : 1024, + "essential" : true, + "portMappings" : [ + { + "containerPort" : 80, + "hostPort" : 80 + } + ], + "dependsOn": [ + { + "containerName": "coralogix-otel-agent", + "condition": "HEALTHY" + } + ] + }, + %{else} + { + "name" : "demo_app", + "image" : "mcr.microsoft.com/windows/servercore:ltsc2019", + "cpu" : 512, + "memory" : 1024, + "essential" : true, + "entryPoint" : [ + "Powershell", + "-Command" + ], + "command" : [ + "$count=1;while(1) { Write-Host \"Hello from console writer.\" $count; sleep 1; $count=$count+1;}" + ], + "dependsOn": [ + { + "containerName": "coralogix-otel-agent", + "condition": "HEALTHY" + } + ] + }, + %{ endif } + { + "name" : "coralogix-otel-agent", + "image" : "${otel_image}", + "essential" : true, + "cpu" : 512, + "memory" : 1024, + "mountPoints" : [ + { + "sourceVolume" : "hostfs", + "containerPath" : "C:\\hostfs", + "readOnly" : true + } + ], + "portMappings" : [ + { + "containerPort" : 4317, + "hostPort" : 4317 + }, + { + "containerPort" : 4318, + "hostPort" : 4318 + }, + { + "containerPort" : 8888, + "hostPort" : 8888 + }, + { + "containerPort" : 13133, + "hostPort" : 13133 + } + ], + "environment" : [ + { + "name" : "CORALOGIX_DOMAIN", + "value" : "${coralogix_domain}" + }, + { + "name" : "PRIVATE_KEY", + "value" : "${api_key}" + }, + { + "name" : "APP_NAME", + "value" : "${application_name}" + }, + { + "name" : "SUB_SYS", + "value" : "${subsystem_name}" + }, + { + "name" : "OTEL_CONFIG", + "value" : ${jsonencode(otel_config)} + } + ], + "healthCheck" : { + "command" : ["CMD-SHELL", "curl localhost:13133"], + "startPeriod" : 60, + "interval" : 30, + "timeout" : 5, + "retries" : 5 + }, + "logConfiguration" : { + "logDriver" : "awslogs", + "options" : { + "awslogs-group" : "/ecs/coralogix-otel-agent", + "awslogs-region" : "${region}", + "awslogs-stream-prefix" : "coralogix-otel-agent", + "awslogs-create-group" : "true" + } + } + } +] \ No newline at end of file diff --git a/modules/ecs-ec2-windows/main.tf b/modules/ecs-ec2-windows/main.tf new file mode 100644 index 00000000..56a77634 --- /dev/null +++ b/modules/ecs-ec2-windows/main.tf @@ -0,0 +1,112 @@ +data "aws_region" "current" {} + +locals { + name = "coralogix-monitor-demo" + coralogix_region_domain_map = { + "europe" = "coralogix.com" + "europe2" = "eu2.coralogix.com" + "india" = "coralogix.in" + "singapore" = "coralogixsg.com" + "us" = "coralogix.us" + "us2" = "cx498.coralogix.com" + "custom" = null + } + coralogix_domain = coalesce(var.custom_domain, local.coralogix_region_domain_map[lower(var.coralogix_region)]) + otel_config_file_path = coalesce(var.otel_config_file, "${path.module}/otel_ecs_ec2_win.config.yaml") + otel_config = templatefile(local.otel_config_file_path, {}) +} + +resource "aws_iam_role" "ecsTaskExecutionRole" { + name = "${local.name}-ecs-task-execution-role" + path = "/" + managed_policy_arns = [ + "arn:aws:iam::aws:policy/service-role/AmazonECSTaskExecutionRolePolicy" + ] + assume_role_policy = jsonencode({ + Version = "2012-10-17" + Statement = [ + { + Action = "sts:AssumeRole" + Sid = "" + Effect = "Allow" + Principal = { + Service = "ecs-tasks.amazonaws.com" + } + }, + ] + }) +} + +resource "aws_iam_role_policy" "ecs_awslogs_policy" { + name = "${local.name}-ecs-awslogs-policy" + role = aws_iam_role.ecsTaskExecutionRole.id + policy = jsonencode({ + Version = "2012-10-17" + Statement = [ + { + Effect = "Allow", + Action = [ + "logs:CreateLogStream", + "logs:CreateLogGroup", + "logs:DescribeLogStreams", + "logs:PutLogEvents" + ], + Resource = "arn:aws:logs:*:*:log-group:/ecs/*" + }, + ] + }) +} + +resource "aws_ecs_task_definition" "demo_task_definition" { + family = "${local.name}-ec2-windows" + execution_role_arn = aws_iam_role.ecsTaskExecutionRole.arn + task_role_arn = aws_iam_role.ecsTaskExecutionRole.arn + cpu = 1024 + memory = 2048 + requires_compatibilities = ["EC2"] + network_mode = "awsvpc" + volume { + name = "hostfs" + host_path = "C:\\" + } + tags = { + "ecs:taskDefinition:createdFrom" = "terraform" + } + container_definitions = templatefile("${path.module}/container_definitions.tftpl.json",{ + region = data.aws_region.current.name + otel_image = var.otel_image + app_image = var.app_image + coralogix_domain = local.coralogix_domain, + application_name = var.application_name, + subsystem_name = var.subsystem_name, + api_key = var.api_key + otel_config = local.otel_config + } + ) +} + +resource "aws_ecs_service" "demo_service" { + name = "${local.name}-service" + cluster = var.ecs_cluster_name + launch_type = "EC2" + task_definition = aws_ecs_task_definition.demo_task_definition.arn + scheduling_strategy = "REPLICA" + deployment_maximum_percent = 100 + deployment_minimum_healthy_percent = 0 + desired_count = 1 + enable_ecs_managed_tags = true + deployment_circuit_breaker { + enable = true + rollback = true + } + deployment_controller { + type = "ECS" + } + service_connect_configuration { + enabled = false + } + network_configuration { + subnets = var.subnet_ids + security_groups = [var.security_group_id] + } +} diff --git a/modules/ecs-ec2-windows/otel_ecs_ec2_win.config.yaml b/modules/ecs-ec2-windows/otel_ecs_ec2_win.config.yaml new file mode 100644 index 00000000..e9ccbd01 --- /dev/null +++ b/modules/ecs-ec2-windows/otel_ecs_ec2_win.config.yaml @@ -0,0 +1,94 @@ +receivers: + + awsecscontainermetricsd: + sidecar: true + + otlp: + protocols: + grpc: + endpoint: 0.0.0.0:4317 + http: + endpoint: 0.0.0.0:4318 + + prometheus: + config: + scrape_configs: + - job_name: otel-collector-metrics + scrape_interval: 60s + static_configs: + - targets: ['localhost:8888'] + + filelog: + start_at: end + include: + - C:\hostfs\ProgramData\docker\containers\*\*-json.log + include_file_path: true + operators: + - type: move + from: attributes["log.file.name"] + to: resource["log.file.name"] + +processors: + batch: + +exporters: + + logging: + verbosity: normal + + coralogix: + domain: "$CORALOGIX_DOMAIN" + private_key: "$PRIVATE_KEY" + application_name: "$APP_NAME" + subsystem_name: "$SUB_SYS" + application_name_attributes: + - "$APP_NAME" + - "aws.ecs.cluster" + - "aws.ecs.cluster.name" + - "aws.ecs.task.definition.family" + - "container.name" + subsystem_name_attributes: + - "$SUB_SYS" + - "aws.ecs.task.family" + - "aws.ecs.container.name" + - "aws.ecs.docker.name" + timeout: 30s + +extensions: + health_check: + +service: + + extensions: + - health_check + + pipelines: + + metrics: + receivers: + - otlp + - awsecscontainermetricsd + processors: + - batch + exporters: + - coralogix + - logging + + logs: + receivers: + - otlp + - filelog + processors: + - batch + exporters: + - coralogix + - logging + + traces: + receivers: + - otlp + processors: + - batch + exporters: + - coralogix + - logging diff --git a/modules/ecs-ec2-windows/variables.tf b/modules/ecs-ec2-windows/variables.tf new file mode 100644 index 00000000..758f05dd --- /dev/null +++ b/modules/ecs-ec2-windows/variables.tf @@ -0,0 +1,73 @@ +variable "ecs_cluster_name" { + description = "Name of the AWS ECS Cluster to deploy the demonstration ECS Service, consisting of 1 Coralogix OTEL Collector and 1 sample app as Windows containers in the task. Supports EC2 Windows instances only, not Fargate." + type = string +} + +variable "subnet_ids" { + description = "List of subnet IDs to deploy the ECS Service into. Must be in the same VPC as the ECS Cluster." + type = list +} + +variable "security_group_id" { + description = "Security Group ID to deploy the ECS Service into. Must be in the same VPC as the ECS Cluster." + type = string +} + +variable "coralogix_region" { + description = "The region of the Coralogix endpoint domain: [Europe, Europe2, India, Singapore, US, US2, Custom]. If \"Custom\" then __custom_domain__ parameter must be specified." + type = string + validation { + condition = can(regex("^(europe|europe2|india|singapore|us|us2|custom)$", lower(var.coralogix_region))) + error_message = "Must be one of [Europe, Europe2, India, Singapore, US, US2, Custom]" + } +} + +variable "custom_domain" { + description = "Optional Coralogix custom domain, e.g. \"private.coralogix.com\" Private Link domain. If specified, overrides the public domain corresponding to the __coralogix_region__ parameter." + type = string + default = null +} + +variable "application_name" { + description = "Optional Application name as Coralogix metadata." + type = string + default = "ECS-Windows-Demo" + validation { + condition = length(var.application_name) >= 1 && length(var.application_name) <= 64 + error_message = "The Default Application Name length should be within 1 and 64 characters" + } +} + +variable "subsystem_name" { + description = "Optional Subsystem name as Coralogix metadata." + type = string + default = "ECS-Windows-Demo" + validation { + condition = length(var.subsystem_name) >= 1 && length(var.subsystem_name) <= 64 + error_message = "The Default Subsystem Name length should be within 1 and 64 characters" + } +} + +variable "otel_image" { + description = "Optional Coralogix Open Telemetry distribution Windows image name and tag." + type = string + default = "coralogixrepo/coralogix-otel-collector:0.1.0-windowsserver-1809" +} + +variable "app_image" { + description = "Optional user-provided demo App as a Windows container image, to demonstrate collection of console logs and metrics. If omitted, defaults to a provided sample Windows logging app." + type = string + default = "" +} + +variable "api_key" { + description = "The Send-Your-Data API key for your Coralogix account. See: https://coralogix.com/docs/send-your-data-api-key/" + type = string + sensitive = true +} + +variable "otel_config_file" { + type = string + description = "Optional file path to a custom opentelemetry configuration file. Defaults to an embedded configuration." + default = null +} diff --git a/modules/ecs-ec2-windows/versions.tf b/modules/ecs-ec2-windows/versions.tf new file mode 100644 index 00000000..05783e3c --- /dev/null +++ b/modules/ecs-ec2-windows/versions.tf @@ -0,0 +1,9 @@ +terraform { + required_version = "~> 1.6.0" + required_providers { + aws = { + source = "hashicorp/aws" + version = ">= 5.0.0" + } + } +} diff --git a/modules/ecs-ec2/README.md b/modules/ecs-ec2/README.md index fb0555f9..bb21c86b 100644 --- a/modules/ecs-ec2/README.md +++ b/modules/ecs-ec2/README.md @@ -1,10 +1,10 @@ -# ECS EC2 Open Telemetry Agent +# ECS EC2 Open Telemetry Agent for Linux -Terraform module to launch Opentelemetry Collector agents on an existing ECS Cluster on EC2 container instances. An ECS Service runs the [Coralogix Opentelemetry Collector](https://hub.docker.com/r/coralogixrepo/coralogix-otel-collector) image as a Daemon task on each active container instance. +Terraform module to launch Opentelemetry Collector agents on an existing AWS ECS Cluster on EC2 Linux container instances. An ECS Service runs the [Coralogix Opentelemetry Collector](https://hub.docker.com/r/coralogixrepo/coralogix-otel-collector) image as a Daemon task on each active container instance. ## Usage -Provision an ECS Service that run the OTEL Collector Agent as a Daemon container on each EC2 container instance. +Provision an ECS Service that run the OTEL Collector Agent as a Daemon container on each Linux EC2 container instance. ```terraform module "ecs-ec2" { diff --git a/tests/ecs-ec2-windows/ecs-ec2-windows.tf b/tests/ecs-ec2-windows/ecs-ec2-windows.tf new file mode 100644 index 00000000..f59e8e43 --- /dev/null +++ b/tests/ecs-ec2-windows/ecs-ec2-windows.tf @@ -0,0 +1,40 @@ +terraform { + required_providers { + aws = { + source = "hashicorp/aws" + version = "~> 5.0" + } + } +} + +variable "api_key" { + type = string + sensitive = true +} + +variable "ecs_cluster_name" { + type = string +} + +variable "security_group_id" { + type = string +} + +variable "subnet_ids" { + type = list +} + +variable "app_image" { + type = string + default = "" +} + +module "ecs_ec2_windows_demo" { + source = "../../modules/ecs-ec2-windows" + ecs_cluster_name = var.ecs_cluster_name + coralogix_region = "Singapore" + api_key = var.api_key + security_group_id = var.security_group_id + subnet_ids = var.subnet_ids + app_image = var.app_image +}