Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
68 changes: 68 additions & 0 deletions typescript/airflow-lambda-dynamodb-approval/.gitignore
Original file line number Diff line number Diff line change
@@ -0,0 +1,68 @@
*.js
!jest.config.js
*.d.ts
node_modules

# CDK asset staging directory
.cdk.staging
cdk.out

# Python
__pycache__/
*.py[cod]
*$py.class
*.so
.Python
build/
develop-eggs/
dist/
downloads/
eggs/
.eggs/
lib/
!lib/*.ts
lib64/
parts/
sdist/
var/
wheels/
*.egg-info/
.installed.cfg
*.egg
MANIFEST

# PyInstaller
*.manifest
*.spec

# Unit test / coverage reports
htmlcov/
.tox/
.coverage
.coverage.*
.cache
nosetests.xml
coverage.xml
*.cover
.hypothesis/
.pytest_cache/

# Virtual environments
.env
.venv
env/
venv/
ENV/
env.bak/
venv.bak/

# IDE
.vscode/
.idea/
*.swp
*.swo
*~

# OS
.DS_Store
Thumbs.db
6 changes: 6 additions & 0 deletions typescript/airflow-lambda-dynamodb-approval/.npmignore
Original file line number Diff line number Diff line change
@@ -0,0 +1,6 @@
*.ts
!*.d.ts

# CDK asset staging directory
.cdk.staging
cdk.out
123 changes: 123 additions & 0 deletions typescript/airflow-lambda-dynamodb-approval/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,123 @@
<!--BEGIN STABILITY BANNER-->
---

![Stability: Stable](https://img.shields.io/badge/stability-Stable-success.svg?style=for-the-badge)

> **This is a stable example.** It should successfully build out of the box

---
<!--END STABILITY BANNER-->

# AWS MWAA (Managed Workflows for Apache Airflow) with CDK

This sample demonstrates how to deploy AWS Managed Workflows for Apache Airflow (MWAA) using AWS CDK with example DAGs for basic workflows, Lambda integration, and human approval processes.

## Overview

This CDK project creates:
- **VPC** with public/private subnets and NAT Gateway
- **S3 bucket** for DAG storage with automatic deployment
- **MWAA environment** with proper IAM roles and security groups
- **DynamoDB table** for human approval workflows
- **Lambda function** for integration testing
- **Three example DAGs** demonstrating different patterns

### DAG Workflows

```mermaid
graph TD
subgraph "example_dag"
A1[Start] --> A2[Print Hello]
A2 --> A3[Print Goodbye]
A3 --> A4[End]
end

subgraph "lambda_invoke_dag"
B1[List Lambda Functions] --> B2[Invoke Demo Lambda]
B2 --> B3[Process Response]
end

subgraph "ddb_approval_dag"
C1[Create Approval Request] --> C5[(DynamoDB Table)]
C1 --> C2[Wait for Human Approval]
C5 --> C2
C2 --> C3[Process Transaction]
C3 --> C4[Complete]

C6[Manual Approval via AWS Console] --> C5
end
```

**Example DAGs included:**
1. **`example_dag.py`** - Basic Airflow workflow with simple tasks
2. **`lambda_invoke_dag.py`** - Demonstrates Lambda function invocation from Airflow
3. **`ddb_approval_dag.py`** - Human approval workflow using DynamoDB sensors

## Build

```bash
npm install
npm run build
```

## Deploy

```bash
cdk deploy
```

**Note the outputs** after deployment:
- **MwaaWebServerUrl** - Access the Airflow web interface
- **S3BucketName** - Where your DAGs are stored
- **ApprovalTableName** - DynamoDB table for approval workflows

## Usage

### Access Airflow Web UI
1. **Login to AWS Console first** - Ensure you're logged into the AWS Console in your browser
2. **Use the deployment output URL** - Copy the `MwaaWebServerUrl` from the deployment outputs
3. **Access Airflow** - Open the URL in the same browser where you're logged into AWS Console

**Note:** MWAA requires AWS authentication even with `PUBLIC_ONLY` access mode. You must be logged into the AWS Console to access the Airflow web interface.

### Test Basic Workflow
1. Trigger the `example_dag` DAG from the Airflow UI
2. Watch it execute the simple "Hello" and "Goodbye" tasks
3. Verify successful completion in the DAG run logs

### Test Lambda Integration
1. Trigger the `lambda_invoke_example` DAG from the Airflow UI
2. View logs to see Lambda function listing and invocation results

### Test Human Approval Workflow
1. Trigger the `dynamodb_human_approval_pipeline` DAG from the Airflow UI
2. Go to AWS Console → DynamoDB → Tables → `mwaa-approval-table-{region}`
3. Find your process record and change `approval_status` from `PENDING` to `APPROVED`
4. Watch the workflow complete automatically

## Clean up

```bash
cdk destroy
```

All resources are configured with `RemovalPolicy.DESTROY` for easy cleanup.

## Architecture

- **Environment Class**: `mw1.small` (1-2 workers)
- **Airflow Version**: 2.7.2
- **Web Access**: Public (configure private access for production)
- **Logging**: All log types enabled at INFO level

## Security Notes

This sample uses demo-friendly settings. For production:

- **Web Access**: Change from `PUBLIC_ONLY` to `PRIVATE_ONLY` for MWAA environment
- **IAM Permissions**: Replace wildcard (`'*'`) permissions with specific resource ARNs
- **S3 Encryption**: Enable server-side encryption for the DAGs bucket
- **VPC Endpoints**: Add endpoints for S3, DynamoDB, and Lambda to avoid internet traffic
- **Resource Policies**: Use `RETAIN` instead of `DESTROY` for production resources
- **DynamoDB**: Enable point-in-time recovery (currently disabled)
- **Monitoring**: Enable CloudTrail and CloudWatch alarms for security monitoring
Original file line number Diff line number Diff line change
@@ -0,0 +1,6 @@
#!/usr/bin/env node
import * as cdk from 'aws-cdk-lib';
import { AwsMwaaCdkStack } from '../lib/aws-mwaa-cdk-stack';

const app = new cdk.App();
new AwsMwaaCdkStack(app, 'AwsMwaaCdkStack');
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Where is this file?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@kaiz-io Fixed. Added lib/aws-mwaa-cdk-stack.ts file in commit 267dabc.

99 changes: 99 additions & 0 deletions typescript/airflow-lambda-dynamodb-approval/cdk.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,99 @@
{
"app": "npx ts-node --prefer-ts-exts bin/aws-mwaa-cdk.ts",
"watch": {
"include": [
"**"
],
"exclude": [
"README.md",
"cdk*.json",
"**/*.d.ts",
"**/*.js",
"tsconfig.json",
"package*.json",
"yarn.lock",
"node_modules",
"test"
]
},
"context": {
"@aws-cdk/aws-lambda:recognizeLayerVersion": true,
"@aws-cdk/core:checkSecretUsage": true,
"@aws-cdk/core:target-partitions": [
"aws",
"aws-cn"
],
"@aws-cdk-containers/ecs-service-extensions:enableDefaultLogDriver": true,
"@aws-cdk/aws-ec2:uniqueImdsv2TemplateName": true,
"@aws-cdk/aws-ecs:arnFormatIncludesClusterName": true,
"@aws-cdk/aws-iam:minimizePolicies": true,
"@aws-cdk/core:validateSnapshotRemovalPolicy": true,
"@aws-cdk/aws-codepipeline:crossAccountKeyAliasStackSafeResourceName": true,
"@aws-cdk/aws-s3:createDefaultLoggingPolicy": true,
"@aws-cdk/aws-sns-subscriptions:restrictSqsDescryption": true,
"@aws-cdk/aws-apigateway:disableCloudWatchRole": true,
"@aws-cdk/core:enablePartitionLiterals": true,
"@aws-cdk/aws-events:eventsTargetQueueSameAccount": true,
"@aws-cdk/aws-ecs:disableExplicitDeploymentControllerForCircuitBreaker": true,
"@aws-cdk/aws-iam:importedRoleStackSafeDefaultPolicyName": true,
"@aws-cdk/aws-s3:serverAccessLogsUseBucketPolicy": true,
"@aws-cdk/aws-route53-patters:useCertificate": true,
"@aws-cdk/customresources:installLatestAwsSdkDefault": false,
"@aws-cdk/aws-rds:databaseProxyUniqueResourceName": true,
"@aws-cdk/aws-codedeploy:removeAlarmsFromDeploymentGroup": true,
"@aws-cdk/aws-apigateway:authorizerChangeDeploymentLogicalId": true,
"@aws-cdk/aws-ec2:launchTemplateDefaultUserData": true,
"@aws-cdk/aws-secretsmanager:useAttachedSecretResourcePolicyForSecretTargetAttachments": true,
"@aws-cdk/aws-redshift:columnId": true,
"@aws-cdk/aws-stepfunctions-tasks:enableEmrServicePolicyV2": true,
"@aws-cdk/aws-ec2:restrictDefaultSecurityGroup": true,
"@aws-cdk/aws-apigateway:requestValidatorUniqueId": true,
"@aws-cdk/aws-kms:aliasNameRef": true,
"@aws-cdk/aws-kms:applyImportedAliasPermissionsToPrincipal": true,
"@aws-cdk/aws-autoscaling:generateLaunchTemplateInsteadOfLaunchConfig": true,
"@aws-cdk/core:includePrefixInUniqueNameGeneration": true,
"@aws-cdk/aws-efs:denyAnonymousAccess": true,
"@aws-cdk/aws-opensearchservice:enableOpensearchMultiAzWithStandby": true,
"@aws-cdk/aws-lambda-nodejs:useLatestRuntimeVersion": true,
"@aws-cdk/aws-efs:mountTargetOrderInsensitiveLogicalId": true,
"@aws-cdk/aws-rds:auroraClusterChangeScopeOfInstanceParameterGroupWithEachParameters": true,
"@aws-cdk/aws-appsync:useArnForSourceApiAssociationIdentifier": true,
"@aws-cdk/aws-rds:preventRenderingDeprecatedCredentials": true,
"@aws-cdk/aws-codepipeline-actions:useNewDefaultBranchForCodeCommitSource": true,
"@aws-cdk/aws-cloudwatch-actions:changeLambdaPermissionLogicalIdForLambdaAction": true,
"@aws-cdk/aws-codepipeline:crossAccountKeysDefaultValueToFalse": true,
"@aws-cdk/aws-codepipeline:defaultPipelineTypeToV2": true,
"@aws-cdk/aws-kms:reduceCrossAccountRegionPolicyScope": true,
"@aws-cdk/aws-eks:nodegroupNameAttribute": true,
"@aws-cdk/aws-ec2:ebsDefaultGp3Volume": true,
"@aws-cdk/aws-ecs:removeDefaultDeploymentAlarm": true,
"@aws-cdk/custom-resources:logApiResponseDataPropertyTrueDefault": false,
"@aws-cdk/aws-s3:keepNotificationInImportedBucket": false,
"@aws-cdk/core:explicitStackTags": true,
"@aws-cdk/aws-ecs:enableImdsBlockingDeprecatedFeature": false,
"@aws-cdk/aws-ecs:disableEcsImdsBlocking": true,
"@aws-cdk/aws-ecs:reduceEc2FargateCloudWatchPermissions": true,
"@aws-cdk/aws-dynamodb:resourcePolicyPerReplica": true,
"@aws-cdk/aws-ec2:ec2SumTImeoutEnabled": true,
"@aws-cdk/aws-appsync:appSyncGraphQLAPIScopeLambdaPermission": true,
"@aws-cdk/aws-rds:setCorrectValueForDatabaseInstanceReadReplicaInstanceResourceId": true,
"@aws-cdk/core:cfnIncludeRejectComplexResourceUpdateCreatePolicyIntrinsics": true,
"@aws-cdk/aws-lambda-nodejs:sdkV3ExcludeSmithyPackages": true,
"@aws-cdk/aws-stepfunctions-tasks:fixRunEcsTaskPolicy": true,
"@aws-cdk/aws-ec2:bastionHostUseAmazonLinux2023ByDefault": true,
"@aws-cdk/aws-route53-targets:userPoolDomainNameMethodWithoutCustomResource": true,
"@aws-cdk/aws-elasticloadbalancingV2:albDualstackWithoutPublicIpv4SecurityGroupRulesDefault": true,
"@aws-cdk/aws-iam:oidcRejectUnauthorizedConnections": true,
"@aws-cdk/core:enableAdditionalMetadataCollection": true,
"@aws-cdk/aws-lambda:createNewPoliciesWithAddToRolePolicy": false,
"@aws-cdk/aws-s3:setUniqueReplicationRoleName": true,
"@aws-cdk/aws-events:requireEventBusPolicySid": true,
"@aws-cdk/core:aspectPrioritiesMutating": true,
"@aws-cdk/aws-dynamodb:retainTableReplica": true,
"@aws-cdk/aws-stepfunctions:useDistributedMapResultWriterV2": true,
"@aws-cdk/s3-notifications:addS3TrustKeyPolicyForSnsSubscriptions": true,
"@aws-cdk/aws-ec2:requirePrivateSubnetsForEgressOnlyInternetGateway": true,
"@aws-cdk/aws-s3:publicAccessBlockedByDefault": true,
"@aws-cdk/aws-lambda:useCdkManagedLogGroup": true
}
}
Loading