Skip to content

Commit b8d0fd9

Browse files
moryachokMichael Greenshtein
andauthored
v0.1.0-release
* readme changed * event-bridge rule script updated --------- Co-authored-by: Michael Greenshtein <greensht@amazon.com>
1 parent bf35689 commit b8d0fd9

File tree

1 file changed

+10
-4
lines changed

1 file changed

+10
-4
lines changed

README.md

Lines changed: 10 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -1,8 +1,9 @@
11
## Monitoring Apache Iceberg Table metadata layer using AWS Lambda, AWS Glue and AWS CloudWatch
22

3-
This repository provides you with sample code that collects metrics of existing Apache Iceberg tables managed in your Amazon S3 and catalogued to AWS Glue Data Catalog. The code consists of AWS Lambda deployment package that collects and submits metrics into AWS CloudWatch. Repository also includes helper script for deploying CloudWatch monitoring dashboard to visualize collected metrics.
3+
This repository provides you with a sample solution that collects metrics of existing Apache Iceberg tables managed in your Amazon S3 and catalogued to AWS Glue Data Catalog. The solution consists of AWS Lambda deployment package that collects and submits metrics into AWS CloudWatch. Repository also includes helper script for deploying CloudWatch monitoring dashboard to visualize collected metrics.
44

55
### Table of Contents
6+
- [Solution Tenets](#solution-tenets)
67
- [Technical implementation](#technical-implementation)
78
- [Metrics collected](#metrics-collected)
89
- [Setup](#setup)
@@ -14,7 +15,10 @@ This repository provides you with sample code that collects metrics of existing
1415
- [Security](#security)
1516
- [License](#license)
1617

17-
18+
### Solution Tenets
19+
* Solution is designed to provide time-series metrics for Apache Iceberg to monitor Apache Iceberg tables over-time to recognize trends and anomalies.
20+
* Solution is designed to be lightweight and collect metrics exclusively from Apache Iceberg metadata layer without scanning the data layer hense without the need for heavy compute capacity.
21+
* In the future we strive to reduce the dependency on AWS Glue in favor of using AWS Lambda compute when required features are available in [PyIceberg](https://py.iceberg.apache.org) library.
1822

1923
### Technical implementation
2024

@@ -126,8 +130,9 @@ import boto3
126130
import json
127131

128132
# Initialize a boto3 client
129-
lambda_client = boto3.client('lambda')
130-
events_client = boto3.client('events')
133+
session = boto3.Session(region_name='<<SET CORRECT AWS REGION>>')
134+
lambda_client = session.client('lambda')
135+
events_client = session.client('events')
131136

132137
# Parameters
133138
lambda_function_arn = '<<REPLACE WITH LAMBDA FUNCTION ARN>>'
@@ -246,6 +251,7 @@ https://docs.docker.com/get-docker/
246251

247252
1. Delete AWS Lambda `sam delete`.
248253
2. Delete CloudWatch Dashboard.
254+
3. Delete EventBridge rule.
249255

250256
## Security
251257

0 commit comments

Comments
 (0)