You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
This repository was archived by the owner on Mar 7, 2020. It is now read-only.
Copy file name to clipboardExpand all lines: README.md
+33-42Lines changed: 33 additions & 42 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -5,19 +5,17 @@
5
5
6
6
## Introduction
7
7
8
-
This is an example [AWS Lambda][aws-lambda] application for processing a [Kinesis][aws-kinesis] stream of events ([introductory blog post][blog-post]). It reads the stream of simple JSON events generated by our event generator. Our lambda function aggregates and buckets events and stores them in DynamoDB.
8
+
This is an example [AWS Lambda][aws-lambda] application for processing a [Kinesis][aws-kinesis] stream of events ([introductory blog post][blog-post]). It reads the stream of simple JSON events generated by our event generator. Our AWS Lambda function aggregates and buckets events and stores them in [DynamoDB][aws-dynamodb].
9
9
10
+
This was built by the Data Science team at [Snowplow Analytics][snowplow], who use AWS Lambda in their projects.
10
11
11
-
This was built by the Data Science team at [Snowplow Analytics][snowplow].
12
-
13
-
**Running this requires an Amazon AWS account, and it will incur charges.**
14
-
15
-
_See also:_[Spark Example Project][spark-example-project] | [Spark Streaming Example Project][spark-streaming-example-project]
12
+
**Running this requires an Amazon AWS account, and will incur charges.**
16
13
14
+
_See also:_[Spark Streaming Example Project][spark-streaming-example-project] | [Spark Example Project][spark-example-project]
17
15
18
16
## Overview
19
17
20
-
We have implemented a super-simple analytics-on-write stream processing job using AWS Lambda. Our AWS Lambda Node.js job reads a Kinesis stream containing events in a JSON format:
18
+
We have implemented a super-simple analytics-on-write stream processing job using AWS Lambda. Our AWS Lambda function, written in JavaScript, reads a Kinesis stream containing events in a JSON format:
You can follow along in [the release blog post][blog-post] to get the project up and running yourself.
51
49
52
-
The below steps assume that you are running inside Vagrant, as per the Developer Quickstart above.
50
+
The following steps assume that you are running inside Vagrant, as per the Developer Quickstart above.
53
51
54
52
### 1. Setting up AWS
55
53
@@ -63,10 +61,10 @@ Default region name [None]: us-east-1
63
61
Default output format [None]: json
64
62
```
65
63
66
-
Now we can create our DynamoDB table, Kinesis Stream, and IAM/CloudFormation/Lambda Roles. We will be using [Cloudformation][http://aws.amazon.com/cloudformation] to make our new role. Using grunt, we will test your credentials by:
64
+
Now we can create our DynamoDB table, Kinesis stream, and IAM role. We will be using [CloudFormation][http://aws.amazon.com/cloudformation] to make our new role. Using Grunt, we can create all like so:
### 2. Connect AWS Lambda service with the new role and building the project
95
94
96
-
Wait a minute to ensure our roles gets created. In this step we connect the new service role to access Kinesis, Cloudwatch, Lambda, and DynamoDB. We will attach an admin policy to the lambda exec role to easily access the services. Using grunt, our javascript project gets assembled into a zip file for upload to the AWS Lambda service. Once its zipped, we attach a service role to it:
95
+
Wait a minute to ensure our IAM service role gets created. Now we connect the new service role to access Kinesis, CloudWatch, Lambda, and DynamoDB. We will attach an admin policy to the lambda exec role to easily access the services. Using Grunt, our AWS Lambda functiongets assembled into a zip file for upload to the AWS Lambda service. Once it's zipped, we attach a service role to it:
97
96
98
97
```bash
99
98
$ grunt role
@@ -114,7 +113,6 @@ Created package at dist/aws-lambda-example-project_0-1-0_latest.zip
114
113
...
115
114
```
116
115
117
-
118
116
### 3. Deploy zip file to AWS Lambda service and connect Kinesis to Lambda
119
117
120
118
In deploy this project to Lambda with the `grunt deploy` command:
@@ -128,7 +126,7 @@ Created AWS Lambda Function...
128
126
129
127
### 4. Connect Kinesis to Lambda
130
128
131
-
The final step to getting this projected ready to start processing events is to connect Kinesis to the Lambda project with this command:
129
+
The final step to getting this projected ready to start processing events is to associate our Kinesis stream to the Lambda function with this command:
First head over to the AWS Lambda service console. Then review the logs inCloudwatch.
167
+
First head over to the AWS Lambda service console, then review the logs in CloudWatch.
172
168
173
169
Finally, let's check the data in our DynamoDB table. Make sure you are in the correct AWS region, then click on `my-table` and hit the `Explore Table` button:
174
170
@@ -181,11 +177,10 @@ For each **BucketStart** and **EventType** pair, we see a **Count**, plus some *
181
177
* Various improvements for the [0.2.0 release] [020-milestone]
182
178
* Expanding our analytics-on-write thinking into our new [Icebucket] [icebucket] project
183
179
184
-
185
180
## Credits
186
181
187
-
Tim Bell [@Tim-B] [tim-b] for his blog post [Writing Functions for AWS Lambda Using NPM and Grunt] [tim-b-post]. Thanks to Ian Meyers and his
188
-
[Amazon-Kinesis-Aggregators-Project][amazon-kinesis-aggregators]. A true inspiration for streaming analytics on write.
182
+
* [Tim Bell] [tim-b] for his blog post [Writing Functions for AWS Lambda Using NPM and Grunt] [tim-b-post]
183
+
* Ian Meyers and his [Amazon-Kinesis-Aggregators-Project][amazon-kinesis-aggregators], a true inspiration for streaming analytics-on-write
189
184
190
185
## Copyright and license
191
186
@@ -200,36 +195,32 @@ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
200
195
See the License for the specific language governing permissions and
0 commit comments