Skip to content
This repository was archived by the owner on Mar 7, 2020. It is now read-only.

Commit ad26df2

Browse files
committed
Prepared for release
1 parent deb3430 commit ad26df2

File tree

7 files changed

+56
-50
lines changed

7 files changed

+56
-50
lines changed

.gitignore

Lines changed: 16 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -3,5 +3,19 @@
33
# Vagrant
44
.vagrant/
55

6-
# build directory
7-
tmp
6+
# node.js
7+
lib-cov
8+
*.seed
9+
*.log
10+
*.csv
11+
*.dat
12+
*.out
13+
*.pid
14+
*.gz
15+
16+
pids
17+
logs
18+
results
19+
20+
npm-debug.log
21+
node_modules

CHANGELOG

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,3 +1,3 @@
1-
Version 0.1.0 (2015-0X-XX)
1+
Version 0.1.0 (2015-07-10)
22
--------------------------
33
Initial release

Gruntfile.js

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -45,7 +45,7 @@ grunt.initConfig({
4545
});
4646

4747
grunt.loadTasks('tasks');
48-
grunt.registerTask('default', ['dynamo','createRole','kinesis']);
48+
grunt.registerTask('init', ['dynamo','createRole','kinesis']);
4949
grunt.registerTask('role', ['attachRole','packaging']);
5050
grunt.registerTask('deploy', ['deployLambda']);
5151
grunt.registerTask('connect', ['associateStream']);

README.md

Lines changed: 33 additions & 42 deletions
Original file line numberDiff line numberDiff line change
@@ -5,19 +5,17 @@
55

66
## Introduction
77

8-
This is an example [AWS Lambda] [aws-lambda] application for processing a [Kinesis] [aws-kinesis] stream of events ([introductory blog post] [blog-post]). It reads the stream of simple JSON events generated by our event generator. Our lambda function aggregates and buckets events and stores them in DynamoDB.
8+
This is an example [AWS Lambda] [aws-lambda] application for processing a [Kinesis] [aws-kinesis] stream of events ([introductory blog post] [blog-post]). It reads the stream of simple JSON events generated by our event generator. Our AWS Lambda function aggregates and buckets events and stores them in [DynamoDB] [aws-dynamodb].
99

10+
This was built by the Data Science team at [Snowplow Analytics] [snowplow], who use AWS Lambda in their projects.
1011

11-
This was built by the Data Science team at [Snowplow Analytics] [snowplow].
12-
13-
**Running this requires an Amazon AWS account, and it will incur charges.**
14-
15-
_See also:_ [Spark Example Project] [spark-example-project] | [Spark Streaming Example Project][spark-streaming-example-project]
12+
**Running this requires an Amazon AWS account, and will incur charges.**
1613

14+
_See also:_ [Spark Streaming Example Project][spark-streaming-example-project] | [Spark Example Project] [spark-example-project]
1715

1816
## Overview
1917

20-
We have implemented a super-simple analytics-on-write stream processing job using AWS Lambda. Our AWS Lambda Node.js job reads a Kinesis stream containing events in a JSON format:
18+
We have implemented a super-simple analytics-on-write stream processing job using AWS Lambda. Our AWS Lambda function, written in JavaScript, reads a Kinesis stream containing events in a JSON format:
2119

2220
```json
2321
{
@@ -42,14 +40,14 @@ Assuming git, [Vagrant] [vagrant-install] and [VirtualBox] [virtualbox-install]
4240
guest$ cd /vagrant
4341
guest# npm install grunt
4442
guest$ npm install
45-
guest$ grunt
43+
guest$ grunt --help
4644
```
4745

4846
## Tutorial
4947

5048
You can follow along in [the release blog post] [blog-post] to get the project up and running yourself.
5149

52-
The below steps assume that you are running inside Vagrant, as per the Developer Quickstart above.
50+
The following steps assume that you are running inside Vagrant, as per the Developer Quickstart above.
5351

5452
### 1. Setting up AWS
5553

@@ -63,10 +61,10 @@ Default region name [None]: us-east-1
6361
Default output format [None]: json
6462
```
6563

66-
Now we can create our DynamoDB table, Kinesis Stream, and IAM/CloudFormation/Lambda Roles. We will be using [Cloudformation] [http://aws.amazon.com/cloudformation] to make our new role. Using grunt, we will test your credentials by:
64+
Now we can create our DynamoDB table, Kinesis stream, and IAM role. We will be using [CloudFormation] [http://aws.amazon.com/cloudformation] to make our new role. Using Grunt, we can create all like so:
6765

6866
```bash
69-
$ grunt
67+
$ grunt init
7068
Running "dynamo:default" (dynamo) task
7169
{ TableDescription:
7270
{ AttributeDefinitions: [ [Object], [Object], [Object] ],
@@ -89,11 +87,12 @@ Running "createRole:default" (createRole) task
8987
Running "kinesis:default" (kinesis) task
9088
{}
9189

90+
Done, without errors.
9291
```
9392
9493
### 2. Connect AWS Lambda service with the new role and building the project
9594
96-
Wait a minute to ensure our roles gets created. In this step we connect the new service role to access Kinesis, Cloudwatch, Lambda, and DynamoDB. We will attach an admin policy to the lambda exec role to easily access the services. Using grunt, our javascript project gets assembled into a zip file for upload to the AWS Lambda service. Once its zipped, we attach a service role to it:
95+
Wait a minute to ensure our IAM service role gets created. Now we connect the new service role to access Kinesis, CloudWatch, Lambda, and DynamoDB. We will attach an admin policy to the lambda exec role to easily access the services. Using Grunt, our AWS Lambda function gets assembled into a zip file for upload to the AWS Lambda service. Once it's zipped, we attach a service role to it:
9796
9897
```bash
9998
$ grunt role
@@ -114,7 +113,6 @@ Created package at dist/aws-lambda-example-project_0-1-0_latest.zip
114113
...
115114
```
116115
117-
118116
### 3. Deploy zip file to AWS Lambda service and connect Kinesis to Lambda
119117
120118
In deploy this project to Lambda with the `grunt deploy` command:
@@ -128,7 +126,7 @@ Created AWS Lambda Function...
128126
129127
### 4. Connect Kinesis to Lambda
130128
131-
The final step to getting this projected ready to start processing events is to connect Kinesis to the Lambda project with this command:
129+
The final step to getting this projected ready to start processing events is to associate our Kinesis stream to the Lambda function with this command:
132130
133131
```bash
134132
$ grunt connect
@@ -146,10 +144,9 @@ arn:aws:kinesis:us-east-1:844709429716:stream/my-stream
146144
Done, without errors.
147145
```
148146
149-
150147
### 5. Sending events to Kinesis
151148
152-
We need to start sending events to our new Kinesis stream. We have created a helper method to do this - run the below and leave it running:
149+
We need to start sending events to our new Kinesis stream. We have created a helper method to do this - run the below and leave it running in a tab:
153150
154151
```bash
155152
$ grunt events
@@ -165,10 +162,9 @@ Writing Kineis Event: {"timestamp":"2015-06-29T20:12:22.708Z","type":"Green"}
165162
...
166163
```
167164
168-
169165
### 6. Monitoring your job
170166
171-
First head over to the AWS Lambda service console. Then review the logs in Cloudwatch.
167+
First head over to the AWS Lambda service console, then review the logs in CloudWatch.
172168
173169
Finally, let's check the data in our DynamoDB table. Make sure you are in the correct AWS region, then click on `my-table` and hit the `Explore Table` button:
174170
@@ -181,11 +177,10 @@ For each **BucketStart** and **EventType** pair, we see a **Count**, plus some *
181177
* Various improvements for the [0.2.0 release] [020-milestone]
182178
* Expanding our analytics-on-write thinking into our new [Icebucket] [icebucket] project
183179
184-
185180
## Credits
186181
187-
Tim Bell [@Tim-B] [tim-b] for his blog post [Writing Functions for AWS Lambda Using NPM and Grunt] [tim-b-post]. Thanks to Ian Meyers and his
188-
[Amazon-Kinesis-Aggregators-Project][amazon-kinesis-aggregators]. A true inspiration for streaming analytics on write.
182+
* [Tim Bell] [tim-b] for his blog post [Writing Functions for AWS Lambda Using NPM and Grunt] [tim-b-post]
183+
* Ian Meyers and his [Amazon-Kinesis-Aggregators-Project][amazon-kinesis-aggregators], a true inspiration for streaming analytics-on-write
189184
190185
## Copyright and license
191186
@@ -200,36 +195,32 @@ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
200195
See the License for the specific language governing permissions and
201196
limitations under the License.
202197
203-
[travis]: https://travis-ci.org/snowplow/spark-streaming-example-project
204-
[travis-image]: https://travis-ci.org/snowplow/spark-streaming-example-project.png?branch=master
198+
[travis]: https://travis-ci.org/snowplow/aws-lambda-nodejs-example-project
199+
[travis-image]: https://travis-ci.org/snowplow/aws-lambda-nodejs-example-project.png?branch=master
205200
[license-image]: http://img.shields.io/badge/license-Apache--2-blue.svg?style=flat
206201
[license]: http://www.apache.org/licenses/LICENSE-2.0
207202
[release-image]: http://img.shields.io/badge/release-0.1.0-blue.svg?style=flat
208-
[releases]: https://github.com/snowplow/spark-streaming-example-project/releases
203+
[releases]: https://github.com/snowplow/aws-lambda-nodejs-example-project/releases
204+
[grunt-image]: https://cdn.gruntjs.com/builtwith.png
205+
206+
[spark-example-project]: https://github.com/snowplow/spark-example-project
207+
[spark-streaming-example-project]: https://github.com/snowplow/spark-streaming-example-project
208+
209+
[vagrant-install]: http://docs.vagrantup.com/v2/installation/index.html
210+
[virtualbox-install]: https://www.virtualbox.org/wiki/Downloads
211+
212+
[blog-post]: http://snowplowanalytics.com/blog/2015/07/10/aws-lambda-nodejs-example-project-0.1.0-released/
213+
[020-milestone]: https://github.com/snowplow/aws-lambda-nodejs-example-project/milestones/Version%200.2.0
214+
[dynamodb-table-image]: /docs/dynamodb-table-image.png?raw=true
209215
210-
[amazon-kinesis-aggregators]: https://github.com/awslabs/amazon-kinesis-aggregators
211216
[aws-lambda]: http://aws.amazon.com/lambda/
212217
[aws-kinesis]: http://aws.amazon.com/kinesis/
213-
[kinesis-example-producer]: https://github.com/snowplow/kinesis-example-scala-consumer
218+
[aws-dynamodb]: http://aws.amazon.com/dynamodb
214219
[vagrant-install]: http://docs.vagrantup.com/v2/installation/index.html
215220
[virtualbox-install]: https://www.virtualbox.org/wiki/Downloads
216-
[snowplow-kinesis]: https://github.com/snowplow/snowplow/wiki/Install-Scala-Kinesis-Enrich
217221
[tim-b]: https://github.com/Tim-B
218222
[tim-b-post]: http://hipsterdevblog.com/blog/2014/12/07/writing-functions-for-aws-lambda-using-npm-and-grunt/
219-
[grunt-aws-lambda]: https://github.com/Tim-B/grunt-aws-lambda
220-
[blog-post]: http://snowplowanalytics.com/blog/2015/06/10/spark-streaming-example-project-0.1.0-released/
221-
[dynamodb-table-image]: /docs/dynamodb-table-image.png?raw=true
222-
[spark-streaming]: https://spark.apache.org/streaming/
223-
[kinesis]: http://aws.amazon.com/kinesis
224-
[dynamodb]: http://aws.amazon.com/dynamodb
223+
[amazon-kinesis-aggregators]: https://github.com/awslabs/amazon-kinesis-aggregators
224+
225225
[snowplow]: http://snowplowanalytics.com
226226
[icebucket]: https://github.com/snowplow/icebucket
227-
[aws-lambda]: http://aws.amazon.com/lambda
228-
[vagrant-install]: http://docs.vagrantup.com/v2/installation/index.html
229-
[virtualbox-install]: https://www.virtualbox.org/wiki/Downloads
230-
[spark-example-project]: https://github.com/snowplow/spark-example-project
231-
[spark-streaming-example-project]: https://github.com/snowplow/spark-streaming-example-project
232-
[grunt-image]: https://cdn.gruntjs.com/builtwith.png
233-
234-
235-
[license]: http://www.apache.org/licenses/LICENSE-2.

dist/.gitignore

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,3 @@
1+
# Nothing in dist gets saved to version control
2+
*
3+
!.gitignore

vagrant/up.guidance

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -3,4 +3,4 @@ vagrant ssh
33
cd /vagrant
44
npm install grunt
55
npm install
6-
grunt
6+
grunt --help

vagrant/up.playbooks

Lines changed: 1 addition & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,3 @@
11
oss-playbooks/nodejs.yml
2-
oss-playbooks/grunt.yml
32
oss-playbooks/aws-cli-and-psql.yml
4-
oss-playbooks/invoke.yml
5-
oss-playbooks/jq.yml
3+
oss-playbooks/grunt.yml

0 commit comments

Comments
 (0)