Benefits & Limitations of Spark on AWS Lambda

Jump to bottom

John Cherian edited this page Mar 1, 2023 · 6 revisions

Benefits:-

Batch Workloads

Cost effective: Tested workloads from 50MB to 300MB per payload on the Spark on Lambda Framework. It consumes 1.5 to 2GB AWS Lambda Memory and takes under a 1 min. It keeps the per workload cost within few cents or free(if you have AWS Lambda quoata) .
AWS Lambda Quota : 1 million free requests per month: This includes both new and existing users, and applies to all functions running in AWS Lambda. After the first 1 million requests, there is a charge of .20 per 1 million requests.

Limitations:-

Streaming:

Kafka AWS Lambda trigger has a limitation of 6 MB per payload. Please ensure that you configure the batch size in the trigger to control the volume.
As of now, Single AWS Lambda execution is tested and functional. Triggering multiple AWS Lambda containers and writing to Apache HUDI is pending.
The event parameter from AWS Lambda to be passed into the Spark script as parameters.