Benefits & Limitations of Spark on AWS Lambda

Benefits:-

Batch Workloads

• Cost effective: Tested workloads from 50MB to 300MB per payload on the Spark on Lambda Framework. It consumes 1.5 to 2GB AWS Lambda Memory and takes under a 1 min.

Limitations:-

Streaming:

• Kafka AWS Lambda trigger has a limitation of 6 MB per payload. Please ensure that you configure the batch size in the trigger to control the volume.
• As of now, Single AWS Lambda execution is tested and functional. Triggering multiple AWS Lambda containers and writing to Apache HUDI is pending.
• The event parameter from AWS Lambda to be passed into the Spark script as parameters.