Need advice about which tool to choose?Ask the StackShare community!

Amazon EMR

542
680
+ 1
54
AWS Lambda

23.7K
18.4K
+ 1
432
Add tool

AWS Lambda vs Amazon EMR: What are the differences?

AWS Lambda and Amazon EMR are both services provided by Amazon Web Services (AWS) that offer compute capabilities for different purposes. Let's explore the key differences between them.

  1. Use Case: AWS Lambda is a serverless compute service that allows you to run code without provisioning or managing servers. It is ideal for executing small, event-driven functions and building serverless applications. On the other hand, Amazon EMR is a managed cluster platform that enables you to process large amounts of data using frameworks like Apache Hadoop, Spark, and Presto. It is designed for big data processing and analytics.

  2. Scalability and Flexibility: AWS Lambda automatically scales the execution of functions in response to incoming events. It can handle a high number of concurrent requests and scale down to zero when there is no traffic. In contrast, Amazon EMR allows you to provision and manage a cluster of virtual servers to process large-scale data. You can scale the cluster up or down based on your processing needs.

  3. Execution Time and Latency: AWS Lambda functions have a maximum execution time of 15 minutes. They are optimized for short-running tasks and provide low-latency compute resources. On the other hand, Amazon EMR jobs can run for a longer period, ranging from minutes to hours or even days, depending on the complexity of the processing task. However, EMR jobs may have higher latency compared to Lambda functions due to the nature of distributed processing.

  4. Cost Model: AWS Lambda follows a pay-as-you-go pricing model where you are billed based on the number of invocations and the execution duration of functions. It is suitable for workloads with sporadic or unpredictable traffic patterns. Amazon EMR, on the other hand, follows a more traditional pricing model where you pay for the EC2 instances in the cluster, storage, and other associated services. It is designed for workloads that require continuous processing and have predictable resource requirements.

  5. Managed vs. Fully Managed: While both AWS Lambda and Amazon EMR are managed services, there is a difference in the level of management provided. AWS Lambda is a fully managed service where you only need to focus on writing and deploying your code. Amazon EMR is also a managed service, but you have more control and responsibility over the configuration and management of the underlying infrastructure.

  6. Supported Frameworks: AWS Lambda supports a variety of programming languages, including JavaScript, Python, Java, C#, and Go. It integrates well with other AWS services and can be easily used in serverless architectures. On the other hand, Amazon EMR supports popular big data frameworks like Apache Hadoop, Spark, and Presto. It provides the flexibility to use the tools and libraries that are commonly used in the big data ecosystem.

In summary, AWS Lambda is a serverless compute service that is ideal for small, event-driven functions and serverless applications. Amazon EMR, on the other hand, is a managed cluster platform designed for big data processing and analytics.

Decisions about Amazon EMR and AWS Lambda

When adding a new feature to Checkly rearchitecting some older piece, I tend to pick Heroku for rolling it out. But not always, because sometimes I pick AWS Lambda . The short story:

  • Developer Experience trumps everything.
  • AWS Lambda is cheap. Up to a limit though. This impact not only your wallet.
  • If you need geographic spread, AWS is lonely at the top.
The setup

Recently, I was doing a brainstorm at a startup here in Berlin on the future of their infrastructure. They were ready to move on from their initial, almost 100% Ec2 + Chef based setup. Everything was on the table. But we crossed out a lot quite quickly:

  • Pure, uncut, self hosted Kubernetes — way too much complexity
  • Managed Kubernetes in various flavors — still too much complexity
  • Zeit — Maybe, but no Docker support
  • Elastic Beanstalk — Maybe, bit old but does the job
  • Heroku
  • Lambda

It became clear a mix of PaaS and FaaS was the way to go. What a surprise! That is exactly what I use for Checkly! But when do you pick which model?

I chopped that question up into the following categories:

  • Developer Experience / DX 🤓
  • Ops Experience / OX 🐂 (?)
  • Cost 💵
  • Lock in 🔐

Read the full post linked below for all details

See more
Get Advice from developers at your company using StackShare Enterprise. Sign up for StackShare Enterprise.
Learn More
Pros of Amazon EMR
Pros of AWS Lambda
  • 15
    On demand processing power
  • 12
    Don't need to maintain Hadoop Cluster yourself
  • 7
    Hadoop Tools
  • 6
    Elastic
  • 4
    Backed by Amazon
  • 3
    Flexible
  • 3
    Economic - pay as you go, easy to use CLI and SDKs
  • 2
    Don't need a dedicated Ops group
  • 1
    Massive data handling
  • 1
    Great support
  • 129
    No infrastructure
  • 83
    Cheap
  • 70
    Quick
  • 59
    Stateless
  • 47
    No deploy, no server, great sleep
  • 12
    AWS Lambda went down taking many sites with it
  • 6
    Event Driven Governance
  • 6
    Extensive API
  • 6
    Auto scale and cost effective
  • 6
    Easy to deploy
  • 5
    VPC Support
  • 3
    Integrated with various AWS services

Sign up to add or upvote prosMake informed product decisions

Cons of Amazon EMR
Cons of AWS Lambda
    Be the first to leave a con
    • 7
      Cant execute ruby or go
    • 3
      Compute time limited
    • 1
      Can't execute PHP w/o significant effort

    Sign up to add or upvote consMake informed product decisions

    What is Amazon EMR?

    It is used in a variety of applications, including log analysis, data warehousing, machine learning, financial analysis, scientific simulation, and bioinformatics.

    What is AWS Lambda?

    AWS Lambda is a compute service that runs your code in response to events and automatically manages the underlying compute resources for you. You can use AWS Lambda to extend other AWS services with custom logic, or create your own back-end services that operate at AWS scale, performance, and security.

    Need advice about which tool to choose?Ask the StackShare community!

    What companies use Amazon EMR?
    What companies use AWS Lambda?
    See which teams inside your own company are using Amazon EMR or AWS Lambda.
    Sign up for StackShare EnterpriseLearn More

    Sign up to get full access to all the companiesMake informed product decisions

    What tools integrate with Amazon EMR?
    What tools integrate with AWS Lambda?

    Sign up to get full access to all the tool integrationsMake informed product decisions

    Blog Posts

    Aug 28 2019 at 3:10AM

    Segment

    PythonJavaAmazon S3+16
    7
    2569
    GitHubPythonNode.js+47
    55
    72425
    GitHubDockerAmazon EC2+23
    12
    6579
    JavaScriptGitHubPython+42
    53
    21955
    GitHubMySQLSlack+44
    109
    50684
    What are some alternatives to Amazon EMR and AWS Lambda?
    Amazon EC2
    It is a web service that provides resizable compute capacity in the cloud. It is designed to make web-scale computing easier for developers.
    Hadoop
    The Apache Hadoop software library is a framework that allows for the distributed processing of large data sets across clusters of computers using simple programming models. It is designed to scale up from single servers to thousands of machines, each offering local computation and storage.
    Amazon DynamoDB
    With it , you can offload the administrative burden of operating and scaling a highly available distributed database cluster, while paying a low price for only what you use.
    Amazon Redshift
    It is optimized for data sets ranging from a few hundred gigabytes to a petabyte or more and costs less than $1,000 per terabyte per year, a tenth the cost of most traditional data warehousing solutions.
    Azure HDInsight
    It is a cloud-based service from Microsoft for big data analytics that helps organizations process large amounts of streaming or historical data.
    See all alternatives