Need advice about which tool to choose?Ask the StackShare community!

AWS Batch

66
193
+ 1
4
AWS Data Pipeline

89
346
+ 1
1
Add tool
Get Advice from developers at your company using Private StackShare. Sign up for Private StackShare.
Learn More
Pros of AWS Batch
Pros of AWS Data Pipeline
  • 2
    Scalable
  • 2
    Containerized
  • 1
    Easy to create DAG and execute it

Sign up to add or upvote prosMake informed product decisions

Cons of AWS Batch
Cons of AWS Data Pipeline
  • 1
    More overhead than lambda
  • 1
    Image management
    Be the first to leave a con

    Sign up to add or upvote consMake informed product decisions

    What is AWS Batch?

    It enables developers, scientists, and engineers to easily and efficiently run hundreds of thousands of batch computing jobs on AWS. It dynamically provisions the optimal quantity and type of compute resources (e.g., CPU or memory optimized instances) based on the volume and specific resource requirements of the batch jobs submitted.

    What is AWS Data Pipeline?

    AWS Data Pipeline is a web service that provides a simple management system for data-driven workflows. Using AWS Data Pipeline, you define a pipeline composed of the “data sources” that contain your data, the “activities” or business logic such as EMR jobs or SQL queries, and the “schedule” on which your business logic executes. For example, you could define a job that, every hour, runs an Amazon Elastic MapReduce (Amazon EMR)–based analysis on that hour’s Amazon Simple Storage Service (Amazon S3) log data, loads the results into a relational database for future lookup, and then automatically sends you a daily summary email.

    Need advice about which tool to choose?Ask the StackShare community!

    What companies use AWS Batch?
    What companies use AWS Data Pipeline?
    See which teams inside your own company are using AWS Batch or AWS Data Pipeline.
    Sign up for Private StackShareLearn More

    Sign up to get full access to all the companiesMake informed product decisions

    What tools integrate with AWS Batch?
    What tools integrate with AWS Data Pipeline?
    What are some alternatives to AWS Batch and AWS Data Pipeline?
    AWS Lambda
    AWS Lambda is a compute service that runs your code in response to events and automatically manages the underlying compute resources for you. You can use AWS Lambda to extend other AWS services with custom logic, or create your own back-end services that operate at AWS scale, performance, and security.
    Beanstalk
    A single process to commit code, review with the team, and deploy the final result to your customers.
    Airflow
    Use Airflow to author workflows as directed acyclic graphs (DAGs) of tasks. The Airflow scheduler executes your tasks on an array of workers while following the specified dependencies. Rich command lines utilities makes performing complex surgeries on DAGs a snap. The rich user interface makes it easy to visualize pipelines running in production, monitor progress and troubleshoot issues when needed.
    Kubernetes
    Kubernetes is an open source orchestration system for Docker containers. It handles scheduling onto nodes in a compute cluster and actively manages workloads to ensure that their state matches the users declared intentions.
    Serverless
    Build applications comprised of microservices that run in response to events, auto-scale for you, and only charge you when they run. This lowers the total cost of maintaining your apps, enabling you to build more logic, faster. The Framework uses new event-driven compute services, like AWS Lambda, Google CloudFunctions, and more.
    See all alternatives