Need advice about which tool to choose?Ask the StackShare community!

Amazon SWF

35
72
+ 1
0
AWS Data Pipeline

91
362
+ 1
1
Add tool

Amazon SWF vs AWS Data Pipeline: What are the differences?

Amazon SWF: Automate the coordination, auditing, and scaling of applications across multiple machines. Amazon Simple Workflow allows you to structure the various processing steps in an application that runs across one or more machines as a set of “tasks.” Amazon SWF manages dependencies between the tasks, schedules the tasks for execution, and runs any logic that needs to be executed in parallel. The service also stores the tasks, reliably dispatches them to application components, tracks their progress, and keeps their latest state; AWS Data Pipeline: Process and move data between different AWS compute and storage services. AWS Data Pipeline is a web service that provides a simple management system for data-driven workflows. Using AWS Data Pipeline, you define a pipeline composed of the “data sources” that contain your data, the “activities” or business logic such as EMR jobs or SQL queries, and the “schedule” on which your business logic executes. For example, you could define a job that, every hour, runs an Amazon Elastic MapReduce (Amazon EMR)–based analysis on that hour’s Amazon Simple Storage Service (Amazon S3) log data, loads the results into a relational database for future lookup, and then automatically sends you a daily summary email.

Amazon SWF belongs to "Cloud Task Management" category of the tech stack, while AWS Data Pipeline can be primarily classified under "Data Transfer".

Some of the features offered by Amazon SWF are:

  • Maintaining application state
  • Tracking workflow executions and logging their progress
  • Holding and dispatching tasks

On the other hand, AWS Data Pipeline provides the following key features:

  • You can find (and use) a variety of popular AWS Data Pipeline tasks in the AWS Management Console’s template section.
  • Hourly analysis of Amazon S3‐based log data
  • Daily replication of AmazonDynamoDB data to Amazon S3
Get Advice from developers at your company using Private StackShare. Sign up for Private StackShare.
Learn More
Pros of Amazon SWF
Pros of AWS Data Pipeline
    Be the first to leave a pro
    • 1
      Easy to create DAG and execute it

    Sign up to add or upvote prosMake informed product decisions

    What is Amazon SWF?

    Amazon Simple Workflow allows you to structure the various processing steps in an application that runs across one or more machines as a set of “tasks.” Amazon SWF manages dependencies between the tasks, schedules the tasks for execution, and runs any logic that needs to be executed in parallel. The service also stores the tasks, reliably dispatches them to application components, tracks their progress, and keeps their latest state.

    What is AWS Data Pipeline?

    AWS Data Pipeline is a web service that provides a simple management system for data-driven workflows. Using AWS Data Pipeline, you define a pipeline composed of the “data sources” that contain your data, the “activities” or business logic such as EMR jobs or SQL queries, and the “schedule” on which your business logic executes. For example, you could define a job that, every hour, runs an Amazon Elastic MapReduce (Amazon EMR)–based analysis on that hour’s Amazon Simple Storage Service (Amazon S3) log data, loads the results into a relational database for future lookup, and then automatically sends you a daily summary email.

    Need advice about which tool to choose?Ask the StackShare community!

    What companies use Amazon SWF?
    What companies use AWS Data Pipeline?
    See which teams inside your own company are using Amazon SWF or AWS Data Pipeline.
    Sign up for Private StackShareLearn More

    Sign up to get full access to all the companiesMake informed product decisions

    What tools integrate with Amazon SWF?
    What tools integrate with AWS Data Pipeline?
    What are some alternatives to Amazon SWF and AWS Data Pipeline?
    Celery
    Celery is an asynchronous task queue/job queue based on distributed message passing. It is focused on real-time operation, but supports scheduling as well.
    Airflow
    Use Airflow to author workflows as directed acyclic graphs (DAGs) of tasks. The Airflow scheduler executes your tasks on an array of workers while following the specified dependencies. Rich command lines utilities makes performing complex surgeries on DAGs a snap. The rich user interface makes it easy to visualize pipelines running in production, monitor progress and troubleshoot issues when needed.
    Amazon SQS
    Transmit any volume of data, at any level of throughput, without losing messages or requiring other services to be always available. With SQS, you can offload the administrative burden of operating and scaling a highly available messaging cluster, while paying a low price for only what you use.
    AWS Step Functions
    AWS Step Functions makes it easy to coordinate the components of distributed applications and microservices using visual workflows. Building applications from individual components that each perform a discrete function lets you scale and change applications quickly.
    Google Keep
    It is a note-taking service developed by Google. It is available on the web, and has mobile apps for the Android and iOS mobile operating systems. Keep offers a variety of tools for taking notes, including text, lists, images, and audio.
    See all alternatives