AWS Step Functions logo

AWS Step Functions

Build Distributed Applications Using Visual Workflows
+ 1

What is AWS Step Functions?

AWS Step Functions makes it easy to coordinate the components of distributed applications and microservices using visual workflows. Building applications from individual components that each perform a discrete function lets you scale and change applications quickly.
AWS Step Functions is a tool in the Cloud Task Management category of a tech stack.

Who uses AWS Step Functions?

76 companies reportedly use AWS Step Functions in their tech stacks, including CRED, Shelf, and backend.

146 developers on StackShare have stated that they use AWS Step Functions.

AWS Step Functions Integrations

Pros of AWS Step Functions
Integration with other services
Easily Accessible via AWS Console
Complex workflows
Workflow Processing
High Availability
Decisions about AWS Step Functions

Here are some stack decisions, common use cases and reviews by companies and developers who chose AWS Step Functions in their tech stack.

Matheus Moreira
Backend Engineer at IntuitiveCare · | 5 upvotes · 238.9K views

We have some lambdas we need to orchestrate to get our workflow going. In the past, we already attempted to use Airflow as the orchestrator, but the need to coordinate the tasks in a database generates an overhead that we cannot afford. For our use case, there are hundreds of inputs per minute and we need to scale to support all the inputs and have an efficient way to analyze them later. The ideal product would be AWS Step Functions since it can manage our load demand graciously, but it is too expensive and we cannot afford that. So, I would like to get alternatives for an orchestrator that does not need a complex backend, can manage hundreds of inputs per minute, and is not too expensive.

See more
Needs advice
AWS LambdaAWS Lambda
AWS Step FunctionsAWS Step Functions

What's the difference between AWS Lambda and AWS Step Functions?

See more
Needs advice
AWS Step FunctionsAWS Step Functions

I am working on a project that grabs a set of input data from AWS S3, pre-processes and divvies it up, spins up 10K batch containers to process the divvied data in parallel on AWS Batch, post-aggregates the data, and pushes it to S3.

I already have software patterns from other projects for Airflow + Batch but have not dealt with the scaling factors of 10k parallel tasks. Airflow is nice since I can look at which tasks failed and retry a task after debugging. But dealing with that many tasks on one Airflow EC2 instance seems like a barrier. Another option would be to have one task that kicks off the 10k containers and monitors it from there.

I have no experience with AWS Step Functions but have heard it's AWS's Airflow. There looks to be plenty of patterns online for Step Functions + Batch. Do Step Functions seem like a good path to check out for my use case? Do you get the same insights on failing jobs / ability to retry tasks as you do with Airflow?

See more

AWS Step Functions Alternatives & Comparisons

What are some alternatives to AWS Step Functions?
AWS Lambda
AWS Lambda is a compute service that runs your code in response to events and automatically manages the underlying compute resources for you. You can use AWS Lambda to extend other AWS services with custom logic, or create your own back-end services that operate at AWS scale, performance, and security.
Use Airflow to author workflows as directed acyclic graphs (DAGs) of tasks. The Airflow scheduler executes your tasks on an array of workers while following the specified dependencies. Rich command lines utilities makes performing complex surgeries on DAGs a snap. The rich user interface makes it easy to visualize pipelines running in production, monitor progress and troubleshoot issues when needed.
AWS Batch
It enables developers, scientists, and engineers to easily and efficiently run hundreds of thousands of batch computing jobs on AWS. It dynamically provisions the optimal quantity and type of compute resources (e.g., CPU or memory optimized instances) based on the volume and specific resource requirements of the batch jobs submitted.
AWS Data Pipeline
AWS Data Pipeline is a web service that provides a simple management system for data-driven workflows. Using AWS Data Pipeline, you define a pipeline composed of the “data sources” that contain your data, the “activities” or business logic such as EMR jobs or SQL queries, and the “schedule” on which your business logic executes. For example, you could define a job that, every hour, runs an Amazon Elastic MapReduce (Amazon EMR)–based analysis on that hour’s Amazon Simple Storage Service (Amazon S3) log data, loads the results into a relational database for future lookup, and then automatically sends you a daily summary email.
Yes, we’re really free. So, how do we keep the lights on? Instead of charging you a monthly fee, we sell ads on your behalf to the top 500 mobile advertisers in the world. With Batch, you earn money each month while accessing great engagement tools for free.
See all alternatives

AWS Step Functions's Followers
381 developers follow AWS Step Functions to keep up with related blogs and decisions.