Need advice about which tool to choose?Ask the StackShare community!
Airflow vs StackStorm: What are the differences?
Key differences between Airflow and StackStorm
Airflow and StackStorm are both popular workflow automation tools, but they have distinct differences that set them apart.
Architecture: Airflow follows a directed acyclic graph (DAG) model, where each task is represented as a node and dependencies between tasks are represented as edges. StackStorm, on the other hand, follows a rule-based approach, where rules are defined and triggered by events.
Language support: Airflow supports Python natively, allowing users to write their workflows using Python code. StackStorm, on the other hand, supports multiple languages including Python, JavaScript, and Ruby, giving users more flexibility in choosing the language they are comfortable with.
Community and ecosystem: Airflow has a larger and more mature community compared to StackStorm. This means that Airflow has a wider range of plugins, integrations, and community support available. StackStorm, although growing, has a smaller community and a more limited ecosystem of integrations and plugins.
Workflow visualization: Airflow provides a web-based user interface that allows users to visualize their workflows as DAGs and track the progress of tasks. StackStorm, on the other hand, does not provide a built-in visualization tool for workflows, making it less intuitive to track the progress and dependencies of tasks.
Event-driven vs time-based scheduling: Airflow primarily uses time-based scheduling, where tasks are scheduled to run at specific times or intervals. StackStorm, on the other hand, focuses on event-driven automation, where workflows are triggered by events or conditions. This makes StackStorm more suitable for real-time and event-driven workflows.
Extensibility: Airflow allows users to extend its functionality by creating custom operators and hooks using Python. StackStorm also allows for extensibility through the use of custom sensors, actions, and rules. However, StackStorm's rule-based approach provides a more flexible and easier way to extend its functionality compared to Airflow's Python-centric approach.
In summary, Airflow and StackStorm have different architectural models, language support, community and ecosystem, workflow visualization capabilities, scheduling approaches, and extensibility options. Understanding these key differences can help organizations choose the right workflow automation tool for their specific needs.
I am so confused. I need a tool that will allow me to go to about 10 different URLs to get a list of objects. Those object lists will be hundreds or thousands in length. I then need to get detailed data lists about each object. Those detailed data lists can have hundreds of elements that could be map/reduced somehow. My batch process dies sometimes halfway through which means hours of processing gone, i.e. time wasted. I need something like a directed graph that will keep results of successful data collection and allow me either pragmatically or manually to retry the failed ones some way (0 - forever) times. I want it to then process all the ones that have succeeded or been effectively ignored and load the data store with the aggregation of some couple thousand data-points. I know hitting this many endpoints is not a good practice but I can't put collectors on all the endpoints or anything like that. It is pretty much the only way to get the data.
For a non-streaming approach:
You could consider using more checkpoints throughout your spark jobs. Furthermore, you could consider separating your workload into multiple jobs with an intermittent data store (suggesting cassandra or you may choose based on your choice and availability) to store results , perform aggregations and store results of those.
Spark Job 1 - Fetch Data From 10 URLs and store data and metadata in a data store (cassandra) Spark Job 2..n - Check data store for unprocessed items and continue the aggregation
Alternatively for a streaming approach: Treating your data as stream might be useful also. Spark Streaming allows you to utilize a checkpoint interval - https://spark.apache.org/docs/latest/streaming-programming-guide.html#checkpointing
Pros of Airflow
- Features51
- Task Dependency Management14
- Beautiful UI12
- Cluster of workers12
- Extensibility10
- Open source6
- Complex workflows5
- Python5
- Good api3
- Apache project3
- Custom operators3
- Dashboard2
Pros of StackStorm
- Auto-remediation7
- Integrations5
- Automation4
- Complex workflows4
- Open source3
- Beautiful UI2
- ChatOps2
- Python2
- Extensibility1
- Slack1
Sign up to add or upvote prosMake informed product decisions
Cons of Airflow
- Observability is not great when the DAGs exceed 2502
- Running it on kubernetes cluster relatively complex2
- Open source - provides minimum or no support2
- Logical separation of DAGs is not straight forward1
Cons of StackStorm
- Complexity3
- There are not enough sources of information1
Sign up to add or upvote consMake informed product decisions
What is Airflow?
What is StackStorm?
Need advice about which tool to choose?Ask the StackShare community!
What companies use StackStorm?
Sign up to get full access to all the companiesMake informed product decisions
What tools integrate with Airflow?
Sign up to get full access to all the tool integrationsMake informed product decisions
Blog Posts
+17