Airflow vs Jenkins

Need advice about which tool to choose?Ask the StackShare community!

Airflow

1.5K
2.4K
+ 1
123
Jenkins

50.5K
42.7K
+ 1
2.2K
Add tool
Advice on Airflow and Jenkins
Needs advice
on
AirflowAirflow
and
JenkinsJenkins

I am looking for an open-source scheduler tool with cross-functional application dependencies. Some of the tasks I am looking to schedule are as follows:

  1. Trigger Matillion ETL loads
  2. Trigger Attunity Replication tasks that have downstream ETL loads
  3. Trigger Golden gate Replication Tasks
  4. Shell scripts, wrappers, file watchers
  5. Event-driven schedules

I have used Airflow in the past, and I know we need to create DAGs for each pipeline. I am not familiar with Jenkins, but I know it works with configuration without much underlying code. I want to evaluate both and appreciate any advise

See more
Replies (2)
Recommends
AirflowAirflow

Hi Teja, Jenkins is more a CI/CD tool for triggering build/test and other CD tasks, from what you're describing you may be able to get along with #Jenkins and add lot's of plugins and create pipelines. But, eventually you're going to need to know #Groovy language to orchestrate all those tasks which is going to be similar to what you do with #Airflow, So, IMHO , Airflow is more for production scheduled tasks and Jenkins is more for CI/CD non-production tasks.

See more
Matthew Rothstein
Recommends
AirflowAirflow

For 1 - 3, Airflow is a better solution.

Cron and OSQuery may be better solutions for 4 and 5, depending on what you're actually trying to do.

In either case, Jenkins is more trouble than it's worth for these types of workloads.

See more
Needs advice
on
Azure PipelinesAzure Pipelines
and
JenkinsJenkins

We are currently using Azure Pipelines for continous integration. Our applications are developed witn .NET framework. But when we look at the online Jenkins is the most widely used tool for continous integration. Can you please give me the advice which one is best to use for my case Azure pipeline or jenkins.

See more
Replies (1)
Recommends
GitHubGitHub

If your source code is on GitHub, also take a look at Github actions. https://github.com/features/actions

See more
Mohammad Hossein Amri
Chief Technology Officer at Axceligent Solutions · | 3 upvotes · 353.7K views
Needs advice
on
GoCDGoCD
and
JenkinsJenkins

I'm open to anything. just want something that break less and doesn't need me to pay for it, and can be hosted on Docker. our scripting language is powershell core. so it's better to support it. also we are building dotnet core in our pipeline, so if they have anything related that helps with the CI would be nice.

See more
Replies (1)
Ankit Malik
Software Developer at CloudCover · | 1 upvotes · 336.3K views

Google cloud build can help you. It is hosted on cloud and also provide reasonable free quota.

See more
Needs advice
on
ConcourseConcourse
and
JenkinsJenkins

I'm planning to setup complete CD-CD setup for spark and python application which we are going to deploy in aws lambda and EMR Cluster. Which tool would be best one to choose. Since my company is trying to adopt to concourse i would like to understand what are the lack of capabilities concourse have . Thanks in advance !

See more
Replies (1)
Maxi Krone
Cloud Engineer at fme AG · | 2 upvotes · 253.2K views
Recommends
ConcourseConcourse

I would definetly recommend Concourse to you, as it is one of the most advanced modern methods of making CI/CD while Jenkins is an old monolithic dinosaur. Concourse itself is cloudnative and containerbased which helps you to build simple, high-performance and scalable CI/CD pipelines. In my opinion, the only lack of skills you have with Concourse is your own knowledge of how to build pipelines and automate things. Technincally there is no lack, i would even say you can extend it way more easily. But as a Con it is more easy to interact with Jenkins if you are only used to UIs. Concourse needs someone which is capable of using CLIs.

See more
Needs advice
on
AirflowAirflowLuigiLuigi
and
Apache SparkApache Spark

I am so confused. I need a tool that will allow me to go to about 10 different URLs to get a list of objects. Those object lists will be hundreds or thousands in length. I then need to get detailed data lists about each object. Those detailed data lists can have hundreds of elements that could be map/reduced somehow. My batch process dies sometimes halfway through which means hours of processing gone, i.e. time wasted. I need something like a directed graph that will keep results of successful data collection and allow me either pragmatically or manually to retry the failed ones some way (0 - forever) times. I want it to then process all the ones that have succeeded or been effectively ignored and load the data store with the aggregation of some couple thousand data-points. I know hitting this many endpoints is not a good practice but I can't put collectors on all the endpoints or anything like that. It is pretty much the only way to get the data.

See more
Replies (1)
Gilroy Gordon
Solution Architect at IGonics Limited · | 2 upvotes · 188K views
Recommends
CassandraCassandra

For a non-streaming approach:

You could consider using more checkpoints throughout your spark jobs. Furthermore, you could consider separating your workload into multiple jobs with an intermittent data store (suggesting cassandra or you may choose based on your choice and availability) to store results , perform aggregations and store results of those.

Spark Job 1 - Fetch Data From 10 URLs and store data and metadata in a data store (cassandra) Spark Job 2..n - Check data store for unprocessed items and continue the aggregation

Alternatively for a streaming approach: Treating your data as stream might be useful also. Spark Streaming allows you to utilize a checkpoint interval - https://spark.apache.org/docs/latest/streaming-programming-guide.html#checkpointing

See more
Needs advice
on
JenkinsJenkinsTravis CITravis CI
and
CircleCICircleCI

From a StackShare Community member: "Currently we use Travis CI and have optimized it as much as we can so our builds are fairly quick. Our boss is all about redundancy so we are looking for another solution to fall back on in case Travis goes down and/or jacks prices way up (they were recently acquired). Could someone recommend which CI we should go with and if they have time, an explanation of how they're different?"

See more
Replies (6)
Dustin Falgout
Developer at Elegant Themes · | 13 upvotes · 384.8K views

We use CircleCI because of the better value it provides in its plans. I'm sure we could have used Travis just as easily but we found CircleCI's pricing to be more reasonable. In the two years since we signed up, the service has improved. CircleCI is always innovating and iterating on their platform. We have been very satisfied.

See more
Peter Thomas
Distinguished Engineer at Intuit · | 9 upvotes · 367.3K views
Recommends
Travis CITravis CI
at

As the maintainer of the Karate DSL open-source project - I found Travis CI very easy to integrate into the GitHub workflow and it has been steady sailing for more than 2 years now ! It works well for Java / Apache Maven projects and we were able to configure it to use the latest Oracle JDK as per our needs. Thanks to the Travis CI team for this service to the open-source community !

See more

I use Google Cloud Build because it's my first foray into the CICD world(loving it so far), and I wanted to work with something GCP native to avoid giving permissions to other SaaS tools like CircleCI and Travis CI.

I really like it because it's free for the first 120 minutes, and it's one of the few CICD tools that enterprises are open to using since it's contained within GCP.

One of the unique things is that it has the Kaniko cache, which speeds up builds by creating intermediate layers within the docker image vs. pushing the full thing from the start. Helpful when you're installing just a few additional dependencies.

Feel free to checkout an example: Cloudbuild Example

See more
Recommends
Travis CITravis CI

I use Travis CI because of various reasons - 1. Cloud based system so no dedicated server required, and you do not need to administrate it. 2. Easy YAML configuration. 3. Supports Major Programming Languages. 4. Support of build matrix 6. Supports AWS, Azure, Docker, Heroku, Google Cloud, Github Pages, PyPi and lot more. 7. Slack Notifications.

See more
Oded Arbel
Recommends
GitLab CIGitLab CI

You are probably looking at another hosted solution: Jenkins is a good tool but it way too work intensive to be used as just a backup solution.

I have good experience with Circle-CI, Codeship, Drone.io and Travis (as well as problematic experiences with all of them), but my go-to tool is Gitlab CI: simple, powerful and if you have problems with their limitations or pricing, you can always install runners somewhere and use Gitlab just for scheduling and management. Even if you don't host your git repository at Gitlab, you can have Gitlab pull changes automatically from wherever you repo lives.

See more
Recommends
BuildkiteBuildkite

If you are considering Jenkins I would recommend at least checking out Buildkite. The agents are self-hosted (like Jenkins) but the interface is hosted for you. It meshes up some of the things I like about hosted services (pipeline definitions in YAML, managed interface and authentication) with things I like about Jenkins (local customizable agent images, secrets only on own instances, custom agent level scripts, sizing instances to your needs).

See more
Decisions about Airflow and Jenkins

Jenkins is a pretty flexible, complete tool. Especially I love the possibility to configure jobs as a code with Jenkins pipelines.

CircleCI is well suited for small projects where the main task is to run continuous integration as quickly as possible. Travis CI is recommended primarily for open-source projects that need to be tested in different environments.

And for something a bit larger I prefer to use Jenkins because it is possible to make serious system configuration thereby different plugins. In Jenkins, I can change almost anything. But if you want to start the CI chain as soon as possible, Jenkins may not be the right choice.

See more
Get Advice from developers at your company using StackShare Enterprise. Sign up for StackShare Enterprise.
Learn More
Pros of Airflow
Pros of Jenkins
  • 50
    Features
  • 14
    Task Dependency Management
  • 12
    Cluster of workers
  • 12
    Beautiful UI
  • 10
    Extensibility
  • 5
    Complex workflows
  • 5
    Python
  • 5
    Open source
  • 3
    Good api
  • 3
    Custom operators
  • 2
    Dashboard
  • 2
    Apache project
  • 521
    Hosted internally
  • 464
    Free open source
  • 314
    Great to build, deploy or launch anything async
  • 243
    Tons of integrations
  • 210
    Rich set of plugins with good documentation
  • 110
    Has support for build pipelines
  • 72
    Open source and tons of integrations
  • 65
    Easy setup
  • 62
    It is open-source
  • 54
    Workflow plugin
  • 11
    Configuration as code
  • 10
    Very powerful tool
  • 9
    Continuous Integration
  • 9
    Many Plugins
  • 8
    Git and Maven integration is better
  • 8
    Great flexibility
  • 7
    100% free and open source
  • 6
    Slack Integration (plugin)
  • 6
    Github integration
  • 5
    Easy customisation
  • 5
    Self-hosted GitLab Integration (plugin)
  • 4
    Docker support
  • 4
    Pipeline API
  • 3
    Platform idnependency
  • 3
    Excellent docker integration
  • 3
    Fast builds
  • 3
    Hosted Externally
  • 2
    Customizable
  • 2
    AWS Integration
  • 2
    It's Everywhere
  • 2
    JOBDSL
  • 2
    Can be run as a Docker container
  • 2
    It`w worked
  • 1
    Easily extendable with seamless integration
  • 1
    Build PR Branch Only
  • 1
    NodeJS Support
  • 1
    PHP Support
  • 1
    Ruby/Rails Support
  • 1
    Universal controller
  • 1
    Loose Coupling

Sign up to add or upvote prosMake informed product decisions

Cons of Airflow
Cons of Jenkins
  • 2
    Running it on kubernetes cluster relatively complex
  • 2
    Open source - provides minimum or no support
  • 1
    Logical separation of DAGs is not straight forward
  • 1
    Observability is not great when the DAGs exceed 250
  • 12
    Workarounds needed for basic requirements
  • 9
    Groovy with cumbersome syntax
  • 7
    Plugins compatibility issues
  • 6
    Lack of support
  • 6
    Limited abilities with declarative pipelines
  • 4
    No YAML syntax
  • 3
    Too tied to plugins versions

Sign up to add or upvote consMake informed product decisions

- No public GitHub repository available -

What is Airflow?

Use Airflow to author workflows as directed acyclic graphs (DAGs) of tasks. The Airflow scheduler executes your tasks on an array of workers while following the specified dependencies. Rich command lines utilities makes performing complex surgeries on DAGs a snap. The rich user interface makes it easy to visualize pipelines running in production, monitor progress and troubleshoot issues when needed.

What is Jenkins?

In a nutshell Jenkins CI is the leading open-source continuous integration server. Built with Java, it provides over 300 plugins to support building and testing virtually any project.

Need advice about which tool to choose?Ask the StackShare community!

Jobs that mention Airflow and Jenkins as a desired skillset
What companies use Airflow?
What companies use Jenkins?
See which teams inside your own company are using Airflow or Jenkins.
Sign up for StackShare EnterpriseLearn More

Sign up to get full access to all the companiesMake informed product decisions

What tools integrate with Airflow?
What tools integrate with Jenkins?

Sign up to get full access to all the tool integrationsMake informed product decisions

Blog Posts

Mar 24 2021 at 12:57PM

Pinterest

GitJenkinsKafka+7
3
1821
GitJenkinsGroovy+4
4
1965
Dec 4 2019 at 8:01PM

Pinterest

JenkinsKubernetesTensorFlow+4
5
3045
GitHubPythonGit+22
17
13802
JavaScriptGitHubPython+42
52
20646
What are some alternatives to Airflow and Jenkins?
Luigi
It is a Python module that helps you build complex pipelines of batch jobs. It handles dependency resolution, workflow management, visualization etc. It also comes with Hadoop support built in.
Apache NiFi
An easy to use, powerful, and reliable system to process and distribute data. It supports powerful and scalable directed graphs of data routing, transformation, and system mediation logic.
AWS Step Functions
AWS Step Functions makes it easy to coordinate the components of distributed applications and microservices using visual workflows. Building applications from individual components that each perform a discrete function lets you scale and change applications quickly.
Pachyderm
Pachyderm is an open source MapReduce engine that uses Docker containers for distributed computations.
Kubeflow
The Kubeflow project is dedicated to making Machine Learning on Kubernetes easy, portable and scalable by providing a straightforward way for spinning up best of breed OSS solutions.
See all alternatives