Alternatives to Resque logo

Alternatives to Resque

Sidekiq, delayed_job, Celery, Beanstalkd, and RabbitMQ are the most popular alternatives and competitors to Resque.
118
126
+ 1
9

What is Resque and what are its top alternatives?

Resque is a popular job queueing system for Ruby applications, built on top of Redis. It provides a simple UI for managing jobs, prioritizing them, and monitoring their status. Resque is known for its reliability, scalability, and performance, making it a go-to choice for background processing in many Ruby projects. However, one of its limitations is that it requires a separate Redis instance to function, adding complexity to the setup process.

  1. Sidekiq: Sidekiq is a popular alternative to Resque that is known for its high performance and low memory usage. It utilizes threads instead of separate processes for better efficiency. Pros: Efficient memory usage, support for multi-threading. Cons: Paid options for some advanced features.
  2. Delayed::Job: Delayed::Job is a simple, database-backed job queue that is easy to set up and use. It is a lightweight alternative to Resque with fewer dependencies. Pros: Easy to set up, minimal configuration required. Cons: May not be as efficient for high-traffic applications.
  3. Que: Que is a high-performance job queue for Ruby applications that is also built on top of PostgreSQL. It offers advanced features like job priority and scheduling. Pros: Seamless integration with PostgreSQL, supports job dependencies. Cons: Limited scalability compared to Redis-based solutions.
  4. Sucker Punch: Sucker Punch is a simple, single-threaded background processing library that is suitable for lightweight job queuing needs. Pros: Easy to set up, lightweight solution. Cons: Limited scalability and performance compared to multi-threaded or multi-process alternatives.
  5. Shoryuken: Shoryuken is a concurrent job processor for Amazon SQS that is highly scalable and efficient. It is designed to work well with Rails applications and offers built-in support for handling large volumes of jobs. Pros: Scalable, efficient processing of jobs. Cons: Specific to Amazon SQS, may not be suitable for other queueing systems.
  6. Quebert: Quebert is a versatile job queuing library that supports multiple backends, including Redis and in-memory queues. It offers features like job retry logic and error handling. Pros: Flexible backend support, robust error handling. Cons: May require additional configuration for specific use cases.
  7. Sneakers: Sneakers is a fast and scalable queuing system for Ruby applications that is built on top of RabbitMQ. It is suitable for high-throughput and mission-critical applications. Pros: High performance, built-in support for RabbitMQ. Cons: Requires a RabbitMQ server to function, may add complexity to the setup.
  8. Kue: Kue is a feature-rich job queuing system for Node.js applications that offers a user-friendly UI for managing jobs. It supports priority queues, delayed jobs, and job progress tracking. Pros: User-friendly interface, comprehensive feature set. Cons: Limited to Node.js applications, may not be suitable for Ruby projects.
  9. Brpoplpush: Brpoplpush is a lightweight Redis queueing library for Ruby that is known for its simplicity and low overhead. It provides basic queueing functionality without the need for additional dependencies. Pros: Lightweight, minimal overhead. Cons: Limited features compared to more advanced job queuing systems.
  10. Backburner: Backburner is a flexible and extensible job queuing system for Ruby applications that offers features like batch processing and job lifecycle management. It is suitable for handling complex background processing needs. Pros: Extensible architecture, advanced features. Cons: May require additional setup for specific use cases, may not be as straightforward to use as simpler alternatives.

Top Alternatives to Resque

  • Sidekiq
    Sidekiq

    Sidekiq uses threads to handle many jobs at the same time in the same process. It does not require Rails but will integrate tightly with Rails 3/4 to make background processing dead simple. ...

  • delayed_job
    delayed_job

    Delayed_job (or DJ) encapsulates the common pattern of asynchronously executing longer tasks in the background. It is a direct extraction from Shopify where the job table is responsible for a multitude of core tasks. ...

  • Celery
    Celery

    Celery is an asynchronous task queue/job queue based on distributed message passing. It is focused on real-time operation, but supports scheduling as well. ...

  • Beanstalkd
    Beanstalkd

    Beanstalks's interface is generic, but was originally designed for reducing the latency of page views in high-volume web applications by running time-consuming tasks asynchronously. ...

  • RabbitMQ
    RabbitMQ

    RabbitMQ gives your applications a common platform to send and receive messages, and your messages a safe place to live until received. ...

  • Rake
    Rake

    It is a software task management and build automation tool. It allows the user to specify tasks and describe dependencies as well as to group tasks in a namespace. ...

  • JavaScript
    JavaScript

    JavaScript is most known as the scripting language for Web pages, but used in many non-browser environments as well such as node.js or Apache CouchDB. It is a prototype-based, multi-paradigm scripting language that is dynamic,and supports object-oriented, imperative, and functional programming styles. ...

  • Git
    Git

    Git is a free and open source distributed version control system designed to handle everything from small to very large projects with speed and efficiency. ...

Resque alternatives & related posts

Sidekiq logo

Sidekiq

1.1K
629
408
Simple, efficient background processing for Ruby
1.1K
629
+ 1
408
PROS OF SIDEKIQ
  • 124
    Simple
  • 99
    Efficient background processing
  • 60
    Scalability
  • 37
    Better then resque
  • 26
    Great documentation
  • 15
    Admin tool
  • 14
    Great community
  • 8
    Integrates with redis automatically, with zero config
  • 7
    Stupidly simple to integrate and run on Rails/Heroku
  • 7
    Great support
  • 3
    Ruby
  • 3
    Freeium
  • 2
    Pro version
  • 1
    Dashboard w/live polling
  • 1
    Great ecosystem of addons
  • 1
    Fast
CONS OF SIDEKIQ
    Be the first to leave a con

    related Sidekiq posts

    Cyril Duchon-Doris

    After splitting our monolith into a Rails API + a React Redux.js frontend app, it became a necessity to monitor frontend errors. Our frontend application is not your typical website, and features a lot of interesting SPA mechanics that need to be followed closely (many async flows, redux-saga , etc.) in addition to regular browser incompatibility issues. Rollbar kicks in so that we can monitor every bug that happens on our frontend, and aggregate this with almost 0 work. The number of occurrences and affected browsers on each occurence helps us understand the priority and severity of bugs even when our users don't tell us about them, so we can decide whether we need to fix this bug that was encountered by 1k users in less than a few days days VERSUS telling this SINGLE user to switch browsers because he's using a very outdated version that no one else uses. Now we also use Rollbar with Rails, Sidekiq and even AWS Lambda errors since the interface is quite convenient.

    See more
    Cyril Duchon-Doris

    We decided to use AWS Lambda for several serverless tasks such as

    • Managing AWS backups
    • Processing emails received on Amazon SES and stored to Amazon S3 and notified via Amazon SNS, so as to push a message on our Redis so our Sidekiq Rails workers can process inbound emails
    • Pushing some relevant Amazon CloudWatch metrics and alarms to Slack
    See more
    delayed_job logo

    delayed_job

    51
    64
    6
    Database backed asynchronous priority queue -- Extracted from Shopify
    51
    64
    + 1
    6
    PROS OF DELAYED_JOB
    • 3
      Easy to get started
    • 2
      Reliable
    • 1
      Doesn't require Redis
    CONS OF DELAYED_JOB
      Be the first to leave a con

      related delayed_job posts

      John Barton

      Docker Compose might have been a bit of overkill for a dev environment as a solo founder, but I'd found so much with past side projects (though this is no longer a side project) that I would frequently waste time every time I came back to work on the project getting my dev env sorted again.

      Made the conscious choice to make a "prod-ish" docker-compose config up front to make sure that didn't bite me again.

      Structured it so I have the following containers running

      • server - the Rails app in API style
      • client - the Create React App
      • ngrok - ngrok to receive webhooks in dev
      • db - PostgreSQL
      • queues - delayed_job worker
      See more
      Jerome Dalbert
      Principal Backend Software Engineer at StackShare · | 4 upvotes · 86.8K views

      delayed_job is a great Rails background job library for new projects, as it only uses what you already have: a relational database. We happily used it during the company’s first two years.

      But it started to falter as our web and database transactions significantly grew. Our app interacted with users via SMS texts sent inside background jobs. Because the delayed_job daemon ran every couple seconds, this meant that users often waited several long seconds before getting text replies, which was not acceptable. Moreover, job processing was done inside AWS Elastic Beanstalk web instances, which were already under stress and not meant to handle jobs.

      We needed a fast background job system that could process jobs in near real-time and integrate well with AWS. Sidekiq is a fast and popular Ruby background job library, but it does not leverage the Elastic Beanstalk worker architecture, and you have to maintain a Redis instance.

      We ended up choosing active-elastic-job, which seamlessly integrates with worker instances and Amazon SQS. SQS is a fast queue and you don’t need to worry about infrastructure or scaling, as AWS handles it for you.

      We noticed significant performance gains immediately after making the switch.

      #BackgroundProcessing

      See more
      Celery logo

      Celery

      1.6K
      1.6K
      280
      Distributed task queue
      1.6K
      1.6K
      + 1
      280
      PROS OF CELERY
      • 99
        Task queue
      • 63
        Python integration
      • 40
        Django integration
      • 30
        Scheduled Task
      • 19
        Publish/subsribe
      • 8
        Various backend broker
      • 6
        Easy to use
      • 5
        Great community
      • 5
        Workflow
      • 4
        Free
      • 1
        Dynamic
      CONS OF CELERY
      • 4
        Sometimes loses tasks
      • 1
        Depends on broker

      related Celery posts

      James Cunningham
      Operations Engineer at Sentry · | 21 upvotes · 357.2K views

      Sentry started as (and remains) an open-source project, growing out of an error logging tool built in 2008. That original build nine years ago was Django and Celery (Python’s asynchronous task codebase), with PostgreSQL as the database and Redis as the power behind Celery.

      We displayed a truly shrewd notion of branding even then, giving the project a catchy name that companies the world over remain jealous of to this day: django-db-log. For the longest time, Sentry’s subtitle on GitHub was “A simple Django app, built with love.” A slightly more accurate description probably would have included Starcraft and Soylent alongside love; regardless, this captured what Sentry was all about.

      #MessageQueue #InMemoryDatabases

      See more
      James Cunningham
      Operations Engineer at Sentry · | 18 upvotes · 1.7M views
      Shared insights
      on
      CeleryCeleryRabbitMQRabbitMQ
      at

      As Sentry runs throughout the day, there are about 50 different offline tasks that we execute—anything from “process this event, pretty please” to “send all of these cool people some emails.” There are some that we execute once a day and some that execute thousands per second.

      Managing this variety requires a reliably high-throughput message-passing technology. We use Celery's RabbitMQ implementation, and we stumbled upon a great feature called Federation that allows us to partition our task queue across any number of RabbitMQ servers and gives us the confidence that, if any single server gets backlogged, others will pitch in and distribute some of the backlogged tasks to their consumers.

      #MessageQueue

      See more
      Beanstalkd logo

      Beanstalkd

      111
      161
      74
      A simple, fast work queue
      111
      161
      + 1
      74
      PROS OF BEANSTALKD
      • 23
        Fast
      • 12
        Free
      • 12
        Does one thing well
      • 9
        Scalability
      • 8
        Simplicity
      • 3
        External admin UI developer friendly
      • 3
        Job delay
      • 2
        Job prioritization
      • 2
        External admin UI
      CONS OF BEANSTALKD
        Be the first to leave a con

        related Beanstalkd posts

        Frédéric MARAND
        Core Developer at OSInet · | 2 upvotes · 232.9K views

        I used Kafka originally because it was mandated as part of the top-level IT requirements at a Fortune 500 client. What I found was that it was orders of magnitude more complex ...and powerful than my daily Beanstalkd , and far more flexible, resilient, and manageable than RabbitMQ.

        So for any case where utmost flexibility and resilience are part of the deal, I would use Kafka again. But due to the complexities involved, for any time where this level of scalability is not required, I would probably just use Beanstalkd for its simplicity.

        I tend to find RabbitMQ to be in an uncomfortable middle place between these two extremities.

        See more
        RabbitMQ logo

        RabbitMQ

        21K
        18.5K
        527
        Open source multiprotocol messaging broker
        21K
        18.5K
        + 1
        527
        PROS OF RABBITMQ
        • 234
          It's fast and it works with good metrics/monitoring
        • 79
          Ease of configuration
        • 59
          I like the admin interface
        • 50
          Easy to set-up and start with
        • 21
          Durable
        • 18
          Intuitive work through python
        • 18
          Standard protocols
        • 10
          Written primarily in Erlang
        • 8
          Simply superb
        • 6
          Completeness of messaging patterns
        • 3
          Scales to 1 million messages per second
        • 3
          Reliable
        • 2
          Distributed
        • 2
          Supports MQTT
        • 2
          Better than most traditional queue based message broker
        • 2
          Supports AMQP
        • 1
          Clusterable
        • 1
          Clear documentation with different scripting language
        • 1
          Great ui
        • 1
          Inubit Integration
        • 1
          Better routing system
        • 1
          High performance
        • 1
          Runs on Open Telecom Platform
        • 1
          Delayed messages
        • 1
          Reliability
        • 1
          Open-source
        CONS OF RABBITMQ
        • 9
          Too complicated cluster/HA config and management
        • 6
          Needs Erlang runtime. Need ops good with Erlang runtime
        • 5
          Configuration must be done first, not by your code
        • 4
          Slow

        related RabbitMQ posts

        James Cunningham
        Operations Engineer at Sentry · | 18 upvotes · 1.7M views
        Shared insights
        on
        CeleryCeleryRabbitMQRabbitMQ
        at

        As Sentry runs throughout the day, there are about 50 different offline tasks that we execute—anything from “process this event, pretty please” to “send all of these cool people some emails.” There are some that we execute once a day and some that execute thousands per second.

        Managing this variety requires a reliably high-throughput message-passing technology. We use Celery's RabbitMQ implementation, and we stumbled upon a great feature called Federation that allows us to partition our task queue across any number of RabbitMQ servers and gives us the confidence that, if any single server gets backlogged, others will pitch in and distribute some of the backlogged tasks to their consumers.

        #MessageQueue

        See more

        Around the time of their Series A, Pinterest’s stack included Python and Django, with Tornado and Node.js as web servers. Memcached / Membase and Redis handled caching, with RabbitMQ handling queueing. Nginx, HAproxy and Varnish managed static-delivery and load-balancing, with persistent data storage handled by MySQL.

        See more
        Rake logo

        Rake

        56
        36
        0
        A software task management and build automation tool
        56
        36
        + 1
        0
        PROS OF RAKE
          Be the first to leave a pro
          CONS OF RAKE
            Be the first to leave a con

            related Rake posts

            JavaScript logo

            JavaScript

            353K
            268.6K
            8.1K
            Lightweight, interpreted, object-oriented language with first-class functions
            353K
            268.6K
            + 1
            8.1K
            PROS OF JAVASCRIPT
            • 1.7K
              Can be used on frontend/backend
            • 1.5K
              It's everywhere
            • 1.2K
              Lots of great frameworks
            • 897
              Fast
            • 745
              Light weight
            • 425
              Flexible
            • 392
              You can't get a device today that doesn't run js
            • 286
              Non-blocking i/o
            • 237
              Ubiquitousness
            • 191
              Expressive
            • 55
              Extended functionality to web pages
            • 49
              Relatively easy language
            • 46
              Executed on the client side
            • 30
              Relatively fast to the end user
            • 25
              Pure Javascript
            • 21
              Functional programming
            • 15
              Async
            • 13
              Full-stack
            • 12
              Setup is easy
            • 12
              Future Language of The Web
            • 12
              Its everywhere
            • 11
              Because I love functions
            • 11
              JavaScript is the New PHP
            • 10
              Like it or not, JS is part of the web standard
            • 9
              Expansive community
            • 9
              Everyone use it
            • 9
              Can be used in backend, frontend and DB
            • 9
              Easy
            • 8
              Most Popular Language in the World
            • 8
              Powerful
            • 8
              Can be used both as frontend and backend as well
            • 8
              For the good parts
            • 8
              No need to use PHP
            • 8
              Easy to hire developers
            • 7
              Agile, packages simple to use
            • 7
              Love-hate relationship
            • 7
              Photoshop has 3 JS runtimes built in
            • 7
              Evolution of C
            • 7
              It's fun
            • 7
              Hard not to use
            • 7
              Versitile
            • 7
              Its fun and fast
            • 7
              Nice
            • 7
              Popularized Class-Less Architecture & Lambdas
            • 7
              Supports lambdas and closures
            • 6
              It let's me use Babel & Typescript
            • 6
              Can be used on frontend/backend/Mobile/create PRO Ui
            • 6
              1.6K Can be used on frontend/backend
            • 6
              Client side JS uses the visitors CPU to save Server Res
            • 6
              Easy to make something
            • 5
              Clojurescript
            • 5
              Promise relationship
            • 5
              Stockholm Syndrome
            • 5
              Function expressions are useful for callbacks
            • 5
              Scope manipulation
            • 5
              Everywhere
            • 5
              Client processing
            • 5
              What to add
            • 4
              Because it is so simple and lightweight
            • 4
              Only Programming language on browser
            • 1
              Test
            • 1
              Hard to learn
            • 1
              Test2
            • 1
              Not the best
            • 1
              Easy to understand
            • 1
              Subskill #4
            • 1
              Easy to learn
            • 0
              Hard 彤
            CONS OF JAVASCRIPT
            • 22
              A constant moving target, too much churn
            • 20
              Horribly inconsistent
            • 15
              Javascript is the New PHP
            • 9
              No ability to monitor memory utilitization
            • 8
              Shows Zero output in case of ANY error
            • 7
              Thinks strange results are better than errors
            • 6
              Can be ugly
            • 3
              No GitHub
            • 2
              Slow

            related JavaScript posts

            Zach Holman

            Oof. I have truly hated JavaScript for a long time. Like, for over twenty years now. Like, since the Clinton administration. It's always been a nightmare to deal with all of the aspects of that silly language.

            But wowza, things have changed. Tooling is just way, way better. I'm primarily web-oriented, and using React and Apollo together the past few years really opened my eyes to building rich apps. And I deeply apologize for using the phrase rich apps; I don't think I've ever said such Enterprisey words before.

            But yeah, things are different now. I still love Rails, and still use it for a lot of apps I build. But it's that silly rich apps phrase that's the problem. Users have way more comprehensive expectations than they did even five years ago, and the JS community does a good job at building tools and tech that tackle the problems of making heavy, complicated UI and frontend work.

            Obviously there's a lot of things happening here, so just saying "JavaScript isn't terrible" might encompass a huge amount of libraries and frameworks. But if you're like me, yeah, give things another shot- I'm somehow not hating on JavaScript anymore and... gulp... I kinda love it.

            See more
            Conor Myhrvold
            Tech Brand Mgr, Office of CTO at Uber · | 44 upvotes · 10.9M views

            How Uber developed the open source, end-to-end distributed tracing Jaeger , now a CNCF project:

            Distributed tracing is quickly becoming a must-have component in the tools that organizations use to monitor their complex, microservice-based architectures. At Uber, our open source distributed tracing system Jaeger saw large-scale internal adoption throughout 2016, integrated into hundreds of microservices and now recording thousands of traces every second.

            Here is the story of how we got here, from investigating off-the-shelf solutions like Zipkin, to why we switched from pull to push architecture, and how distributed tracing will continue to evolve:

            https://eng.uber.com/distributed-tracing/

            (GitHub Pages : https://www.jaegertracing.io/, GitHub: https://github.com/jaegertracing/jaeger)

            Bindings/Operator: Python Java Node.js Go C++ Kubernetes JavaScript OpenShift C# Apache Spark

            See more
            Git logo

            Git

            292.1K
            175.1K
            6.6K
            Fast, scalable, distributed revision control system
            292.1K
            175.1K
            + 1
            6.6K
            PROS OF GIT
            • 1.4K
              Distributed version control system
            • 1.1K
              Efficient branching and merging
            • 959
              Fast
            • 845
              Open source
            • 726
              Better than svn
            • 368
              Great command-line application
            • 306
              Simple
            • 291
              Free
            • 232
              Easy to use
            • 222
              Does not require server
            • 27
              Distributed
            • 22
              Small & Fast
            • 18
              Feature based workflow
            • 15
              Staging Area
            • 13
              Most wide-spread VSC
            • 11
              Role-based codelines
            • 11
              Disposable Experimentation
            • 7
              Frictionless Context Switching
            • 6
              Data Assurance
            • 5
              Efficient
            • 4
              Just awesome
            • 3
              Github integration
            • 3
              Easy branching and merging
            • 2
              Compatible
            • 2
              Flexible
            • 2
              Possible to lose history and commits
            • 1
              Rebase supported natively; reflog; access to plumbing
            • 1
              Light
            • 1
              Team Integration
            • 1
              Fast, scalable, distributed revision control system
            • 1
              Easy
            • 1
              Flexible, easy, Safe, and fast
            • 1
              CLI is great, but the GUI tools are awesome
            • 1
              It's what you do
            • 0
              Phinx
            CONS OF GIT
            • 16
              Hard to learn
            • 11
              Inconsistent command line interface
            • 9
              Easy to lose uncommitted work
            • 7
              Worst documentation ever possibly made
            • 5
              Awful merge handling
            • 3
              Unexistent preventive security flows
            • 3
              Rebase hell
            • 2
              When --force is disabled, cannot rebase
            • 2
              Ironically even die-hard supporters screw up badly
            • 1
              Doesn't scale for big data

            related Git posts

            Simon Reymann
            Senior Fullstack Developer at QUANTUSflow Software GmbH · | 30 upvotes · 9.8M views

            Our whole DevOps stack consists of the following tools:

            • GitHub (incl. GitHub Pages/Markdown for Documentation, GettingStarted and HowTo's) for collaborative review and code management tool
            • Respectively Git as revision control system
            • SourceTree as Git GUI
            • Visual Studio Code as IDE
            • CircleCI for continuous integration (automatize development process)
            • Prettier / TSLint / ESLint as code linter
            • SonarQube as quality gate
            • Docker as container management (incl. Docker Compose for multi-container application management)
            • VirtualBox for operating system simulation tests
            • Kubernetes as cluster management for docker containers
            • Heroku for deploying in test environments
            • nginx as web server (preferably used as facade server in production environment)
            • SSLMate (using OpenSSL) for certificate management
            • Amazon EC2 (incl. Amazon S3) for deploying in stage (production-like) and production environments
            • PostgreSQL as preferred database system
            • Redis as preferred in-memory database/store (great for caching)

            The main reason we have chosen Kubernetes over Docker Swarm is related to the following artifacts:

            • Key features: Easy and flexible installation, Clear dashboard, Great scaling operations, Monitoring is an integral part, Great load balancing concepts, Monitors the condition and ensures compensation in the event of failure.
            • Applications: An application can be deployed using a combination of pods, deployments, and services (or micro-services).
            • Functionality: Kubernetes as a complex installation and setup process, but it not as limited as Docker Swarm.
            • Monitoring: It supports multiple versions of logging and monitoring when the services are deployed within the cluster (Elasticsearch/Kibana (ELK), Heapster/Grafana, Sysdig cloud integration).
            • Scalability: All-in-one framework for distributed systems.
            • Other Benefits: Kubernetes is backed by the Cloud Native Computing Foundation (CNCF), huge community among container orchestration tools, it is an open source and modular tool that works with any OS.
            See more
            Tymoteusz Paul
            Devops guy at X20X Development LTD · | 23 upvotes · 8.8M views

            Often enough I have to explain my way of going about setting up a CI/CD pipeline with multiple deployment platforms. Since I am a bit tired of yapping the same every single time, I've decided to write it up and share with the world this way, and send people to read it instead ;). I will explain it on "live-example" of how the Rome got built, basing that current methodology exists only of readme.md and wishes of good luck (as it usually is ;)).

            It always starts with an app, whatever it may be and reading the readmes available while Vagrant and VirtualBox is installing and updating. Following that is the first hurdle to go over - convert all the instruction/scripts into Ansible playbook(s), and only stopping when doing a clear vagrant up or vagrant reload we will have a fully working environment. As our Vagrant environment is now functional, it's time to break it! This is the moment to look for how things can be done better (too rigid/too lose versioning? Sloppy environment setup?) and replace them with the right way to do stuff, one that won't bite us in the backside. This is the point, and the best opportunity, to upcycle the existing way of doing dev environment to produce a proper, production-grade product.

            I should probably digress here for a moment and explain why. I firmly believe that the way you deploy production is the same way you should deploy develop, shy of few debugging-friendly setting. This way you avoid the discrepancy between how production work vs how development works, which almost always causes major pains in the back of the neck, and with use of proper tools should mean no more work for the developers. That's why we start with Vagrant as developer boxes should be as easy as vagrant up, but the meat of our product lies in Ansible which will do meat of the work and can be applied to almost anything: AWS, bare metal, docker, LXC, in open net, behind vpn - you name it.

            We must also give proper consideration to monitoring and logging hoovering at this point. My generic answer here is to grab Elasticsearch, Kibana, and Logstash. While for different use cases there may be better solutions, this one is well battle-tested, performs reasonably and is very easy to scale both vertically (within some limits) and horizontally. Logstash rules are easy to write and are well supported in maintenance through Ansible, which as I've mentioned earlier, are at the very core of things, and creating triggers/reports and alerts based on Elastic and Kibana is generally a breeze, including some quite complex aggregations.

            If we are happy with the state of the Ansible it's time to move on and put all those roles and playbooks to work. Namely, we need something to manage our CI/CD pipelines. For me, the choice is obvious: TeamCity. It's modern, robust and unlike most of the light-weight alternatives, it's transparent. What I mean by that is that it doesn't tell you how to do things, doesn't limit your ways to deploy, or test, or package for that matter. Instead, it provides a developer-friendly and rich playground for your pipelines. You can do most the same with Jenkins, but it has a quite dated look and feel to it, while also missing some key functionality that must be brought in via plugins (like quality REST API which comes built-in with TeamCity). It also comes with all the common-handy plugins like Slack or Apache Maven integration.

            The exact flow between CI and CD varies too greatly from one application to another to describe, so I will outline a few rules that guide me in it: 1. Make build steps as small as possible. This way when something breaks, we know exactly where, without needing to dig and root around. 2. All security credentials besides development environment must be sources from individual Vault instances. Keys to those containers should exist only on the CI/CD box and accessible by a few people (the less the better). This is pretty self-explanatory, as anything besides dev may contain sensitive data and, at times, be public-facing. Because of that appropriate security must be present. TeamCity shines in this department with excellent secrets-management. 3. Every part of the build chain shall consume and produce artifacts. If it creates nothing, it likely shouldn't be its own build. This way if any issue shows up with any environment or version, all developer has to do it is grab appropriate artifacts to reproduce the issue locally. 4. Deployment builds should be directly tied to specific Git branches/tags. This enables much easier tracking of what caused an issue, including automated identifying and tagging the author (nothing like automated regression testing!).

            Speaking of deployments, I generally try to keep it simple but also with a close eye on the wallet. Because of that, I am more than happy with AWS or another cloud provider, but also constantly peeking at the loads and do we get the value of what we are paying for. Often enough the pattern of use is not constantly erratic, but rather has a firm baseline which could be migrated away from the cloud and into bare metal boxes. That is another part where this approach strongly triumphs over the common Docker and CircleCI setup, where you are very much tied in to use cloud providers and getting out is expensive. Here to embrace bare-metal hosting all you need is a help of some container-based self-hosting software, my personal preference is with Proxmox and LXC. Following that all you must write are ansible scripts to manage hardware of Proxmox, similar way as you do for Amazon EC2 (ansible supports both greatly) and you are good to go. One does not exclude another, quite the opposite, as they can live in great synergy and cut your costs dramatically (the heavier your base load, the bigger the savings) while providing production-grade resiliency.

            See more