What is ELK and what are its top alternatives?
ELK, which stands for Elasticsearch, Logstash, and Kibana, is a popular open-source stack used for log management and analytics. Elasticsearch is a distributed, RESTful search and analytics engine capable of storing and searching large volumes of data in near real-time. Logstash is a flexible data collection, processing, and pipeline tool that can ingest data from multiple sources. Kibana is a data visualization platform that allows users to explore, analyze, and visualize data stored in Elasticsearch. Key features of ELK include scalability, flexibility, and a strong community support. However, setting up and maintaining the ELK stack can be complex for beginners, and it may require considerable resources in terms of hardware and maintenance.
- Grafana: Grafana is a multi-platform open-source analytics and visualization platform that allows users to query, visualize, alert on, and understand data. Key features include support for multiple data sources, high flexibility in creating dashboards, and a large plugin ecosystem. Pros of Grafana compared to ELK include easier setup and a more user-friendly interface, while a potential con could be less robust log management capabilities.
- Splunk: Splunk is a proprietary software platform for searching, monitoring, and analyzing machine-generated data. Key features include real-time visibility, machine learning capabilities, and a user-friendly interface. Pros of Splunk compared to ELK include comprehensive out-of-the-box features and ease of use, while a potential con could be higher cost for enterprise use.
- Sumo Logic: Sumo Logic is a cloud-based log management and analytics service that offers real-time insights into application and infrastructure logs. Key features include easy scalability, machine learning-based analytics, and centralized log management. Pros of Sumo Logic compared to ELK include ease of use and maintenance, while a potential con could be limited customization options.
- Logz.io: Logz.io is a cloud-native observability platform that offers log management, monitoring, and security analytics. Key features include AI-powered log analysis, built-in integrations with popular tools, and support for Kubernetes monitoring. Pros of Logz.io compared to ELK include ease of setup and management, while a potential con could be higher cost for enterprise features.
- Graylog: Graylog is an open-source log management platform that allows users to collect, index, and analyze log data in real-time. Key features include a robust search functionality, alerting capabilities, and support for plugins. Pros of Graylog compared to ELK include easier setup and configuration, while a potential con could be a less polished user interface.
- Fluentd: Fluentd is an open-source data collector that allows users to unify log data collection and consumption for better use and understanding of data. Key features include flexibility in data source inputs and outputs, plugins for various data sources, and easy integration with other tools. Pros of Fluentd compared to ELK include lightweight resource usage and high performance, while a potential con could be a steeper learning curve for beginners.
- Loggly: Loggly is a cloud-based log management service that offers real-time log analysis, monitoring, and alerting. Key features include centralized log data storage, searchable log data, and customizable dashboards. Pros of Loggly compared to ELK include ease of use and setup, while a potential con could be limited customization options.
- Datadog: Datadog is a cloud monitoring and analytics platform that provides visibility into the performance of applications, infrastructure, and logs. Key features include integrations with over 400 technologies, alerting capabilities, and customizable dashboards. Pros of Datadog compared to ELK include a centralized platform for monitoring and logs, while a potential con could be higher cost for enterprise features.
- Scalyr: Scalyr is a log management and observability platform that offers high-speed log search and analysis. Key features include advanced search capabilities, real-time log monitoring, and integrations with popular tools. Pros of Scalyr compared to ELK include high performance and speed in log processing, while a potential con could be limited scalability options.
- Logstash-Forwarder: Logstash-Forwarder, now known as Filebeat, is a lightweight log shipper for forwarding and centralizing log data. Key features include scalability, extensibility, and support for multiple data sources. Pros of Logstash-Forwarder compared to ELK include easy integration with Elasticsearch and Kibana, while a potential con could be limited data processing capabilities.
Top Alternatives to ELK
- Datadog
Datadog is the leading service for cloud-scale monitoring. It is used by IT, operations, and development teams who build and operate applications that run on dynamic or hybrid cloud infrastructure. Start monitoring in minutes with Datadog! ...
- Splunk
It provides the leading platform for Operational Intelligence. Customers use it to search, monitor, analyze and visualize machine data. ...
- Graylog
Centralize and aggregate all your log files for 100% visibility. Use our powerful query language to search through terabytes of log data to discover and analyze important information. ...
- Logstash
Logstash is a tool for managing events and logs. You can use it to collect logs, parse them, and store them for later use (like, for searching). If you store them in Elasticsearch, you can view and analyze them with Kibana. ...
- SLF4J
It is a simple Logging Facade for Java (SLF4J) serves as a simple facade or abstraction for various logging frameworks allowing the end user to plug in the desired logging framework at deployment time. ...
- Logback
It is intended as a successor to the popular log4j project. It is divided into three modules, logback-core, logback-classic and logback-access. The logback-core module lays the groundwork for the other two modules, logback-classic natively implements the SLF4J API so that you can readily switch back and forth between logback and other logging frameworks and logback-access module integrates with Servlet containers, such as Tomcat and Jetty, to provide HTTP-access log functionality. ...
- Papertrail
Papertrail helps detect, resolve, and avoid infrastructure problems using log messages. Papertrail's practicality comes from our own experience as sysadmins, developers, and entrepreneurs. ...
- Fluentd
Fluentd collects events from various data sources and writes them to files, RDBMS, NoSQL, IaaS, SaaS, Hadoop and so on. Fluentd helps you unify your logging infrastructure. ...
ELK alternatives & related posts
Datadog
- Monitoring for many apps (databases, web servers, etc)137
- Easy setup107
- Powerful ui87
- Powerful integrations83
- Great value70
- Great visualization54
- Events + metrics = clarity46
- Custom metrics41
- Notifications41
- Flexibility39
- Free & paid plans19
- Great customer support16
- Makes my life easier15
- Adapts automatically as i scale up10
- Easy setup and plugins9
- Super easy and powerful8
- AWS support7
- In-context collaboration7
- Rich in features6
- Docker support5
- Cost4
- Source control and bug tracking4
- Automation tools4
- Cute logo4
- Monitor almost everything4
- Full visibility of applications4
- Simple, powerful, great for infra4
- Easy to Analyze4
- Best than others4
- Expensive3
- Best in the field3
- Free setup3
- Good for Startups3
- APM2
- Expensive19
- No errors exception tracking4
- External Network Goes Down You Wont Be Logging2
- Complicated1
related Datadog posts
Our primary source of monitoring and alerting is Datadog. We’ve got prebuilt dashboards for every scenario and integration with PagerDuty to manage routing any alerts. We’ve definitely scaled past the point where managing dashboards is easy, but we haven’t had time to invest in using features like Anomaly Detection. We’ve started using Honeycomb for some targeted debugging of complex production issues and we are liking what we’ve seen. We capture any unhandled exceptions with Rollbar and, if we realize one will keep happening, we quickly convert the metrics to point back to Datadog, to keep Rollbar as clean as possible.
We use Segment to consolidate all of our trackers, the most important of which goes to Amplitude to analyze user patterns. However, if we need a more consolidated view, we push all of our data to our own data warehouse running PostgreSQL; this is available for analytics and dashboard creation through Looker.
Hey there! We are looking at Datadog, Dynatrace, AppDynamics, and New Relic as options for our web application monitoring.
Current Environment: .NET Core Web app hosted on Microsoft IIS
Future Environment: Web app will be hosted on Microsoft Azure
Tech Stacks: IIS, RabbitMQ, Redis, Microsoft SQL Server
Requirement: Infra Monitoring, APM, Real - User Monitoring (User activity monitoring i.e., time spent on a page, most active page, etc.), Service Tracing, Root Cause Analysis, and Centralized Log Management.
Please advise on the above. Thanks!
- API for searching logs, running reports3
- Alert system based on custom query results3
- Dashboarding on any log contents2
- Custom log parsing as well as automatic parsing2
- Ability to style search results into reports2
- Query engine supports joining, aggregation, stats, etc2
- Splunk language supports string, date manip, math, etc2
- Rich GUI for searching live logs2
- Query any log as key-value pairs1
- Granular scheduling and time window support1
- Splunk query language rich so lots to learn1
related Splunk posts
I use Kibana because it ships with the ELK stack. I don't find it as powerful as Splunk however it is light years above grepping through log files. We previously used Grafana but found it to be annoying to maintain a separate tool outside of the ELK stack. We were able to get everything we needed from Kibana.
We are currently exploring Elasticsearch and Splunk for our centralized logging solution. I need some feedback about these two tools. We expect our logs in the range of upwards > of 10TB of logging data.
- Open source19
- Powerfull13
- Well documented8
- Alerts6
- User authentification5
- Flexibel query and parsing language5
- User management3
- Easy query language and english parsing3
- Alerts and dashboards3
- Easy to install2
- A large community1
- Manage users and permissions1
- Free Version1
- Does not handle frozen indices at all1
related Graylog posts
- Free69
- Easy but powerful filtering18
- Scalable12
- Kibana provides machine learning based analytics to log2
- Great to meet GDPR goals1
- Well Documented1
- Memory-intensive4
- Documentation difficult to use1
related Logstash posts
Often enough I have to explain my way of going about setting up a CI/CD pipeline with multiple deployment platforms. Since I am a bit tired of yapping the same every single time, I've decided to write it up and share with the world this way, and send people to read it instead ;). I will explain it on "live-example" of how the Rome got built, basing that current methodology exists only of readme.md and wishes of good luck (as it usually is ;)).
It always starts with an app, whatever it may be and reading the readmes available while Vagrant and VirtualBox is installing and updating. Following that is the first hurdle to go over - convert all the instruction/scripts into Ansible playbook(s), and only stopping when doing a clear vagrant up
or vagrant reload
we will have a fully working environment. As our Vagrant environment is now functional, it's time to break it! This is the moment to look for how things can be done better (too rigid/too lose versioning? Sloppy environment setup?) and replace them with the right way to do stuff, one that won't bite us in the backside. This is the point, and the best opportunity, to upcycle the existing way of doing dev environment to produce a proper, production-grade product.
I should probably digress here for a moment and explain why. I firmly believe that the way you deploy production is the same way you should deploy develop, shy of few debugging-friendly setting. This way you avoid the discrepancy between how production work vs how development works, which almost always causes major pains in the back of the neck, and with use of proper tools should mean no more work for the developers. That's why we start with Vagrant as developer boxes should be as easy as vagrant up
, but the meat of our product lies in Ansible which will do meat of the work and can be applied to almost anything: AWS, bare metal, docker, LXC, in open net, behind vpn - you name it.
We must also give proper consideration to monitoring and logging hoovering at this point. My generic answer here is to grab Elasticsearch, Kibana, and Logstash. While for different use cases there may be better solutions, this one is well battle-tested, performs reasonably and is very easy to scale both vertically (within some limits) and horizontally. Logstash rules are easy to write and are well supported in maintenance through Ansible, which as I've mentioned earlier, are at the very core of things, and creating triggers/reports and alerts based on Elastic and Kibana is generally a breeze, including some quite complex aggregations.
If we are happy with the state of the Ansible it's time to move on and put all those roles and playbooks to work. Namely, we need something to manage our CI/CD pipelines. For me, the choice is obvious: TeamCity. It's modern, robust and unlike most of the light-weight alternatives, it's transparent. What I mean by that is that it doesn't tell you how to do things, doesn't limit your ways to deploy, or test, or package for that matter. Instead, it provides a developer-friendly and rich playground for your pipelines. You can do most the same with Jenkins, but it has a quite dated look and feel to it, while also missing some key functionality that must be brought in via plugins (like quality REST API which comes built-in with TeamCity). It also comes with all the common-handy plugins like Slack or Apache Maven integration.
The exact flow between CI and CD varies too greatly from one application to another to describe, so I will outline a few rules that guide me in it: 1. Make build steps as small as possible. This way when something breaks, we know exactly where, without needing to dig and root around. 2. All security credentials besides development environment must be sources from individual Vault instances. Keys to those containers should exist only on the CI/CD box and accessible by a few people (the less the better). This is pretty self-explanatory, as anything besides dev may contain sensitive data and, at times, be public-facing. Because of that appropriate security must be present. TeamCity shines in this department with excellent secrets-management. 3. Every part of the build chain shall consume and produce artifacts. If it creates nothing, it likely shouldn't be its own build. This way if any issue shows up with any environment or version, all developer has to do it is grab appropriate artifacts to reproduce the issue locally. 4. Deployment builds should be directly tied to specific Git branches/tags. This enables much easier tracking of what caused an issue, including automated identifying and tagging the author (nothing like automated regression testing!).
Speaking of deployments, I generally try to keep it simple but also with a close eye on the wallet. Because of that, I am more than happy with AWS or another cloud provider, but also constantly peeking at the loads and do we get the value of what we are paying for. Often enough the pattern of use is not constantly erratic, but rather has a firm baseline which could be migrated away from the cloud and into bare metal boxes. That is another part where this approach strongly triumphs over the common Docker and CircleCI setup, where you are very much tied in to use cloud providers and getting out is expensive. Here to embrace bare-metal hosting all you need is a help of some container-based self-hosting software, my personal preference is with Proxmox and LXC. Following that all you must write are ansible scripts to manage hardware of Proxmox, similar way as you do for Amazon EC2 (ansible supports both greatly) and you are good to go. One does not exclude another, quite the opposite, as they can live in great synergy and cut your costs dramatically (the heavier your base load, the bigger the savings) while providing production-grade resiliency.
Hi everyone. I'm trying to create my personal syslog monitoring.
To get the logs, I have uncertainty to choose the way: 1.1 Use Logstash like a TCP server. 1.2 Implement a Go TCP server.
To store and plot data. 2.1 Use Elasticsearch tools. 2.2 Use InfluxDB and Grafana.
I would like to know... Which is a cheaper and scalable solution?
Or even if there is a better way to do it.
related SLF4J posts
related Logback posts
- Log search85
- Easy log aggregation across multiple machines43
- Integrates with Heroku43
- Simple interface37
- Backup to S326
- Easy setup, independent of existing logging setup19
- Heroku add-on15
- Command line interface3
- Alerting1
- Good for Startups1
- Expensive2
- External Network Goes Down You Wont Be Logging1
related Papertrail posts
Logentries, LogDNA, Timber.io, Papertrail and Sumo Logic provide free pricing plan for #Heroku application. You can add these applications as add-ons very easily.
- Open-source11
- Great for Kubernetes node container log forwarding9
- Lightweight9
- Easy8