Prometheus

Since the beginning, Cal Henderson has been the CTO of Slack. Earlier this year, he commented on a Quora question summarizing their current stack.

• Web: a mix of JavaScript/ES6 and React.
• Desktop: And Electron to ship it as a desktop application.
• Android: a mix of Java and Kotlin.
• iOS: written in a mix of Objective C and Swift.

• The core application and the API written in PHP/Hack that runs on HHVM.
• The data is stored in MySQL using Vitess.
• Caching is done using Memcached and MCRouter.
• The search service takes help from SolrCloud, with various Java services.
• The messaging system uses WebSockets with many services in Java and Go.
• Load balancing is done using HAproxy with Consul for configuration.
• Most services talk to each other over gRPC,
• Some Thrift and JSON-over-HTTP
• Voice and video calling service was built in Elixir.

• Built using open source tools including Presto, Spark, Airflow, Hadoop and Kafka.

• For server configuration and management we use Terraform, Chef and Kubernetes.
• We use Prometheus for time series metrics and ELK for logging.

At Kong while building an internal tool, we struggled to route metrics to Prometheus and logs to Logstash without incurring too much latency in our metrics collection.

We replaced nginx with OpenResty on the edge of our tool which allowed us to use the lua-nginx-module to run Lua code that captures metrics and records telemetry data during every request’s log phase. Our code then pushes the metrics to a local aggregator process (written in Go) which in turn exposes them in Prometheus Exposition Format for consumption by Prometheus. This solution reduced the number of components we needed to maintain and is fast thanks to NGINX and LuaJIT.

Why we spent several years building an open source, large-scale metrics alerting system, M3, built for Prometheus:

By late 2014, all services, infrastructure, and servers at Uber emitted metrics to a Graphite stack that stored them using the Whisper file format in a sharded Carbon cluster. We used Grafana for dashboarding and Nagios for alerting, issuing Graphite threshold checks via source-controlled scripts. While this worked for a while, expanding the Carbon cluster required a manual resharding process and, due to lack of replication, any single node’s disk failure caused permanent loss of its associated metrics. In short, this solution was not able to meet our needs as the company continued to grow.

To ensure the scalability of Uber’s metrics backend, we decided to build out a system that provided fault tolerant metrics ingestion, storage, and querying as a managed platform...

https://eng.uber.com/m3/

(GitHub : https://github.com/m3db/m3)

We recently implemented Thanos alongside Prometheus into our Kubernetes clusters, we had previously used a variety of different metrics systems and we wanted to make life simpler for everyone by just picking one.

Prometheus seemed like an obvious choice due to its powerful querying language, native Kubernetes support and great community. However we found it somewhat lacking when it came to being highly available, something that would be very important if we wanted this to be the single source of all our metrics.

Thanos came along and solved a lot of these problems. It allowed us to run multiple Prometheis without duplicating metrics, query multiple Prometheus clusters at once, and easily back up data and then query it. Now we have a single place to go if you want to view metrics across all our clusters, with many layers of redundancy to make sure this monitoring solution is as reliable and resilient as we could reasonably make it.

If you're interested in a bit more detail feel free to check out the blog I wrote on the subject that's linked.

We primarily use Prometheus to gather metrics and statistics to display them in Grafana. Prometheus

We primarily use Prometheus to gather metrics and statistics to display them in Grafana. Aside from that we poll Prometheus for our orchestration-solution "JCOverseer" to determine, which host is least occupied at the moment.

While there are existing orchestration softwares/suites like Kubernetes, that we also plan to adopt in the future, we're of the opinion that those solutions do not fit our special environment within minecraft and our own solution will outperform them in the limited scope that it needs to cover.

Gather metrics from systems and applications. Evaluate alerting rules. Alerts are pushed to OpsGenie and Slack. Prometheus