How LaunchDarkly Serves Over 4 Billion Feature Flags Daily

14,675
LaunchDarkly
Serving over 200 billion feature flags daily to help software teams build better software, faster. LaunchDarkly helps eliminate risk for developers and operations teams from the software development cycle.

Editor's note: By John Kodumal, CTO, LaunchDarkly



LaunchDarkly Platform


Background

Feature flagging (wrapping a feature in a flag that’s controlled outside of deployment) is a technique for effective continuous delivery. For example, you can wrap a new signup form in a feature flag and then control which users see that form, all without having to redeploy code or modify a database. Engineering-driven companies (think Google, Facebook, Twitter) invest heavily in custom-built feature flag management systems to roll features out to whom they want, when they want. Smaller companies build and maintain their own feature flagging infrastructure or using simple open source projects that often don't even have a UI. I was previously an engineering manager at Atlassian, where I’d seen a team work on an internal feature flagging system, so I was aware of the complexity of the problem and the investment required to build a product that addressed the needs of larger development teams and enterprises. That’s where we saw an opportunity to start LaunchDarkly.


LaunchDarkly Platform


We're currently serving over 4 billion feature flag requests per day for companies like Microsoft, Atlassian, Ten-X, and CircleCI. Many of our customers report that we’ve changed the way they do development-- we de-risk new feature launches, eliminate the need for painful long-lived branches, and empower product managers, QA, and others to use feature flags to improve their users’ experience.

General Architecture

You can think of LaunchDarkly as being split up into three pieces: a monolithic web application, a streaming API that serves feature flags, and an analytics processing pipeline that's structured as a set of microservices. We've written almost all of this in Go.

Go has really worked well for us. We love that our services compile from scratch in seconds, and produce small statically linked binaries that can be deployed easily and run in a small footprint. I'd done a lot with Scala at Atlassian, but I'd grown frustrated with the slow compilation times and overhead of the JVM. Our monolith has about a 6MB memory footprint— try that on the JVM!

I'm generally not a fan of large web frameworks like Django or Rails. Too much "magic" for me. I prefer to build on top of smaller libraries that serve specific needs. To that end, both our monolith and our microservices rely heavily on a home-built framework layer that uses libraries like Gorilla Mux.

Our framework makes it trivial to add a new resource to our REST API and get a ton of essential functionality out of the box-- with a few lines of code, you get authentication, APM with New Relic, metrics pumped to Graphite, CORS support, and more.

The web application monolith has a pretty standard architecture. Some of the technologies we use include:

  • MongoDB -- as our core application data store. It's popular to make fun of Mongo these days, but we've found it to be a great database technology as long as you don't store too many things in it. Anything you can count on your fingers and toes should be fine.
  • ElasticSearch -- handles user search and segmentation.
  • Redis -- caching, of course.
  • HAProxy -- as a load balancer.


LaunchDarkly Architecture


Serving feature flags, fast

One of the cool and novel parts of LaunchDarkly is our streaming architecture, which allows us to serve feature flag changes instantly. Think of it like a real-time, in-memory database containing feature flag settings. The closest comparison would be something like Firebase, except Firebase is really more focused on the client-side web and mobile, whereas we do that and the server-side.

We use several technologies to drive our streaming API. The most important is Pushpin / Fanout. These technologies abstract us away from managing these long-lived streaming connections and focus on building simple REST APIs.

We also use Fastly as a CDN. Fastly is perfect for us-- we can use VCL to write custom caching rules, and can purge content in milliseconds. If you're caching dynamic content (as opposed to say cat GIFs), or you find yourself needing to purge content programmatically, or you want the flexibility of Varnish in addition to the global network of POPs a CDN can provide, Fastly is the best choice out there. Their support team is also fantastic.

When assembled together, these technologies allow our customers to change their feature flag settings on our dashboard and have their new rollout settings streamed to thousands of servers in a hundred milliseconds or less.

Analytics at scale

The other huge component of LaunchDarkly is our analytics processing pipeline. Our customers request over 4 billion feature flags per day, and we use analytics data from these requests to power a lot of the features in our product. A/B testing is an obvious example, but we also do things like determine when a feature flag has stopped being requested, so that you can manage technical debt and clean up old flags.

Our current pipeline involves an HTTP microservice that writes analytics data to DynamoDB. If we need to do any further processing (say, for A/B testing), then we enqueue another job into SQS. Another microservice reads jobs off of the SQS queue and processes them. Right now, we're actively evolving this pipeline. We've found that when we're under heavy load, we need to buffer calls to DynamoDB while we expand capacity instead of trying to process them immediately. Kafka is perfect for this-- so we're splitting that HTTP microservice into a smaller HTTP service that simply queues events to Kafka, and another service that processes Kafka queues.

We actually use LaunchDarkly to control this evolution. We have a feature flag that controls whether a request goes through our old analytics pipeline, or the new Kafka-based pipeline we're rolling out. Once the new pipeline is enabled for all customers, we can clean up the code and switch over completely to the Kafka pipeline. This is a use case that surprises a lot of customers-- they think of feature flags in terms of controlling user-visible features (release toggles), but they are extremely valuable for other use cases like ops toggles, experiments, and permission management.

LaunchDarkly Platform

As we scaled this service out to handle tens of thousands of request per second, we learned an important lesson about microservice construction. When we first built many of these services, we thought in terms of building a separate service per concern. For example, we’d build a service that would read in analytics events and serve the autocomplete functionality on the site. The web application would make a sub-request to this service when it had an autocomplete request from the site.

We quickly learned that the need for fault tolerance and isolation trumps the conceptual neatness of having a service per concern. With fault tolerance in mind, we sliced our services along a different axis-- separating high-throughput analytics writes from the lower-volume read requests coming from the site. This shift dramatically improved the performance of our site, as well as our ability to evolve and scale the huge write load we see on the analytics side.

Infrastructure

As you might have inferred, we use AWS as our hosting provider. We’re fairly conservative when it comes to adopting new technologies-- deployment for us consists of a set of Ansible scripts that spin up EC2 boxes for our various services. We don’t yet use ECS or Docker containers-- which by extension means we don’t use anything for container orchestration. A long while back, we spiked a migration to Mesosphere but we ran into enough issues that we didn’t proceed forward. We do think that these technologies are the future, but that future is not now, at least for us.

So maturity is one issue that prevents us from adopting some of the latest whiz-bang ops technology. There are other technologies that we find interesting, like Amazon’s API Gateway but the pricing models just don’t work for us-- at tens of thousands of requests per second, they’re non-starters.

Other services

For customer communications and support, we use Intercom, Slack, and GrooveHQ. We also recently started using elevio, and we've found it's a great way to turn Intercom questions into trackable support tickets.

We use ReadMe.io for our product and developer API documentation, GitHub holds all our code hostage, and CircleCI helps us integrate continuously.

What’s next?

We’re constantly evolving our service to improve efficiency and scale. Besides the Kafka switchover, we’re looking at using Cassandra for some of the work that DynamoDB is doing right now. We also are keenly interested in Disque as a queuing solution, especially because we’ve had so much positive experience with Redis.

More aspirationally, we might try spiking some of our new services in Rust. I’m a functional programmer at heart, and while I am appreciative of the speed and tooling around Go, it would be nice to regain some of the expressiveness and elegance of a functional language while retaining what we like about Go (the fast compilation times, ease of deployment). If we do try it out, we’ll do so in a cautious manner, and isolate the trial to a new microservice somewhere.

LaunchDarkly
Serving over 200 billion feature flags daily to help software teams build better software, faster. LaunchDarkly helps eliminate risk for developers and operations teams from the software development cycle.
Tools mentioned in article
Open jobs at LaunchDarkly
Application Engineer
- US

As a SaaSOps Engineer, you are passionate about learning and automation technologies. You will work closely with your IT engineering peers to identify bottlenecks, improve system efficiency by reducing manual tasks through automation and ensure our core systems are designed to scale as our team grows. 

The ideal candidate is an excellent communicator and fast learner who is looking for a supportive environment to continue to grow their technical skills and expertise.

You will be a stakeholder in the discovery, procurement, implementation and management of our growing corporate infrastructure. You have a passion for innovation and creative problem solving, as well as the ability to be adaptable to the constantly changing technology.

LaunchDarkly is a rapidly growing software company with a strong mission and vision carried out by a talented and diverse team of employees. Our goal is to help teams build better software, faster. You'll join a small team from companies like Atlassian, Intercom, and GitHub, and you'll have an immediate impact with our product and customers.

What you'll do:

  • Perform day-to-day administrative tasks on SaaS applications within our stack.
  • Handle technical escalations, provide timely resolution of problems or engage lead staff for assistance.
  • Support SSO implementations using Okta.
  • Identify ways to improve efficiency and scale services leveraging tools and automation.
  • Build workflows to enable seamless and secure access to applications that support our internal customers.
  • Help build, scale, and maintain system monitoring and alerting tools
  • Compose scripts in an administrative language (Python, Ruby, or Shell)
  • Customize, create, and package software, scripts, and applications for deployment. 
  • Collaborate with cross functional teams to solve business problems.
  • Create IT systems’ compliance policies and provide compliance proof for audits.
  • Knowledge of SSH, keystores, security certificates, user and password management, Single-Sign-On integration, and authorization tokens.
  • Document and update processes and procedures for supporting applications across the org.
  • Drive tool adoption and proficiency through training and documentation.
  • Support initiatives to continuously improve, integrate, and consolidate systems company-wide

Who you are:

  • Experience with SaaS application management including: Okta, GSuite, Slack and AWS
  • Project management, system implementation, and system integration experience
  • Scripting expertise with GAM, Python, TerraForm, JSON, Perl, Powershell or equivalent tools
  • Proficient with API integrations
  • JAMF administration, package creation, policy scripting
  • Strong analytical and problem-solving skills; ability to work creatively in a problem-solving environment
  • Ability to balance technical skills with business savvy – ability to transform end-user needs into technical and functional requirements

You may also know: (Bonus Skills)

  • Experience with Terraform, Docker, Container Management, and all things AWS

About LaunchDarkly:

LaunchDarkly is a Feature Management Platform that serves hundreds of billions of feature flags daily to help software teams build better software, faster. Feature flagging is an industry standard methodology of wrapping a new or risky section of code or infrastructure change with a flag. Each flag can easily be turned off independent of code deployment (aka "dark launching"). LaunchDarkly has SDKs for all major web and mobile platforms. We are building a diverse team so that we can offer robust products and services. Our team culture is dynamic, friendly, and supportive. Our headquarters are in Oakland.

At LaunchDarkly, we believe in the power of teams. We're building a team that is humble, open, collaborative, respectful and kind. We are an equal opportunity employer and value diversity at our company. We do not discriminate on the basis of race, religion, color, national origin, gender, gender identity, sexual orientation, age, marital status, veteran status, or disability status.

Don't let the confidence gap get in the way of applying! We'd love to hear from you.

We've partnered with KeyValues to help demonstrate the amazing culture we've built here at LaunchDarkly, find more info at https://www.keyvalues.com/launchdarkly

Backend Engineer
- US
As a Backend Engineer, you will help us build features, design and implement API methods, and improve the performance and reliability of our systems. We're looking for someone who knows what it takes to deliver value to customers and takes pride in the quality of their work.

Our platform serves over twenty billion feature flags daily. The core technologies we use daily include Golang, MongoDB, ElasticSearch, Redis, and NATS. As part of our you-build-it-you-run-it culture, all developers may be responsible of support applications in production, including on-call. On call is compensated in addition.

What you'll get to do: 

  • Build and expand our APIs and services, written in Go
  • Collaborate with frontend engineers to deliver user-facing features
  • Monitor and improve server-side performance
  • Write unit, integration, and load tests as necessary
  • Actively participate in code reviews
  • Write and review technical proposals
  • Improve engineering standards, tooling, and processes
On day one, you should have:
  • Proven experience and fluency with server-side web development (e.g. in Java / Scala, Ruby, Python, Golang, Node.js)
  • Experience building RESTful APIs
  • Strong computer science fundamentals: data structures, distributed systems, concurrency, and threading
  • Strong communication skills, a positive attitude, and empathy
  • You write code that can be easily understood by others, with an eye towards maintainability
  • You hold yourself and others to a high bar when working with production systems
  • You value high code quality, automated testing, and other engineering best practices

Bonus Points:

  • Experience with NoSQL databases (MongoDB, ElasticSearch)
  • A deep understanding of networking technologies (TCP, HTTP, websockets, server-sent events, etc.)

About LaunchDarkly:

LaunchDarkly is a Feature Management Platform that serves hundreds of billions of feature flags daily to help software teams build better software, faster. Feature flagging is an industry standard methodology of wrapping a new or risky section of code or infrastructure change with a flag. Each flag can easily be turned off independent of code deployment (aka "dark launching"). LaunchDarkly has SDKs for all major web and mobile platforms. We are building a diverse team so that we can offer robust products and services. Our team culture is dynamic, friendly, and supportive. Our headquarters are in Oakland.

At LaunchDarkly, we believe in the power of teams. We're building a team that is humble, open, collaborative, respectful and kind. We are an equal opportunity employer and value diversity at our company. We do not discriminate on the basis of race, religion, color, national origin, gender, gender identity, sexual orientation, age, marital status, veteran status, or disability status.

Don't let the confidence gap get in the way of applying! We'd love to hear from you.

We've partnered with KeyValues to help demonstrate the amazing culture we've built here at LaunchDarkly, find more info at https://www.keyvalues.com/launchdarkly

 

Demo Engineer
Oakland, California, United States

As a Demo Engineer, you will be in a unique position to build the foundation and strategy for how we visually articulate our technology to our prospects and customers. You will be responsible for building and maintaining the demo environments that the Revenue organization uses during a deal cycle and will directly impact every single opportunity the team works on. You will be working with the latest technologies in the space, allowing you to use your creativity to craft compelling storylines that speak to the value proposition of our solution. Collaboration will be key in this role and will require you to work closely with our amazing SEs, PMs, Engineers, and Marketing to ensure that we are building a strong and relevant message.

LaunchDarkly is a rapidly growing software company with a strong mission and vision carried out by a talented and diverse team of employees. Our goal is to help teams build better software, faster. 

Software powers the world and LaunchDarkly empowers all teams to deliver and control their software.

Responsibilities:

  • Solicit feedback from SEs, Product, Marketing, and our customers to initially understand the current state of our demo product & process, and continuously drive this feedback loop to ensure we’re constantly adapting to our changing product and market
  • Own developing the strategy for all things demo-related and driving priorities within that strategy to align to the highest impact gaps
  • Become a LaunchDarkly expert understanding our SDKs and platform in details, as well as the growing number of integrations to tell a rich end to end story around feature management
  • Collaborate closely with Product Managers as we release new product features to guarantee that our demo experience highlights new functionality in LaunchDarkly's platform as it is available
  • Write and maintain documentation on how the demo environments work in-depth and how to use it

Basic Qualifications:

  • 2+ years of experience in a demo org, technically oriented, or pre-sales role: Demo Engineer, Solutions Engineer, Solutions Consultant, or similar experience in an engineering role
  • Programming experience required, preferably in Java, .NET. GO and/or python
  • Ability to learn and synthesize large amounts of information with little context
  • A self‐starter and problem solver, willing to take on hard problems and work independently when necessary.
  • Superior communications skills (presentation, written, and verbal) and demonstrated ability to communicate/present/motivate effectively across technical and selling teams
  • Technical proficiency creating and maintaining demo environment

Preferred Qualifications:

  • Experience working with teams that underwent development process transformation
  • Experience with data persistence technologies like Varnish or Redis
  • Full-stack and mobile development experience

About LaunchDarkly:

LaunchDarkly is a Feature Management Platform that serves hundreds of billions of feature flags daily to help software teams build better software, faster. Feature flagging is an industry standard methodology of wrapping a new or risky section of code or infrastructure change with a flag. Each flag can easily be turned off independent of code deployment (aka "dark launching"). LaunchDarkly has SDKs for all major web and mobile platforms. We are building a diverse team so that we can offer robust products and services. Our team culture is dynamic, friendly, and supportive. Our headquarters are in Oakland.

At LaunchDarkly, we believe in the power of teams. We're building a team that is humble, open, collaborative, respectful and kind. We are an equal opportunity employer and value diversity at our company. We do not discriminate on the basis of race, religion, color, national origin, gender, gender identity, sexual orientation, age, marital status, veteran status, or disability status.

Don't let the confidence gap get in the way of applying! We'd love to hear from you.

We've partnered with KeyValues to help demonstrate the amazing culture we've built here at LaunchDarkly, find more info at https://www.keyvalues.com/launchdarkly.

Developer Advocate
Oakland, California, United States

As a Developer Advocate, you will join our Developer Marketing team, writing, speaking, and creating other forms of content to help expand the awareness of Feature Management. Our team helps make developers more successful by improving their software development practices. The content we provide covers both technical as well as human-centered topics. We are looking to expand our team with a Developer Advocate whose primary focus is writing but does not hesitate to take the stage (virtually or eventually in-person).

Responsibilities:

  • Write, speak, and create content about technology trends with the goal of engaging our developers, developer managers, and senior technical leaders.
  • You will drive awareness about best practices for feature management and modern application architecture with the goal of influencing customer success.
  • Develop a persona as a trusted advisor in Feature Management and modern application development trends.

About You:

  • You have passion, curiosity, technical depth, and extraordinary written and verbal communication skills.
  • You are a creative and skilled storyteller.
  • You are able to converse with a broad range of programming language communities (Java, .NET, Node.js, Python, Ruby, iOS, Android, etc.), and have a real passion for modern application development trends at the intersection of development and operations.
  • You understand that the hardest parts about DevOps practices revolve around the people, not the process or technology.
  • You manage multiple deadlines and projects independently.
  • You have publicly available writing samples and/or recordings of presentations.

Bonus points if you also:

  • Experience in the DevOps space
  • Interest in producing short videos for a technically-savvy audience.

Specific Requirement:

  • 25% travel required (when travel is a thing again)
  • Must be authorized to work in the country you reside in (no visa sponsorship / relocation)
  • Must have published writing samples
  • Minimum 5 years work experience

About LaunchDarkly:

LaunchDarkly is a Feature Management Platform that serves hundreds of billions of feature flags daily to help software teams build better software, faster. Feature flagging is an industry standard methodology of wrapping a new or risky section of code or infrastructure change with a flag. Each flag can easily be turned off independent of code deployment (aka "dark launching"). LaunchDarkly has SDKs for all major web and mobile platforms. We are building a diverse team so that we can offer robust products and services. Our team culture is dynamic, friendly, and supportive. Our headquarters are in Oakland.

At LaunchDarkly, we believe in the power of teams. We're building a team that is humble, open, collaborative, respectful and kind. We are an equal opportunity employer and value diversity at our company. We do not discriminate on the basis of race, religion, color, national origin, gender, gender identity, sexual orientation, age, marital status, veteran status, or disability status.

Don't let the confidence gap get in the way of applying! We'd love to hear from you.

We've partnered with KeyValues to help demonstrate the amazing culture we've built here at LaunchDarkly, find more info at https://www.keyvalues.com/launchdarkly.

Verified by
Special Circumstances
Head of Ecosystem Partnerships
Demand Program Manager
Engineering Lead
Director Marketing
VP of Product and Engineering
You may also like