Nine Experimentation Best Practices

1,204
LaunchDarkly
Serving over 20 trillion feature flags daily to help software teams build better software, faster. LaunchDarkly helps eliminate risk for developers and operations teams from the software development cycle.

This post is by Dawn Parzych of LaunchDarkly

Running experiments helps you bridge gaps between technical and business teams as you learn about user engagement patterns and application behavior. Experiments can validate that your development teams’ efforts align with your business objectives. You can configure an experiment from any feature flag (front-end, back-end, or operational), but what makes a good experiment?

Experiments use metrics to validate or disprove gut feelings about whether a new feature is meeting customers’ expectations. With an experiment, you can create and test a hypothesis regarding any aspect of your feature development process.

  • Will adding white space to the page result in people spending more time on the site?
  • Do alternate images lead to increased sales?
  • Will adding sorting to the page significantly increase load times?

Experiments provide concrete reporting and measurements to ensure that you are launching the best version of a feature that positively impacts company metrics.

General experimentation best practices

Let’s first talk about best practices around creating an experiment. Even with a solid foundation of feature flagging in your organization, missteps in these areas can yield flawed results. Consider these best practices for experimentation.

1. Create a culture of experimentation

Experiments can help you prove or disprove a hypothesis, but only if you are willing to trust the outcome and not try to game the experiment. Creating a culture of experimentation means:

  • People feel safe asking questions and questioning the answer. Sometimes the results of an experiment may not be what you were expecting. It’s ok to question the results and explore anomalies.
  • Ideas are solicited from all team members—business stakeholders, data analysts, product managers, and developers.
  • A data-driven approach is used to put metrics first, not as something that is only useful in dashboards after a feature ships.

Part of creating a culture of experimentation is providing the tools and training to allow teams to test and validate their features.

Tools needed for experimentation:

  • Tools to help you collect relevant metrics can include monitoring, observability, and business analytics tools.
  • Tools to help you segment users.
  • Tools to clean and analyze the results.

2. Define what success looks like as a team

Experiments help you determine when a feature is good enough to release. How do you define what “good enough” is, and who is involved in creating the definition? Experiments can involve a cross-functional team of people or only a handful, depending on the focus of the experiment. If an experiment is on whether to add a “free trial” button to the home page, you may need to involve people from demand generation, design, and business development.

A good experiment requires well-defined and agreed-upon goals and metrics by all stakeholders. Ask yourself, “What does success look like?” Success means improving a specific metric by a specific amount. “Waiting to see what happens” is not a goal befitting a true experiment. Goals need to be concrete and avoid ambiguous words. Tie goals and metrics to business objectives like increasing revenue, time spent on a page, or growing the subscription base.

Examples of poor goals include vague and ambiguous statements:

  • Users will be happier with the home page.
  • The response time of search results will improve.

Examples of better goals include concrete statistics:

  • Paid conversions from a trial will improve by 7%.
  • The response time of search results will decrease by 450ms.

Looking at a single metric is good; looking at related metrics is even better. Identify complementary metrics and examine how they behave. If you get more new subscribers—that’s great, but you also want to look at the total number of subscribers. Say you get additional subscribers, but it inadvertently causes existing subscribers to cancel. Your total number of subscribers may be down as a result, does that make the experiment a success? I would say no.

3. Statistical significance

The number of samples will depend on your weekly or monthly active users and your application. No matter the size of the samples, it is important to maintain a relatively equal sample size. Significantly more samples in one cohort can skew the results.

Think about how and when users access your application when starting an experiment. If users primarily use your application on the weekends, experiments should include those days. If users visit a site multiple times over the course of a couple of weeks before converting and before the experiment starts, early results may be skewed, showing a positive effect when it was neutral or negative.

4. Proper segmentation

There are two aspects to segmentation. First, how will you segment your users, and second, how will you segment the data? A successful experiment needs two or more cohorts. How you segment your users will vary based on your business and users. Some ways to segment users:

  • Logged in vs. anonymous
  • By company
  • By geography
  • Randomly

However you segment users, make sure the sample sizes are balanced. Having skewed cohorts will result in skewed results.

When analyzing the data after an experiment, you may want to do additional segmentation to see if the results vary based on other parameters.

In AirBnB’s experimentation blog, they described an experiment where they received neutral results when they were expecting to see an uplift. Analyzing the results per browser, they uncovered a bug in IE that skewed the results. When the bug was fixed, they saw the improvement they expected. If there was not a culture of experimentation where it was ok to question the result and look for answers, this feature might not have rolled out.

Caution: don’t go digging to find data to support your hypothesis. The segmentation should be done to understand anomalies better, not make a case to call the experiment a success.

5. Recognize your biases

And this brings us to the subject of biases. Everybody is biased. This isn’t a bad thing; it is a matter of how our brains work. Understanding your biases and the biases of others around you will help when running experiments. Our biases influence how we process information, make decisions, and form judgments.

Some biases that may creep in:

  • Anchoring bias – our tendency to rely on the first piece of information we receive.
  • Confirmation bias – searching for information that confirms our beliefs, preconceptions, or hypotheses.
  • Default effect – the tendency to favor the default option when presented with choices.
  • And my personal favorite, the bias blind spot – our belief that we are less biased than others.

It is common in an experiment to scrub the data to remove outliers. This is acceptable, but make sure you aren’t eliminating outliers due to bias. Don’t invalidate the results of your experiment by removing data points that don’t fit the story.

6. Conduct a retro

Experiments are about learning. So after every experiment, hold a retrospective. Ask:

  • Was the experiment a success? Did we get the results we expected? Why or why not?
  • What questions were raised by the experiment? What questions were answered?
  • What did we learn?

And most importantly, should we launch the feature? You can still decide to launch a feature if the results of an experiment were neutral.

Experimentation feature flags best practices

Now that we’ve covered the basics of experiment best practices, let’s dive into best practices around using feature flags in your experiment. Well done feature flagging is a foundation for a well-run experiment.

If you’ve embraced feature management as part of your teams’ DNA, any group that wants to run an experiment can quickly do so without involving engineering. Follow these best practices to get the most from your existing feature flags.

7. Consider experiments during the planning phase

The decision of whether to wrap a feature in a flag begins during the planning phase. This is also the right time to think about experiments. When creating a feature, evaluate the metrics that will indicate whether the feature is a success—clicks, page load time, registrations, sales, etc.

Talk with the various stakeholders to determine whether an experiment may be necessary and provide them with the essential information on the flag, such as the name of the flag, to configure an experiment.

8. Empower others

When giving others the ability to run experiments, make sure you have the proper controls in place to avoid flags accidentally being changed or deleted. Empower other teams to create targeting rules, define segments, and configure roll-out percentages while preventing them from toggling or deleting flags.

9. Avoid technical debt

Experimentation flags should be short-lived. This means removing a flag after an experiment completes and is either rolled out to 100% of users or not released. Put processes and reminders in place to remove the flag. We highlighted some creative ways to do this in our short-term and permanent best practices blog.

To get started with your own experiments, check out the LaunchDarkly documentation.

Are you looking for other best practices? Check out the other blog posts in the series.

Long-term and short-term flags best practices

Operational flags best practices

Release management flags best practices

LaunchDarkly
Serving over 20 trillion feature flags daily to help software teams build better software, faster. LaunchDarkly helps eliminate risk for developers and operations teams from the software development cycle.
Tools mentioned in article
Open jobs at LaunchDarkly
Platform Engineer
- US
<p data-renderer-start-pos="2137"><strong data-renderer-mark="true">About the Job:&nbsp;</strong></p> <p data-renderer-start-pos="2154">As a Platform Engineer, you will help us extend and scale LaunchDarkly's engineering infrastructure. You will build an internal development platform that brings joy to your end users - LaunchDarkly developers - and you are passionate about multiplying our engineering force through automation, orchestration, documentation, and productivity tools.</p> <p data-renderer-start-pos="2503">LaunchDarkly is a rapidly growing software company with a strong mission and vision carried out by a talented and diverse team of employees. Our goal is to help teams build better software, faster. You'll join a small team from companies like Atlassian, Intercom, and Twitter, and you'll have an immediate impact with our product and customers.</p> <h4>Responsibilities:</h4> <ul> <li> <p data-renderer-start-pos="2874">Collaborate with engineering teams to identify bottlenecks and areas for improvement in their development lifecycle, including Infrastructure as Code (IaC) templates for consistent provisioning.</p> </li> <li> <p data-renderer-start-pos="3072">Implement tools and processes to increase developer productivity and reduce time-to-market.</p> </li> <li> <p data-renderer-start-pos="3167">Automate infrastructure provisioning, scaling, and management to ensure efficiency and reliability.</p> </li> <li> <p data-renderer-start-pos="3270">Lead the design and implementation of our internal developer platform, empowering development teams to build and operate their services faster and more effectively.</p> </li> <li> <p data-renderer-start-pos="3438">Ensure the platform is scalable, secure, and aligned with best practices.</p> </li> <li> <p data-renderer-start-pos="3515">Troubleshoot and resolve delivery pipeline issues to ensure a reliable release process.</p> </li> <li> <p data-renderer-start-pos="3606">Keep up-to-date with platform engineering industry trends and emerging technologies.</p> </li> </ul> <h4>Qualifications:</h4> <ul> <li> <p data-renderer-start-pos="3713">You possess excellent communication and collaboration skills. You play well on a small, tight-knit team.</p> </li> <li> <p data-renderer-start-pos="3821">You are a self‐starter and problem solver, willing to solve hard problems and work independently when necessary. You identify potential problems and nip them in the bud before they surface.</p> </li> <li> <p data-renderer-start-pos="4014">You have a passion for reducing toil for yourself and your colleagues, and love solving problems with scripts and automation (e.g. with Python and Bash.)</p> </li> <li> <p data-renderer-start-pos="4171">You have successfully run large-scale production systems on public cloud infrastructure.</p> </li> <li> <p data-renderer-start-pos="4263">You have strong proficiency in building and maintaining CI/CD pipelines with tools like CircleCI and Spinnaker.</p> </li> <li> <p data-renderer-start-pos="4378">You are familiar with running systems with a service-oriented architecture.</p> </li> <li> <p data-renderer-start-pos="4457">You have written software in languages like Go or Python.</p> </li> <li> <p data-renderer-start-pos="4518">You have extensive experience codifying infrastructure with tools like Terraform.</p> </li> </ul> <div> <p><strong>Pay:</strong></p> <p>Target pay ranges based on Geographic Zones* for Level P3:</p> <ul> <li>Zone 1: San Francisco/Bay Area or New York City Metropolitan Area: $150,000-176,000**</li> <li>Zone 2: Boston, DC, Irvine, LA, Monterey, Santa Barbara, Santa Rosa, Seattle: $135,000-158,000**</li> <li>Zone 3: All other US locations: $127,000-150,000**</li> </ul> <p>Target pay ranges based on Geographic Zones* for Level P5:</p> <ul> <li>Zone 1: San Francisco/Bay Area or New York City Metropolitan Area: $200,000-235,000**</li> <li>Zone 2: Boston, DC, Irvine, LA, Monterey, Santa Barbara, Santa Rosa, Seattle: $180,000-212,000**</li> <li>Zone 3: All other US locations: $170,000 - $200,000**</li> </ul> <p data-renderer-start-pos="6572"><em data-renderer-mark="true">*Within the United States, our geographic pay zones are defined by counties surrounding major metropolitan areas.<br>**Restricted Stock Units (RSUs), health, vision, and dental insurance, and mental health benefits in addition to salary.</em></p> <p data-renderer-start-pos="6572"><em data-renderer-mark="true">LaunchDarkly operates from a place of high trust and transparency; we are happy to state the pay range for our open roles to best align with your needs. Exact compensation may vary based on skills, experience, degree level, and location.</em></p> </div> <h4><strong>About LaunchDarkly:</strong></h4> <p>Modern software delivery was supposed to be the foundation for a thriving digital business but reality has proven otherwise. Slow, inefficient development cycles, costly outages, and fragmented customer experiences are preventing developers from building their best software. The LaunchDarkly platform helps developers innovate on new features faster while protecting them with a safety valve to instantly rewind when things go wrong. Developers can target product experiences to any customer segment and maximize the business impact of every feature. And by gradually rolling out new application components, they escape nightmare "big-bang" technology migrations.&nbsp;</p> <p>The LaunchDarkly platform was built to guide engineers to the next frontier of DevOps by:</p> <ul> <li>Improving the velocity and stability of software releases, without the fear of end customer outages</li> <li>Delivering targeted experiences by easily personalizing features to customer cohorts</li> <li>Maximizing the business impact of every feature through the ability to experiment and optimize</li> <li>Coordinating the release and optimization of software to provide consistent experiences across mobile platforms and device types</li> <li>Improving the effectiveness and productivity of engineering teams, by providing insights into engineering cadence and stability</li> </ul> <p>At LaunchDarkly, we believe in the power of teams. We're building a team that is humble, open, collaborative, respectful and kind. We are an equal opportunity employer and value diversity at our company. We do not discriminate on the basis of race, religion, color, national origin, gender, gender identity, sexual orientation, age, marital status, veteran status, or disability status.<br><br>One of our company values is 'Widen the Circle'. Which means we seek out diversity of perspectives to get better results. We understand everyone has their own unique talents and experiences. We encourage you to apply to this role even if you don’t think you meet 100% of the qualifications outlined above. We can find out together if it's the right match for your skillset.</p> <p class="p1">Learn more about the interview process &amp; what it’s like to work with <a class="css-tgpl01" href="https://www.gem.com/lp/landing-pages/launchdarkly-product-delivery" data-testid="link-with-safety" data-renderer-mark="true">LaunchDarkly's Product Delivery team</a>!</p> <p class="p1"><span class="s1"><strong>Do you need a disability accommodation?</strong></span></p> <p class="p1">Fill out this&nbsp;<a href="https://docs.google.com/forms/d/e/1FAIpQLSdYb_7upYMtdRVXzvXGHGfQw0pU2FNma-6Rwp-I6NjKm7SYNw/viewform"><span class="s2">accommodations request form</span></a>&nbsp;and someone from our People Operations team will contact you for assistance.&nbsp;</p>
Enterprise Solutions Engineer (Pre-Sa...
London, England, United Kingdom
<h4>About the Job:</h4> <p>As a Solutions Engineer, you will educate and guide prospects on the proper implementation of LaunchDarkly's SaaS product and Private Instances. You are passionate about trends and technologies involved in modern application development.&nbsp;You will be the technical voice during our sale and ensure our customers are comfortable with the way our systems work. You are passionate about the developer tools space and helping development teams eliminate risk and deliver value.</p> <p data-renderer-start-pos="1601">LaunchDarkly is a rapidly growing software company with a strong mission and vision carried out by a talented and diverse team of employees. Our goal is to help teams build better software, faster. Software powers the world and LaunchDarkly empowers all teams to deliver and control their software.</p> <p data-renderer-start-pos="1601">Please note this position is a Pre-Sales Engineer role, working alongside our existing SE team and providing key support and technical insight throughout the sales cycle.&nbsp;</p> <h4>Responsibilities:</h4> <ul> <li data-renderer-start-pos="1924">Evangelize and advise customers on the importance and different uses of feature flags and how to administer them</li> <li data-renderer-start-pos="2040">Presenting solutions to customer's challenges implementing feature flags across large monolith and microservice applications, large organizations, and different technology stacks</li> <li data-renderer-start-pos="2218">Become a domain expert on LaunchDarkly architecture</li> <li data-renderer-start-pos="2273">Demo LaunchDarkly product to technical and business audiences</li> <li data-renderer-start-pos="2338">Become a subject matter expert on LaunchDarkly and communicate our value and features to potential customers</li> <li data-renderer-start-pos="2450">Be the voice of the customer by translating, aggregating, and representing customer feedback to the Product and Engineering teams</li> </ul> <h4>Qualifications:</h4> <ul> <li data-renderer-start-pos="2608">4+ years of experience consulting with enterprise customers and large development teams</li> <li data-renderer-start-pos="2700">You led successful technical proof of concepts&nbsp;</li> <li data-renderer-start-pos="2751">Proven success in building strong customer relationships</li> <li data-renderer-start-pos="2811">Ability to learn and synthesize large amounts of information with little context</li> <li data-renderer-start-pos="2895">Effective communicator with the ability to simplify complex technical concepts</li> <li data-renderer-start-pos="2977">A self‐starter and problem solver, willing to take on hard problems and work independently when necessary.</li> <li data-renderer-start-pos="3116">Experience working with teams that underwent development process transformation</li> <li data-renderer-start-pos="3199">Familiarity with at least one of our supported languages: Java, .NET, GO, JS, Python, PHP, Node, Ruby, Rails, iOS, or Android</li> <li data-renderer-start-pos="3328">Experience with data persistence technologies like Varnish or Redis</li> </ul> <p><strong>Specific Requirements:</strong></p> <ul> <li>Must be able and willing to travel as needed</li> </ul> <h4><strong>About LaunchDarkly:</strong></h4> <p>Modern software delivery was supposed to be the foundation for a thriving digital business but reality has proven otherwise. Slow, inefficient development cycles, costly outages, and fragmented customer experiences are preventing developers from building their best software. The LaunchDarkly platform helps developers innovate on new features faster while protecting them with a safety valve to instantly rewind when things go wrong. Developers can target product experiences to any customer segment and maximize the business impact of every feature. And by gradually rolling out new application components, they escape nightmare "big-bang" technology migrations.&nbsp;</p> <p>The LaunchDarkly platform was built to guide engineers to the next frontier of DevOps by:</p> <ul> <li>Improving the velocity and stability of software releases, without the fear of end customer outages</li> <li>Delivering targeted experiences by easily personalizing features to customer cohorts</li> <li>Maximizing the business impact of every feature through the ability to experiment and optimize</li> <li>Coordinating the release and optimization of software to provide consistent experiences across mobile platforms and device types</li> <li>Improving the effectiveness and productivity of engineering teams, by providing insights into engineering cadence and stability</li> </ul> <p>At LaunchDarkly, we believe in the power of teams. We're building a team that is humble, open, collaborative, respectful and kind. We are an equal opportunity employer and value diversity at our company. We do not discriminate on the basis of race, religion, color, national origin, gender, gender identity, sexual orientation, age, marital status, veteran status, or disability status.<br><br>One of our company values is 'Widen the Circle'. Which means we seek out diversity of perspectives to get best results. We understand everyone has their own unique talents and experiences. We encourage you to apply to this role even if you don’t think you meet 100% of the qualifications outlined above. We can find out together if it's the right match for your skillset.</p> <p class="p1"><span class="s1"><strong>Do you need a disability accommodation?</strong></span></p> <p class="p1">Fill out this&nbsp;<a href="https://docs.google.com/forms/d/e/1FAIpQLSdYb_7upYMtdRVXzvXGHGfQw0pU2FNma-6Rwp-I6NjKm7SYNw/viewform"><span class="s2">accommodations request form</span></a>&nbsp;and someone from our People Operations team will contact you for assistance.&nbsp;</p> <p>&nbsp;</p>
Senior Analytics Engineer
Oakland, California, United States
<h4><strong>About the Job:<br></strong></h4> <p>LaunchDarkly is seeking an experienced Analytics Engineer to join our data team. You will work with other Data Engineers, Analytics Engineers, and stakeholders across the company to deliver data insights that create material impacts to our business.</p> <h4>Responsibilities:</h4> <ul class="ak-ul" data-indent-level="1"> <li> <p data-renderer-start-pos="2997">Develop data-driven solutions to critical business problems by leveraging our data warehouse, analytics tools, and scripting languages.</p> </li> <li> <p data-renderer-start-pos="3136">Partner with stakeholders throughout the company to drive improved business results in their areas by providing creative technical solutions to their most pressing business challenges.</p> </li> <li> <p data-renderer-start-pos="3325">Translate quantitative analyses and findings into clear visuals, written documentation, and other data assets for non-technical end users.</p> </li> <li> <p data-renderer-start-pos="3468">Provide suggestions for improving the reliability, ease of use, and performance of the Data Team’s technical infrastructure.</p> </li> <li> <p data-renderer-start-pos="3596">Form strong relationships with your team and refine our best practices e.g. naming convention, data modeling, and data quality testing.</p> </li> <li> <p data-renderer-start-pos="3735">Enable users across the organization to self-serve requests using our explorable data sets</p> </li> </ul> <h4>Qualifications:</h4> <ul> <li> <p data-renderer-start-pos="3848">5+ years of professional experience as an Analytics Engineer, Data Scientist, or similar.</p> </li> <li> <p data-renderer-start-pos="3941">A Bachelor’s degree in a technical field (Computer Science, Physics, Math, Economics, etc)</p> </li> <li> <p data-renderer-start-pos="4036">An ability to balance humility with rigorous standards for work quality</p> </li> <li> <p data-renderer-start-pos="4111">A curious and growth-oriented mindset towards learning and teaching.</p> </li> <li> <p data-renderer-start-pos="4183">Ability to understand, tackle, and communicate problems from both technical and business perspectives</p> </li> <li> <p data-renderer-start-pos="4288">Experience with data modeling and warehousing concepts</p> </li> <li> <p data-renderer-start-pos="4346">Proficient in Snowflake (SQL), Python, and visualization tools</p> </li> <li> <p data-renderer-start-pos="4412">Proficient with advanced statistical techniques</p> </li> <li> <p data-renderer-start-pos="4463">Comfortable working in DBT and AWS Data Tools.</p> </li> </ul> <p><strong>Pay:</strong></p> <p data-renderer-start-pos="6395"><em data-renderer-mark="true">Target pay range for a Level P4 in San Francisco/Bay Area: </em><em>$173,000 - $204,000*</em></p> <p data-renderer-start-pos="6451"><em data-renderer-mark="true">*Restricted Stock Units (RSUs), health, vision, and dental insurance, and mental health benefits in addition to salary.</em></p> <p data-renderer-start-pos="6572"><em data-renderer-mark="true">LaunchDarkly operates from a place of high trust and transparency; we are happy to state the pay range for our open roles to best align with your needs. Exact compensation may vary based on skills, experience, degree level, and location.</em></p> <h4><strong>About LaunchDarkly:</strong></h4> <p>Modern software delivery was supposed to be the foundation for a thriving digital business but reality has proven otherwise. Slow, inefficient development cycles, costly outages, and fragmented customer experiences are preventing developers from building their best software. The LaunchDarkly platform helps developers innovate on new features faster while protecting them with a safety valve to instantly rewind when things go wrong. Developers can target product experiences to any customer segment and maximize the business impact of every feature. And by gradually rolling out new application components, they escape nightmare "big-bang" technology migrations.&nbsp;</p> <p>The LaunchDarkly platform was built to guide engineers to the next frontier of DevOps by:</p> <ul> <li>Improving the velocity and stability of software releases, without the fear of end customer outages</li> <li>Delivering targeted experiences by easily personalizing features to customer cohorts</li> <li>Maximizing the business impact of every feature through the ability to experiment and optimize</li> <li>Coordinating the release and optimization of software to provide consistent experiences across mobile platforms and device types</li> <li>Improving the effectiveness and productivity of engineering teams, by providing insights into engineering cadence and stability</li> </ul> <p>At LaunchDarkly, we believe in the power of teams. We're building a team that is humble, open, collaborative, respectful and kind. We are an equal opportunity employer and value diversity at our company. We do not discriminate on the basis of race, religion, color, national origin, gender, gender identity, sexual orientation, age, marital status, veteran status, or disability status.<br><br>One of our company values is 'Widen the Circle'. Which means we seek out diversity of perspectives to get better results. We understand everyone has their own unique talents and experiences. We encourage you to apply to this role even if you don’t think you meet 100% of the qualifications outlined above. We can find out together if it's the right match for your skillset.</p> <p class="p1"><span class="s1"><strong>Do you need a disability accommodation?</strong></span></p> <p class="p1">Fill out this&nbsp;<a href="https://docs.google.com/forms/d/e/1FAIpQLSdYb_7upYMtdRVXzvXGHGfQw0pU2FNma-6Rwp-I6NjKm7SYNw/viewform"><span class="s2">accommodations request form</span></a>&nbsp;and someone from our People Operations team will contact you for assistance.&nbsp;</p>
Backend Engineer, Data Pipeline
Oakland, California, United States
<p data-renderer-start-pos="1895"><strong data-renderer-mark="true">About the Job:</strong></p> <p data-renderer-start-pos="1911">We are currently looking for a talented Software Engineer to join our Experimentation Product team. You will help us architect and write fast, reliable, and scalable data processing tools to process data from our thousands of customers and their hundreds of millions of users around the world. We're looking for someone who knows what it takes to deliver value to customers and takes pride in the quality of their work.</p> <p data-renderer-start-pos="2332">Our data platform processes events from over twenty billion feature flag evaluations, and billions of experimentation metrics events daily. The core technologies we use daily include Golang, Scala, Kinesis, and Flink. As part of our you-build-it-you-run-it culture, all developers may be responsible for supporting applications in production, including on-call.</p> <p data-renderer-start-pos="2695">Your work will directly reflect on the customer experience to allow them to run the most effective experiments possible.</p> <h4 id="Responsibilities:" data-renderer-start-pos="2821"><strong data-renderer-mark="true">Responsibilities:</strong></h4> <ul class="ak-ul" data-indent-level="1"> <li> <p data-renderer-start-pos="2842">Working with a modern data stack and open source frameworks</p> </li> <li> <p data-renderer-start-pos="2905">Build and expand our data platform, pipeline, and services</p> </li> <li> <p data-renderer-start-pos="2967">Design data pipelines for new products and evolving customer needs</p> </li> <li> <p data-renderer-start-pos="3037">Collaborate with product team to influence and deliver user-facing features</p> </li> <li> <p data-renderer-start-pos="3116">Monitor and improve data pipeline performance for continuous improvement</p> </li> <li> <p data-renderer-start-pos="3192">Champion engineering standards, tooling, and processes</p> </li> </ul> <p data-renderer-start-pos="3250"><strong data-renderer-mark="true">Qualifications:</strong></p> <ul> <li data-renderer-start-pos="3269">Proven experience and fluency in a JVM language or Golang</li> <li> <p data-renderer-start-pos="3330">Experience building data platforms (e.g. using Flink, Kafka, DataFlow, Hadoop, Spark)</p> </li> <li> <p data-renderer-start-pos="3419">Experience with data pipelines, distributed systems, and large-scale data processing</p> </li> <li> <p data-renderer-start-pos="3507">Strong communication skills, a positive attitude, and empathy</p> </li> <li> <p data-renderer-start-pos="3572">You hold yourself and others to a high bar when working with data systems</p> </li> <li> <p data-renderer-start-pos="3649">You value high code quality, automated testing, and other engineering best practices</p> </li> <li> <p data-renderer-start-pos="3737">Experience with AWS</p> </li> <li> <p data-renderer-start-pos="3760">Experience with experimentation systems / products</p> </li> <li> <p data-renderer-start-pos="3814">Working knowledge of ETL and a query language</p> </li> </ul> <div> <p><strong>Pay:</strong></p> <p data-renderer-start-pos="3872"><em data-renderer-mark="true">Target pay range for a Level P5 in San Francisco/Bay Area: $200,000-$235,000*</em></p> <p data-renderer-start-pos="6451"><em data-renderer-mark="true">*Restricted Stock Units (RSUs), health, vision, and dental insurance, and mental health benefits in addition to salary.</em></p> <p data-renderer-start-pos="6572"><em data-renderer-mark="true">LaunchDarkly operates from a place of high trust and transparency; we are happy to state the pay range for our open roles to best align with your needs. Exact compensation may vary based on skills, experience, degree level, and location.</em></p> </div> <h4><strong>About LaunchDarkly:</strong></h4> <p>Modern software delivery was supposed to be the foundation for a thriving digital business but reality has proven otherwise. Slow, inefficient development cycles, costly outages, and fragmented customer experiences are preventing developers from building their best software. The LaunchDarkly platform helps developers innovate on new features faster while protecting them with a safety valve to instantly rewind when things go wrong. Developers can target product experiences to any customer segment and maximize the business impact of every feature. And by gradually rolling out new application components, they escape nightmare "big-bang" technology migrations.&nbsp;</p> <p>The LaunchDarkly platform was built to guide engineers to the next frontier of DevOps by:</p> <ul> <li>Improving the velocity and stability of software releases, without the fear of end customer outages</li> <li>Delivering targeted experiences by easily personalizing features to customer cohorts</li> <li>Maximizing the business impact of every feature through the ability to experiment and optimize</li> <li>Coordinating the release and optimization of software to provide consistent experiences across mobile platforms and device types</li> <li>Improving the effectiveness and productivity of engineering teams, by providing insights into engineering cadence and stability</li> </ul> <p>At LaunchDarkly, we believe in the power of teams. We're building a team that is humble, open, collaborative, respectful and kind. We are an equal opportunity employer and value diversity at our company. We do not discriminate on the basis of race, religion, color, national origin, gender, gender identity, sexual orientation, age, marital status, veteran status, or disability status.<br><br>One of our company values is 'Widen the Circle'. Which means we seek out diversity of perspectives to get better results. We understand everyone has their own unique talents and experiences. We encourage you to apply to this role even if you don’t think you meet 100% of the qualifications outlined above. We can find out together if it's the right match for your skillset.</p> <p class="p1">Learn more about the interview process &amp; what it’s like to work with <a class="css-tgpl01" href="https://www.gem.com/lp/landing-pages/launchdarkly-product-delivery" data-testid="link-with-safety" data-renderer-mark="true">LaunchDarkly's Product Delivery team</a>!</p> <p class="p1"><span class="s1"><strong>Do you need a disability accommodation?</strong></span></p> <p class="p1">Fill out this&nbsp;<a href="https://docs.google.com/forms/d/e/1FAIpQLSdYb_7upYMtdRVXzvXGHGfQw0pU2FNma-6Rwp-I6NjKm7SYNw/viewform"><span class="s2">accommodations request form</span></a>&nbsp;and someone from our People Operations team will contact you for assistance.&nbsp;</p>
Verified by
Software and Product Engineer
Software Engineer
Computer Science
Physics
Director Marketing
Software Engineer
Engineering Manager
Software Engineer
Software Engineeer
VP of Product and Engineering
Engineering Lead
Software Engineer
Special Circumstances
Demand Program Manager
You may also like