How to Disable Code: The Developer’s Production Kill Switch

727
CloudBees
CloudBees, the enterprise software delivery company, provides the industry’s leading DevOps technology platform. CloudBees enables developers to focus on what they do best: Build stuff that matters, while providing peace of mind to management with powerful risk mitigation, compliance and governance tools.

The following is a guest post written by Carlos Schults.

Being able to disable code in production is a power that many developers aren’t aware of. And that’s a shame. The ability to switch off some portions—or even complete features—of the codebase can dramatically improve the software development process by allowing best practices that can shorten feedback cycles and increase the overall quality.

So, that’s what this post will cover: the mechanisms you can use to perform this switching off, why they’re useful and how to get started. Let’s dig in.

Why Would You Want to Disable Code?

Before we take a deep dive into feature flags, explaining what they are and how they’re implemented, you might be asking: Why would people want to switch off some parts of their codebase? What’s the benefit of doing that?

To answer these questions, we need to go back in time to take a look at how software was developed a couple of decades ago. Time for a history lesson!

The Dark Ages: Integration Hell

Historically, integration has been one of the toughest challenges for teams trying to develop software together.

Picture several teams inside an organization, working separately for several months, each one developing its own feature. While the teams were working in complete isolation, their versions of the application were evolving in different directions. Now they need to converge again into a single, non conflicting version. This is a Herculean task.

That’s what “integration hell” means: the struggle to merge versions of the same application that have been allowed to diverge for too long.

Enter the Solution: Continuous Integration

“If it hurts, do it more often.” What this saying means is that there are problems we postpone solving because doing so is hard. What you often find with these kinds of problems is that solving them more frequently, before they accumulate, is way less painful—or even trivial.

So, how can you make integrations less painful? Integrate more often.

That’s continuous integration (CI) in a nutshell: Have your developers integrate their work with a public shared repository, at the very least once a day. Have a server trigger a build and run the automated test suite every time someone integrates their work. That way, if there are problems, they’re exposed sooner rather than later.

How to Handle Partially Completed Features

One challenge that many teams struggle with in CI is how to deal with features that aren’t complete. If developers are merging their code to the mainline, that means that any developments that take more than one day to complete will have to be split into several parts.

How can you avoid the customer accessing unfinished functionality? There are some trivial scenarios with similarly trivial solutions, but harder scenarios call for a different approach: the ability to switch off a part of the code completely.

Feature Flags to the Rescue

Defining Feature Flags

There are many names for the mechanisms that allow developers to switch a portion of their code off and on. Some call them “feature toggles” or “kill switches.” But “feature flags” is the most popular name, so that’s what we’ll use for the remainder of this post. So, what are feature flags?

Put simply, feature flags are techniques that allow teams to change the behavior of an application without modifying the code. In general, flags are used to prevent users from accessing and using the changes introduced by some piece of code, because they’re not adequate for production yet for a number of reasons.

Disable Code: What Are the Use Cases?

We’ll now cover some of the most common use cases for disabling code in production.

Switching Off Unfinished Features

As you’ve seen, one of the main use cases for feature flags is preventing users from accessing features that aren’t ready for use yet.

That way, programmers developing features that are more complex and take a longer time to complete aren’t prevented from integrating their work often and benefiting from it.

Enabling A/B Testing

The adoption of feature flags enables the use of several valuable practices in the software development process, one of which is A/B testing.

A/B testing is a user experience research technique that consists of comparing two versions of a website or application to decide which one to keep. It entails randomly splitting users into two groups, A and B, and then delivering a different version of the application to each group. One group might receive the current production version, which we call the “control,” whereas the second group would receive the candidate for the new version, called the “treatment.”

The testers then monitor the behavior of both groups and determine which of the versions achieved better results.

Feature flags are a practical way to enable A/B testing because they allow you to quickly and conveniently change between the control and treatment versions of your application.

Enabling Canary Releases

If you deliver the new version of your app to your entire userbase at once, 100 percent of your users will be impacted if the release is bad in some way. What if you could gradually roll out the new version instead? You’d first deploy to a small subset of users, monitoring that group to detect issues. If something went wrong, you could roll it back. If everything looked fine, you could then gradually release the version for larger groups. That’s a canary release in a nutshell. It’s another powerful technique that feature flags might help with.

Customizing Features According to Users’ Preferences

It’s not uncommon to have to customize your application according to the needs of specific users, and there are several ways in which software teams can accomplish that—some more efficient, and others less so (companies that create separate branches or entire repositories for each client come to mind).

This is another area where feature flags could help, allowing teams to dynamically switch between different versions of the same functionality.

Disable Code in Production 101

How do you go about disabling code? That’s what we’re going to see now, in three increasingly sophisticated phases.

First Stage: The Most Basic Approach

We start with an approach that’s so primitive, it maybe shouldn’t be considered a feature flag at all. Consider the pseudocode below:

calculateAdditionalWorkHours(Employee employee, Date start, Date end) {     
    // return calculateAdditionalWorkHoursSameOldWay(employee, start, end);
    return calculateAdditionalWorkHoursImproved(employee, start, end); 
    }

In the code above, we're just commenting out the old version of some method and replacing it with a new version. When we want the older version to be used, we just do the opposite. (Well, I said it was primitive.) This approach lacks one of the most fundamental properties of a feature flag—the ability to change how the application behaves without changing its code.

However, it plants the seed for more sophisticated approaches.

Second Stage: Taking the Decision Out of the Code

The previous approach didn't allow developers to select the desired version of the feature without changing the code. Fortunately, that's not so hard to do. First, we introduce a logical variable to determine which version we're going to use:

calculateAdditionalWorkHours(Employee employee, Date start, Date end) {

    var result = useNewCalculation
        ? calculateAdditionalWorkHoursImproved(employee, start, end)
        : calculateAdditionalWorkHoursSameOldWay(employee, start, end);

    return result;
}

Then, we use some mechanism to be able to assign the value to the variable from an external source. We could use a configuration file:

var useNewCalculation = config[newCalculation];

Passing arguments to the application might be another option. What matters is that we now have the ability to modify how the app behaves from the outside, which is a great step toward "true" feature flagging.

Keep in mind that the code examples you see are all pseudocode. Using your favorite programming language, there's nothing stopping you from starting with this approach and taking it up a notch. You could, for instance, use classes to represent the features and design patterns (e.g., factories) to avoid if statements.

Stage 3: Full-Fledged Feature Flag Management

The previous approach might be enough when your application has only a small number of flags. But as that number grows, things start to become messy.

First, you have the issue of technical debt. Manually implemented feature flags can create terribly confusing conditional flows in your codebase. That only grows worse with new flags being introduced each day. Additionally, they might make the code harder to understand and navigate, especially for more junior developers, which is an invitation for bugs.

Another problem is that as the number of flags grows, it becomes more and more common to forget to delete old, obsolete ones.

The main problem of a homegrown approach is that it doesn't give you an easy way to see and manage all of your flags at once. That's why our third and final stage is a single piece of advice: Instead of rolling out your own feature flags approach, adopt a third-party feature flag management system.

Feature Flags Are a CI/CD Enabler

We've covered the mechanisms developers can use to disable portions of their codebase in production without having to touch the code. This capability is powerful and enables techniques such as A/B testing and canary releases, which are all hallmarks of a modern, agile-based software development process.

The names for the techniques might vary—feature flags, feature toggles, feature flipper, and so on. The way in which the techniques are implemented can also vary—from a humble if statement to sophisticated cloud-based solutions.

But no matter what you call them, you can't overstate the benefit these mechanisms offer. They're an enabler of Continuous Integration, which is essential for any modern software organization that wants to stay afloat.

CloudBees
CloudBees, the enterprise software delivery company, provides the industry’s leading DevOps technology platform. CloudBees enables developers to focus on what they do best: Build stuff that matters, while providing peace of mind to management with powerful risk mitigation, compliance and governance tools.
Tools mentioned in article
Open jobs at CloudBees
Security Software Engineer in Jenkins...
US East or EMEA

About CloudBees

CloudBees provides the leading software delivery platform for enterprises, enabling them to continuously innovate, compete, and win in a world powered by the digital experience. Designed for the world's largest organizations with the most complex requirements, CloudBees enables software development organizations to deliver scalable, compliant, governed, and secure software from the code a developer writes to the people who use it. The platform connects with other best-of-breed tools, improves the developer experience, and enables organizations to bring digital innovation to life continuously, adapt quickly, and unlock business outcomes that create market leaders and disruptors.

CloudBees was founded in 2010 and is backed by Goldman Sachs, Morgan Stanley, Bridgepoint Credit, HSBC, Golub Capital, Delta-v Capital, Matrix Partners, and Lightspeed Venture Partners. Visit www.cloudbees.com and follow us on Twitter, LinkedIn, and Facebook.

Why this role

We are looking for a highly motivated (security) engineer to join a team focusing on improving the security of Jenkins and CloudBees CI, our Jenkins based product.

What You’ll Do

  • Learn the internals of Jenkins and its plugin system from the perspective of web application security
  • Participate in the Jenkins security team working on the full lifecycle of vulnerabilities
  • Different security aspects: architecture, design, review, implementation
  • Engineer and automate our security tooling/process
  • Security education / awareness
  • Our Engineering teams are distributed by design. You can work from the offices or remotely without feeling disconnected

What The Role Requires

  • Solid experience in Java web application development (JavaScript is a plus)
  • Interest in web application security
  • Experience in scripting is a plus (Groovy, Shell, Python)
  • Hacker mindset
  • Experience using CI/CD tools (Jenkins is a plus)

What You’ll Get

  • Highly competitive benefits and vacation package 
  • Ability to work for one of the fastest growing companies with some of the most talented people in the industry 
  • Team outings
  • Fun, Hardworking, and Casual Environment
  • Endless Growth Opportunities

At CloudBees, we truly believe that the more diverse we are, the better we serve our customers.  A global community like Jenkins demands a global focus from CloudBees. Organizations with greater diversity—gender, racial, ethnic, and global—are stronger partners to their customers.  Whether by creating more innovative products, or better understanding our worldwide customers, or establishing a stronger cross-section of cultural leadership skills, diversity strengthens all aspects of the CloudBees organization.

In the technology industry, diversity creates a competitive advantage.  CloudBees customers demand technologies from us that solve their software development, and therefore their business problems, so that they can better serve their own customers.  CloudBees attributes much of its success to its worldwide work force and commitment to global diversity, which opens our proprietary software to innovative ideas from anywhere. Along the way, we have witnessed firsthand how employees, partners, and customers with diverse perspectives and experiences contribute to creative problem solving and better solutions for our customers and their businesses.

Business Systems Architect

The Company

A remote first, fast paced, international growing software technology company looking for the best talent to join The Hive during a rapid growth 3 year plan. We are looking for curious, self-driven and talented new team members for an exciting opportunity. 

Over 3,500 of the world’s best-known brands and over 50% of the Fortune 500, invest in CloudBees because of its ability to work across any cloud, in any development environment and to balance corporate governance and control with developer flexibility and freedom.

CloudBees is home to the world’s leading DevOps experts helping thousands of companies harness the power of “continuous everything” and putting them on the fastest path from great idea, to great software, to great business value.

The role

The Manager, Business Systems Architecture is a leadership role. S/he will be responsible for establishing a strong Business Solution team within Cloudbees. This Business Solution team manages the ongoing use and  integration of various critical third party business systemsIncluding, but not limited to, end-user support, maintenance, development and implementation of reporting and dashboards, workflow modification, and on-going tighter integration among various business systems to increase efficiency and productivity of Cloudbees business operations. Although this position is in the Business Systems organization, the individual works closely with subject matter experts in Sales, Marketing, Product, and other business areas to identify, deploy, and maintain both existing and new features and functionality to support the entire organization.

 

Main responsibilities:

  • Provide strategy, integration, and operation expertise on business solution systems such as Salesforce, Gainsight, Atlassian, Zendesk, Oracle, Marketo, NetSuite, Celigo, Clari, Dun & Bradstreet etc.
  • Review business requirements from various departments, analyze requirements, design solution, and develop project plan 
  • Collaborate with various people in different departments and able to sell ideas, design proposal, and project plan effectively
  • Complete regular internal system audits and prepare for upgrades
  • Coordinate the evaluation, scope and completion of new integration or upgrade projects
  • Work independently with members of the user community to define and document business and functional requirements.
  • Execute new functionality in coordination with the Business Systems team and impacted teams in the user community
  • Partner with various business teams/departments with evaluation of appexchange connectors to support expanding organization requirements

 

Reporting structure:

  • This role is within Business Operations department
  • Business Operations department provides Cloudbees the ability to monitor and manage its business through real-time data and provide scalable and efficient business intelligence and IT Services
  • This role will report to Area Vice President of Business Operations

Location

 

  • USA, East Coast

 

Requirements

 

  • Bachelor’s degree in Business, Communications, Information Technology or equivalent.
  • 7-10 years of experience with tier 1 and 2 commercial business solutions such as Salesforce, Gainsight, Atlassian, Zendesk, Oracle, Marketo, NetSuite, Celigo, Clari, Dun & Bradstreet etc. 
  • Demonstrated leadership experience with security profiles and configuration, reports and dashboards, data integration tools and application integration
  • Proven ability to deliver implementation best practices and/or new processes and work with various groups to facilitate user adoption
  • Understanding of business platforms and business systems that made up business processes, with the ability to build and modify flows, custom views, and other content of intermediate complexity
  • Strong ability to critically evaluate information gathered from multiple sources, reconcile conflicts, decompose high-level information into details, abstract up from low-level information to a general understanding, and distinguish user requests from the underlying true needs
  • Strong ability to assess the impact of new requirements and connected applications, systems and processes
  • Ability to work independently as well as in a team environment.
  • Strong attention to detail and demonstrate exceptional ability to communicate verbally and in writing 
  • Strong ability to meet deadlines, handle and prioritize simultaneous requests, and manage laterally and upwards

 

Developer Relations Engineer
EMEA

OUR CUSTOMERS DEVELOP SOFTWARE AT THE SPEED OF IDEAS

CloudBees, the enterprise software delivery company, provides the industry’s leading DevOps technology platform. CloudBees enables developers to focus on what they do best: Build stuff that matters while providing peace of mind to management with powerful risk mitigation, compliance, and governance tools. Used by many of the Fortune 100, CloudBees is helping thousands of companies harness the power of continuous everything and gets them on the fastest path from a great idea, to great software, to amazing customer experiences, to being a business that changes lives.

Backed by Matrix Partners, Lightspeed Venture Partners, Verizon Ventures, Delta-v Capital, Golub Capital, and Unusual Ventures, CloudBees was founded in 2010 by former JBoss CTO Sacha Labourey and an elite team of continuous integration, continuous delivery, and DevOps

The CloudBees Community team is seeking a Developer Relations engineer.  In this role, you’ll own developer relations projects to increase developer adoption of Jenkins.  You’ll share ideas, foster community interactions, highlight effective software development practices, and inspire others to help the Jenkins project.  You will develop, test, and deliver crucial open source improvements.  You’ll interact with leading software engineers in our team and in the Jenkins community.  

You will be part of a team that is encouraging and assisting others as they adopt Jenkins.  You will strengthen and support the people, processes, infrastructure, and software in the Jenkins project.  You will interact with an international community of open source contributors both inside and outside of CloudBees.  Your work will be widely visible and deeply appreciated by users around the world.

What You’ll Do

  • Define and deliver projects that increase developer adoption of Jenkins
  • Improve the onboarding experience for new contributors to the Jenkins project by blog posts, writing documentation, and presenting online meetups
  • Lead outreach efforts to bring new developers into the Jenkins community
  • Mentor new participants in Jenkins development.  Use the results of those mentoring efforts to improve the developer experience for others
  • Document, describe, and present results both inside and outside CloudBees
  • Work closely with team members, the Jenkins community, the Continuous Delivery Foundation and other entities to identify and prioritize improvements and include those improvements in the Jenkins roadmap
  • Communicate effectively inside and outside CloudBees

What the Role Requires

  • Deep experience with Jenkins and its use in software development
  • Strong English language communications skills, both written and verbal
  • Java development skills for work in Jenkins core and Jenkins plugins
  • Willingness to learn rapidly and interact with others while learning
  • Strong attention to detail; ability to monitor, manage, and track work, and present data accurately to support decision making
  • Good organization and planning skills with the ability to anticipate needs and to determine the best use of time

Desired skills

  • Presentation, blogging, and advocacy experience
  • Hands-on experience with Docker, Kubernetes, or OpenShift and related container platform ecosystems.
  • Solid understanding of Amazon Web Services; Azure, Google Cloud
  • Release engineering or continuous delivery experience

What You'll Get

  • Highly competitive benefits and vacation package
  • Ability to work for one of the fastest growing companies with some of the most talented people in the industry
  • Team outings
  • Fun, Hardworking, and Casual Environment
  • Endless Growth Opportunities

We have a culture of movers and shakers and are leading the way for everyone else with a vision to transform the industry. We are authentic in who we are. We believe in our abilities and strengths to change the world for the better. Being inclusive and working together is at the heart of everything we do. We are naturally curious. We ask the right questions, challenge what can be done differently and come up with intelligent solutions to the problems we find. If that’s you, get ready to bee impactful and join the hive.

At CloudBees, we truly believe that the more diverse we are, the better we serve our customers. A global community like Jenkins demands a global focus from CloudBees. Organizations with greater diversity—gender, racial, ethnic, and global—are stronger partners to their customers. Whether by creating more innovative products, or better understanding our worldwide customers, or establishing a stronger cross-section of cultural leadership skills, diversity strengthens all aspects of the CloudBees organization.

In the technology industry, diversity creates a competitive advantage. CloudBees customers demand technologies from us that solve their software development, and therefore their business problems, so that they can better serve their own customers. CloudBees attributes much of its success to its worldwide workforce and commitment to global diversity, which opens our proprietary software to innovative ideas from anywhere. Along the way, we have witnessed firsthand how employees, partners, and customers with diverse perspectives and experiences contribute to creative problem solving and better solutions for our customers and their business.

Front End Engineer
EMEA

OUR CUSTOMERS DEVELOP SOFTWARE AT THE SPEED OF IDEAS

CloudBees, the enterprise software delivery company, provides the industry’s leading DevOps technology platform. CloudBees enables developers to focus on what they do best: Build stuff that matters while providing peace of mind to management with powerful risk mitigation, compliance, and governance tools. Used by many of the Fortune 100, CloudBees is helping thousands of companies harness the power of continuous everything and gets them on the fastest path from a great idea, to great software, to amazing customer experiences, to being a business that changes lives.

Backed by Matrix Partners, Lightspeed Venture Partners, Verizon Ventures, Delta-v Capital, Golub Capital, and Unusual Ventures, CloudBees was founded in 2010 by former JBoss CTO Sacha Labourey and an elite team of continuous integration, continuous delivery, and DevOps professionals.

WHAT YOU WILL BE PART OF

CloudBees Feature Management is a feature flag management solution used by engineering and product teams to mitigate release risk, improve developer productivity, and shorten user feedback loops.

Forward-looking enterprises use feature flags as a tool to expedite their continuous delivery and bring new innovation to market faster, with less risk. 

WHAT YOU’LL DO

  • Write quality code using Typescript, React and Jest
  • Own your code end to end. This means every aspect, starting from the design phase all the way through coding, building, deploying, and monitoring it after production
  • Work closely with product managers, designers, technical writers, and support engineers to ensure our user experience is as simple and frictionless as possible
  • Be mindful of security and privacy concerns and follow best practices.
  • Demonstrate flexibility by assisting with other projects when needed

WHAT THE ROLE REQUIRES

  • 2+ years experience  with React (including Hooks and Redux)
  • Solid knowledge of Javascript, Typescript and HTML/CSS
  • Solid Experience with JavaScript build tooling (e.g. WebPack and Babel)
  • Solid Experience with either Playwright, Cypress, Selenium or similar e2e testing framework
  • Experience with Bootstrap, React Bootstrap and good understanding of the internal workings of these specific or similar frameworks
  • Familiarity with Cloud Technologies such as GCP, AWS, Docker, Kubernetes and Helm preferred 
  • Familiarity with continuous delivery and continuous deployment practices 
  • Comfortable with working in a Agile environment with a high degree of autonomy and accountability
  • Strong desire to constantly learn new technologies with a general passion for improving developer productivity
  • Adaptability and a positive attitude

WHAT YOU’LL GET

  • Highly competitive benefits and vacation package
  • Ability to work for one of the fastest growing companies with some of the most talented people in the industry
  • Fun, Hardworking, and Casual Environment
  • Endless Growth Opportunities

We have a culture of movers and shakers and are leading the way for everyone else with a vision to transform the industry. We are authentic in who we are. We believe in our abilities and strengths to change the world for the better. Being inclusive and working together is at the heart of everything we do. We are naturally curious. We ask the right questions, challenge what can be done differently and come up with intelligent solutions to the problems we find. If that’s you, get ready to bee impactful and join the hive.

At CloudBees, we truly believe that the more diverse we are, the better we serve our customers. A global community like Jenkins demands a global focus from CloudBees. Organizations with greater diversity—gender, racial, ethnic, and global—are stronger partners to their customers. Whether by creating more innovative products, or better understanding our worldwide customers, or establishing a stronger cross-section of cultural leadership skills, diversity strengthens all aspects of the CloudBees organization.

In the technology industry, diversity creates a competitive advantage. CloudBees customers demand technologies from us that solve their software development, and therefore their business problems, so that they can better serve their own customers. CloudBees attributes much of its success to its worldwide workforce and commitment to global diversity, which opens our proprietary software to innovative ideas from anywhere. Along the way, we have witnessed firsthand how employees, partners, and customers with diverse perspectives and experiences contribute to creative problem solving and better solutions for our customers and their business.

Verified by
Technical Evangelist
You may also like