Level Up With Feature Flags

Sep 28, 2020 - 8 min read

Photo by SpaceX

What if you had the power to release a new feature at a click of a button without a code merge? What about having the power to "turn on" the giant code refactor your team has been working on for the last six months at the flip of a switch? What if you could give the ability for someone in your organization to click that button - outside of the development team? Or what about if you could only release features to a small subset of your users? How is it possible? All of this and more can made possible with feature flagging.

What is feature flagging?

Feature flagging is a technique that allows any authorized member of the team to modify the way a piece of software works at the click of a button, all without needing an immediate code release. It's a powerful way to release new features, bug fixes, or code refactors in a calculated, safe way. I've been using feature flags for around four years at two different companies. This post is meant to educate you if you've never used feature flags, and explain some of the wins and challenges I've face while leveraging feature flags as part of my teams' development cycle.

Simple Example Using React with Launch Darkly

Here's a simple client-side example using Launch Darkly. In this example we are refactoring a page in our app. When the flag for the new page is turned off (evaluates to false) we show the old page, otherwise we show the new one. We configure the SDK in the root of the application. That is where we let the SDK know who the logged in user is.

import { useFlags } from 'launchdarkly-react-client-sdk'


function Page() {
  // Always a good idea to add a default here, especially
  // if the default is true
  const { showNewPage = false } = useFlags() 
  return
    <div>
      {showNewPage && <NewPage />}
      {!showNewPage && <OldPage />}
    </div>
}

"The Traditional Way"

Traditionally, when development teams are ready to start end-to-end testing a new feature, they follow a branching strategy to get their code into the various development environments. Once the feature is tested and they feel confident it's ready for prime time, typically what happens is the code is merged to a master branch, a deployment is triggered, and the new feature is released to the world.

While this approach is effective to ship code, it doesn't come without risk. Once you release your code 100% of your user base can now see your new feature. No matter how effective your test coverage or QA team is, it's highly unlikely you released a perfectly pristine feature that is without bugs or performance issues. If this code happened to introduce a mission critical bug in the system, there is a good chance you don't have a backup plan to revert the release at the click of a button. For bigger features this usually means a long-lived feature branch. If you are not rebasing or merging in from the base branch frequently, you are in big trouble when it comes to finally getting the code into the base branch. Likely you have several merge conflicts to deal with. What a nightmare!

Feature Flagging to the Rescue

Feature flagging can save you many headaches of releasing your code. One of its biggest selling points is risk mitigation. "D-Day" is a term I use to describe the day in the future that you and your team agreed is the day the new feature you have been working for months on will be released to the public. Feature flagging makes your "D-Day" experience less anxious.

Here are a few of the big wins feature flags provide:

Features can get released to a small subset of users slowly over some period of time
The QA team / automated tests can point at production instead of staging / other environments
If something catastrophic goes wrong after releasing a feature, you can revert things back at the click of the button
You as a developer don't have to wake up early to queue the release; you can have any member of the team "flip the switch"
Avoid long running feature branches by getting "feature flagged" code into production quicker.
Regressions with new features are caught quicker in the release cycle which helps keep releases ahead of schedule

Canary Releases

A "Canary Release" is a term that refers to a release technique which starts by making the initial release of a new feature only available to a small subset of your user base. Then, over time you slowly enable the feature for more and more of the user base until everyone is using the new feature. Canary releases are the ultimate risk mitigation. If there are bugs, performance issues or user experience woes, these can get caught quickly and addressed just in time to roll out the feature to a larger subset of users. Feature flags make doing Canary releases super simple. You just need to enable the feature for different segments of your user base and you are ready to go!

Feature Flag Driven Development

When you start coding with the assumption that things will be hidden behind a flag, it changes the way you think about writing your code. Every new thing you ask yourself can we feature flag this? If the thing we are building or changing is going to take us more than a day or two to work on, or it's something that won't be "ready for the world" for a few days or weeks, we will typically feature flag it. Once a branch is created that contains this work, we typically merge this branch early and often to the base branch even if the work is partially completed. We don't feature flag trivial things like small bug fixes, text changes, or small features. Once you start using feature flags you will get a feel for what makes the most sense to feature flag on your team.

Why you Should Seriously Consider Using a Third Party Provider Instead of Rolling Your Own Feature Flag System

Getting started with feature flags can start simple. You can start with a file in your app that contains a list of features and true / false values. This is fine, but it requires a code release if you want to change the file. The next best thing might be to make it dynamic by putting all your features in a database, then reading from the database to check your flag. This is great, too, but it requires a dev to do the work to "flip the switch" and it's still all or nothing.

Unless you are a part of a large organization with deep pockets and plenty of developers to go around, I would vote you go with a third party provider for your feature flags. Most providers are very cheap relative to the value they provide. Making this call was a game changer for both myself and the teams I've been a part of.

Here are some of the things you get for "free" after signing up with a third party provider

A GUI with users to create and enable flags with ease
An SDK in most programming languages to get feature flags in code
An easy way to break up feature flags by environment, so you can turn on flags in certain environments, but keep prod disabled
Real time updates on both client, server side implementations of their SDK
Apis provided to build triggers, mechanisms to set, unset, and create feature flags
A way to build segments of users to make it easier to lump users into groups and enable features for multiple users
You don't have to maintain the system yourself.
The system is already optimized for most use cases
Easy A / B tests

There are several great companies that specialize in feature flagging. They make it their business to provide customers with a reliable, feature-dense system that helps developers do their jobs better. Here are a few solid options:

https://launchdarkly.com/

https://www.optimizely.com/

https://rollout.io/

Some Challenges I've faced

Feature flagging doesn't come with its own set of challenges. My team had an incident recently where the third party provider we were using perfect fine for over six months went down. They were completely down for around 30 minutes. Luckily this only affected us minimally because we were pretty diligent about removing flags for features that have been turned on for everyone. Under the hood using their client-side SDK, the requests silently failed which meant that the end user saw no issues other than the fact that a couple features were missing. We pushed an update which made the default values for a couple features "true" instead of "false", and we were out of the woods. I bring up this story to highlight the fact that using feature flags, whether it be with a third party provider, or one you rolled yourself, doesn't come without risks. Here are some of the issues I've encountered after working with feature flags:

If you use a third party provider, you are at the mercy of their uptime
It can be a challenge to fit this into a QA workflow, and to be transparent about what flags are turned on, and in what environment
Regressions need to be run with the flag on and the flag off
Old feature flags can remain in codebases for long periods of time unless developers are diligent about cleaning them up later
It's possible to accidentally expose new features to users without the feature being done yet, especially if you don't have regression tests

What My Teams Have Done to Mitigate These Issues

My team has become diligent about deleting old flags when they are finally available for all users. We also ensure that new flags default false as to not expose the new feature if we can't access our third party provider.
The stories that QA work on have a field which marks if it is feature flagged or not. This allows them to test stories for new features without it being enabled yet. We also turn all new features on in our dev staging environments
We have a suite of automated tests which regression test all of our environments every hour. If code was deployed or a feature flag was turned on, the regression would get caught within an hour and a developer would be notified.
We set a reminder to go off in our dev Slack channel once a month asking "have you deleted old feature flags?"
Write unit tests for both flag on and flag off

In Summary

Feature flagging is something development teams should try. I've found that they have made a substantial improvement how how often we ship new features. We get the features into our customers' hands sooner because we can go to beta before the feature is fully polished. Our product manager loves it because it's super easy for him to coordinate our release with marketing. He also is able to opt in customers into the beta at the click of a button without bothering the devs. The dev team loves it because we merge branches for new features all day every day, even if the feature is only partially completed. There's no more long lived branches with an insane amount of merge conflicts. I guarantee you will notice the benefits after hooking up your first flag. Feature flags will change your life! Give it a shot - you will never look back.