Sam Phillips Startups, Technology, Agile.

Continuous Deployment With Feature Flagging August 27, 2012

An important part of our product development process is being able to deploy new versions of features to only a portion of our users, so that we can gather feedback and to improve whilst not disrupting the workflows of our entire customer base.

We do this using feature flags - this mixes perfectly with both continuous deployment and having a customer-focused process, and in this post I'll discuss how we did it and the lessons we've learned.

Rollout gem

We chose to use the rollout gem to fulfil this requirement - it's simple, lightweight and has just the right amount of features. For a detailed introduction to rollout, check out the Railscast Pro episode.

We were keen to allow our product managers to be able to activate features for certain users using a UI - rather than developers activating users through a CLI. For this, we used [rollout ui](, which is a simple rails engine/sinatra app that interfaces with the rollout redis stores. Through a into a directory and deploy to heroku, and you're ready to rock. Check out the interface above - again, simple and just enough features to be useful.

Integrating into the application

So, we start with a test:

The step implementation is pretty straightforward - simple, generic features that call methods from a helper.

&hellips; and the helper:

As you can see, we're disabling all rollout features after each run - this is just as essential as clearing out your primary transactional database.

In terms of the implementation, we were keen to avoid sprinkling conditional logic/renders/routes throughout our application - so as to preserve code quality and to make the code easy to remove later. As such, we utilised the decorator pattern to decorate the subject objects with methods that could be used in the view:

&hellips; and the different versions of the decorator would provide different view artefacts:

Here, you can see that we implement two methods:

  1. `navigation_partial`, which rendes navigation links, and
  2. `show_path`, which returns the path to the current resource.

In version 2 of the decorator, these methods are totally different:

&hellips; as you can see, we've retired the navigation partial entirely (but the view doesn't need to know that, as this method will just return `nil`), and we've moved the resource to a RESTful location. By using this approach, the controller and view both stay entirely clear of conditional logic.

Code is temporary

Next time you're agonising over the smallest detail of a piece of code you've written, remember that *all code is temporary* and, just like your project, will one day not exist. In the case of code written to support and test dual-running during feature rollout, its lifetime is often only a couple of weeks.

Our policy is that once a feature is ready to be released to all customers, then we make it the 'default' version of that feature, and remove any dual running. Using separate decorators and controllers rather than hundreds of `if` statements dotted around your codebase means that when a feature is ready to be deployed to the entire user base, you can remove this code easily and with confidence.

Integrating into our process

To represent feature flagging within our value stream, we added a 'beta' column as the penultimate state for a given piece of work. Not all tasks/features stay in this column (some skip it), but having the column there gives us visibility into the features that are currently in this state.

The customer is the focus of our process - our first column after the icebox is 'Customer Conversations', which represents the initial requirements refinement that happens with customers. In this context, customers means 'end user' - requirements gathering, business analysis and liaising with internal stakeholders follows this stage and refines it further. Having the 'beta' column at the other end of the value stream is a great way of understanding the success, in customer terms, of the work we've done to address their use cases.

Check out the sketch adjacent to see these portions of the board in action.

Lessons learned

Beta does not mean 'done'

It's easy to get a feature to beta and leave it there for way too long, especially if it won't need active development during the feedback part of the cycle. The 'beta' column gives us great visibility into this and are always very clear about when a feature is in a beta state when talking to stakeholders. This makes it clear that there is still work to be done, and we're not finished yet.

Feedback and internal releases

Although there are up-front benefits to this approach, it is not 'free' in terms of effort, so it's important to make sure that you're actually get the feedback from customers - email them and call them up, and if you're releasing a feature for internal review, chase down your most fussy users and watch them use what you've released.

Redis reliability and hosting

In terms of technical implementation, this is the biggest question mark. Redis cloud/SaaS hosting is very much in its infancy, and we've found even the better-known providers to be far from 'highly-available' - by adding redis as a first-class dependency to your application, you're introducing a reliance on your redis host that may not have existed before.

Deploying code and releasing features are not the same thing

The change in thinking that's required for this approach is pretty simple - to un-learn the idea that pushing code to a production system is the only way to release new features to users, and that the two actions are inseparable. In fact, releasing features need not be a binary on/off state - it can be a sliding scale where you can roll things out gradually (and if they're not quite right, roll them back).

As a development team, it's great to be able to keep code for new features in your `master` branch, knowing that they're protected by a feature switch and won't be released to users without your knowledge. Once you can do this, you can use branching solely for grouping changes to code together, rather than having to 'hold' work in branches because it can't be deployed yet.

I love this idea, because it means that feature rollout benefits both the engineering practices of a development team at the same time as providing a useful function to the wider business.

PS: There's a talk!

This blog post is also a talk - most recently given at LRUG in July 2012. The talk gives additional background into other approaches to continuous deployment, and also discusses examples of how we've used this approach. Slides below!