Simple Apache Based Blue Green Deployments

Overview

Blue / Green Deployment is a common pattern for managing deployment of a sizeable estate to allow teams to release frequently. It gives you a simple way of smoke testing deployments in the environment which they will run before cutting over to the new servers. It also gives you intrinsic rollback support as you can cut back to the old slice in case of problems.

This article will discuss some of the more practical aspects of bringing this pattern to life in a JVM based stack. It will mainly focus on the cutover, as I think there's plenty of literature already out there which discusses the deployment of packages themselves.

Blue green Deployment

Stack

Lets start by discussing the technology that we use to deliver our software to production.

  • httpd/apache2 - This will be used as a health check to determine if the node should be serving traffic. It will also be used to report on what colour a machine, for the purposes of automating scripts.

  • Tomcat - This will be the application server on which our software is deployed. Obviously this is pretty much interchangable with your favourite application server.

  • Jenkins - This will be .used to orchestrate the cutover, with a build step that plugs in after your deploy

  • Ant - Simple and lightweight, this will be the tool which actually performs the cutover. It's interchangable with gradle or bash directly.

Loadbalancer Healthchecks

Apache / Httpd Setup

First lets configure httpd to serve some sort of simple response on whether its staging or active. This assumes you have no existing httpd installation, if you do, you can probably figure out enough to expose a virtualhost on 81 to do the same thing.

Set Listen Address To :81

This is important because, in all likelyhood, you're going to be serving your main traffic on 80 / 443 and we don't want the health check to interfere with that. In your /etc/httpd/conf/httpd.conf set the following contents

ServerName my_server_name  
ServerRoot /etc/httpd  
PidFile /var/run/httpd/httpd.pid  
User apache  
Group apache  
Timeout 400  
ErrorLog /var/log/httpd/error_log  
LogLevel warn  
KeepAlive On  
MaxKeepAliveRequests 100  
KeepAliveTimeout 5  
DefaultType None  
HostnameLookups off

Listen 0.0.0.0:81

Include conf.d/*.conf  
Include conf.d/*.load  

and then in your /etc/httpd/conf.d/default.conf set contents to something like :

<VirtualHost *:81>  
    ServerAdmin cxtdev@betgenius.com

    DocumentRoot /var/www/html
    CustomLog /var/log/httpd/cxthealth.log combined


    <Directory /var/www/html>
        Options Indexes FollowSymLinks MultiViews
        AllowOverride None
        Order allow,deny
        allow from all
    </Directory>

    LogLevel warn
</VirtualHost>

Next in your /var/www/html directory create two files lb_check and colour. Colour should either contain Green or Blue and lb_check should contain LIVE or STAGING. I suggest using some simple scheme to separate the servers like odds and events but its actually pretty flexible. Obviously one colour should be Live and another Staging.

Start apache and test this setup by navigating to both http://your_server_name:81/lb_check and http://your_server_name:81/colour and checking a response is correctly issued.

Configuring Loadbalancer

The next step is to configure our load balancer to test the lb_check response to determine whether a server is supposed to be active or staging.

At Betgenius, we use f5s to loadbalance web traffic to our servers. They support some quite simple health checks which can be configured to check a particular url on a particular port and expect a particular response. We configure two pools, one to serve live traffic and one to serve staging traffic ( i.e. for performance tests ). Each pool should have both the live and staging nodes assigned, where 'activeness' is determined by the health check. This health check should be looking for LIVE for the live pool and STAGING for the staging pool on port 81, which is what we set up earlier in httpd.

You should now be able to see two pools where the active nodes are flipped. Here's how one of ours looks

health

Scripting The Cutover

We're using ant to do our deployments currently on the Connextra team, I know we have other teams who have slightly nicer deployment tooling and we'll probably invest some more time in this in the near future, but it works for now so i'll discuss how we use it.

Firstly, we actually have only minimal coupling between the blue green paradigm and our deploymently scripts. Rather there is a common deployment script which we use to deploy a release version of the software, and the environment is configurable. With that in mind, we simply configure separate environments for green and blue.

Scripting the change in machine status can be as easy as

<macrodef name="make-machine-live">  
    <attribute name="hostname"/>
    <sequential>
         <echo>Making @{hostname} LIVE…</echo>
         <sshexec host="@{hostname}" username="apache" password="${apachepw}"
           command="echo LIVE > /var/www/html/lb_check" trust="true" failonerror="false"/>
    </sequential>
</macrodef>  
<macrodef name="make-machine-dead">  
    <attribute name="hostname"/>
    <sequential>
        <echo>Making @{hostname} STAGING…</echo>
        <sshexec host="@{hostname}" username="apache" password="${apachepw}"
                     command="echo STAGING > /var/www/html/lb_check" trust="true" failonerror="false"/>
    </sequential>
</macrodef>  

And thus enabling a group becomes as easy as

<macrodef name="enable-group">  
   <attribute name="hostname-list"/>
   <sequential>
       <echo>Enabling ${application.version} for @{hostname-list}…</echo>
       <for list="@{hostname-list}" delimiter=" " trim="true" param="hostname" parallel="true">
          <sequential>
               <make-machine-live hostname="@{hostname}"/>
          </sequential>
       </for>
   </sequential>
</macrodef>  

Disabling as easy as

 <macrodef name="disable-group">
        <attribute name="hostname-list"/>
        <sequential>
            <echo>Disabling ${application.version} for @{hostname-list}…</echo>
            <for list="@{hostname-list}" delimiter=" " trim="true" param="hostname" parallel="true">
                <sequential>
                    <make-machine-dead hostname="@{hostname}"/>
                </sequential>
            </for>
        </sequential>
    </macrodef>

Jenkins

Orchestrating the cutover using jenkins simple.

Your jenkins job should enable one slice and disable the other. I would suggest to hook it up as a manual step after your deployment.

Heres how the build section of our job looks.

jenkins_build

This particular job assumes you're trying to activate the blue slice and deactivate the green.

And Thats It!

Hopefully this can serve as a simple guide on how to plug blue/green deployments into your stack, in a way thats hassle free and quick!

comments powered by Disqus