Terraform(ing) Blue-Green Swarms of Docker (VMware vCenter)

Preface

Terraform(ing) Blue-Green Swarms of Docker will enable you to update your Docker Swarm hosts to the actual Docker-CE version without an outage. For example imagine the following situation. Your have some Docker Swarm manager up and running and of course a bunch of Docker Swarm workers. If you are forced to update your operating system or if you like to update from a previous version of Docker to a newer one, you will have to handle this change it in place on your Docker Swarm workers. The result is, that you will drain one Docker host, update it and bring it back active to the Docker swarm. If you have five Docker Swarm worker hosts this will result in a loss of a fifth of your capacity and the remaining Docker Swarm worker hosts will have to handle a plus of a twentieth of workload. And if something goes wrong, maybe the new Docker version have a bug which hits you, you might be out of order shortly.

Therefore it is much better, if you can create fresh Docker Swarm workers side by side with the existing ones and then, if all is up and running, you can drain on old version Docker Swarm worker. The load will be pulled over to the new Docker Swarm workers and if something goes wrong, you can just switch back by activating the old Docker Swarm worker host and draining the new Docker Swarm workers afterwards.

The downside of working this way is, that you need a lot of resources while you are running both, the blue and the green Docker Swarms workers and you have to install the Docker Swarm worker hosts. The first issue will cost money, the second one time. Loosing time is always worse, therefore we will use Terraform to do it for us.

Prerequisites

You will need existing Docker Swarm managers to do this job. In the best case the Docker Swarm managers are not used as shared Docker Swarm workers, they should not have workload containers running. They do not need to have as much resources as the Docker Swarm workers. If you handle it this way, you can update the Docker Swarm mangers during work hours without any hassle. Therefore, separate the Docker Swarm mangers from your Docker Swarm workers.

It might be possible to create the Docker Swarm managers through Terraform, but that is not an easy task. Terraform has only limited provisioning capabilities, which is obvious but evident as it is a tool to build infrastructure. Don’t use it to handle software installation tasks. If you need them, use Puppet, Chef, whatever or write something yourself

Example Terraform file

Terraform file explaination

In this Terraform file we use VMware templates to distinguish between the Ubuntu versions and the installed Docker versions. It is similar to the Docker image usage (line 56). We are using PowerDNS to register the Docker Worker hosts automatically in our DevOps DNS (optional, lines 65-70). The most important part of this Terraform file are the provisioners (lines 123-134). These lines will take care, that the newly created Docker host will join the Docker Swarm as worker and of course it will leave the Docker Swarm, if you destroy the Docker Swarm workers through Terraform destroy.

Conclusion

You can take this file as a boilerplate. You can use this file to bring up the blue Docker Swarm workers. Later, you copy this file, change the configuration eg. ip addresses, and bring up the green Docker Swarm workers. After you have transferred the workload from blue to green, you can destroy the blue Docker Swarm worker and prepare for the next update. Todos: You might need to put a small script on your Terraform created Docker Swarm worker hosts to perform additional tasks after creation or before destroy. For example, the PowerDNS entry creation is a bad hack, because it deletes all entries. It would be better to have a script which does this task after startup from the Docker Swarm worker host point of view.

Have fun -M

Mario Kleinsasser on GithubMario Kleinsasser on LinkedinMario Kleinsasser on Twitter
Mario Kleinsasser
Mario Kleinsasser
Doing Linux since 2000 and containers since 2009. Like to hack new and interesting stuff. Containers, Python, DevOps, automation and so on. Interested in science and I like to read (if I found the time). Einstein said "Imagination is more important than knowledge. For knowledge is limited." - I say "The distance between faith and knowledge is infinite. (c) by me". Interesting contacts are always welcome - nice to meet you out there - if you like, don't hesitate and contact me! - M