Terraform(ing) Docker hosts with LinuxKit on-premise (VMware vCenter)

Preface

This blog post aims not be a fully-fledged step by step tutorial on how you can create and bootstrap a Docker Swarm cluster with VMware vCenter on-premise. Instead, it should give you an idea about what is possible and why we do it this way. There are different ways to achieve different goals and the way we explain here shows only one way how you can do it.

Today we are running around 30 Docker hosts which are installed as classic VMware virtual mashines and these Docker hosts are all based on Ubuntu Linux. This leads to the fact, that we have to update our Docker hosts every three months manually to get in touch with the up-to-date Docker-ce version. Pointless work. We are provisioning our Docker hosts with puppet. Even if we are provisioning our hosts with puppet, it would take time to create new ones to let the Docker Swarm workers rotate. Yes, we could create a blue-green infrastructure to rotate them, but this will use computing resources if we create them beforehand or let them running all the time. If we would create them on-demand, every three month, it would take time to bring them up and running, update them, and so on. This is now the point to introduce a new players, Docker LinuxKit as Docker host OS and HashiCorp Terraform as IaC (Infrastructure as Code). Yes, there is also Docker InfraKit an we are currently evaluating it, but there will be more work to do.

LinuxKit

LinuxKit is the operating system project started by Docker to provide a toolkit for building custom minimal, immutable Linux distributions. The benefit you get if you choose to go with LinuxKit is not only to have a custom made Linux distribution which reflects your needs, furthermore you get a platform which enables you to push a resulting iso-image to a VMware datastore for example. There are are different cloud providers already build into the LinuxKit toolkit.

LinuxKit example

Downloading and building LinuxKit is very well described on the LinuxKit github page. After you have build the LinuxKit binary, you need a yaml file that describes the compositions of the LinuxKit operating system you would like to create. The example linuxkit.yaml included in the sources is a good starting point. But after some work you will recognize that the simple example will maybe be not enough. Therefore, the next lines will show our basic example where we are starting from to include some additional packages we need.

LinuxKit example explanation

The first thing you need to know is, that LinuxKit uses containerd as container engine to power most of the needed operating system daemons. As you can see in the linuxkit.yml, the different sections are using images as known as container-images to build up the system. Important: This images are using the OCI open container initiative image standard. I will reference back to this point later, so just keep this information in mind.

The first line of the linuxkit.yml takes a Linux kernel image and loads it. The file is not using any :latest tag as image definition and you would like to avoid them too because you would to know what versions are operating in your operation system. Please do not copy the file as it is, because it will be outdated nearly immediately!

After the definition of the kernel there comes the init section. In this section, there are those things located which will be needed immediately for example, the containerd image. This image will be responsible for the upcoming service containers.

Next in the row, there are the services you will probably need for your environment. The open-vm-tools image for needed for us, because we are running the resulting image on the VMware ESXi infrastructure and without the tools, it would not be possible to retrieve the ip address information from the VMware vCenter and this is a must, as we are going to build the virtual machines with Terraform later.

The ssh daemon should be self explaining, but you will need some root/.ssh/authorized_keys to access the running LinuxKit OS from the outside. Therefore, look at the files section, where you can describe your configuration needs.

Now we are installing the Docker engine because we would like build up a Docker Swarm worker and this Docker Swarm worker should join an existing Docker swarm manager later. Important: You will notice, that for the Docker engine, we are using the Docker hub image as usual, no special LinuxKit image! But how does this work? As I said before, the containerd is using OCI standard images which are not the same as Docker images. Lets have a look into it.

LinuxKit persistent disk magic

When it comes to the point, that you have to startup your Docker swarm cluster or even a Docker swarm host with LinuxKit, you will sooner or later ask yourself how you can manage it to persist your Docker Swarm data or at least the information which Docker containers are started.

The Docker data lives inside the /var/lib/docker folder. Therefore, if you persist this folder, you are able to persist the current state of the Docker host.

The collegues at Docker done their work the right way. Lock at the lines, where the images format and mount are loaded. This two images are doing the magic which enables LinuxKit to persit it’s data. You can lookup the documentation at github for details, but for the impatient, here’s the summary.

The format image will take the first block device it finds and if there is no Linux partition on it, it will format it. If it finds a Linux partition, nothing happens. Very comfortable! After the disk format is done (or not) the partition gets mounted via the mount image an the corresponding mountie configuration lines. Et voila there you go. Magical persistence with LinuxKit iso-turbo-boost mode. Genius!

This is one of the most important features, when it comes to infrastructure as code because regardless what you take, InfraKit or Terraform, you might eventually need some kind of persistent to get a reliable infrastructure.

LinuxKit OCI and Docker images

The really cool stuff about LinuxKit is, that it is a toolset. This means, that during the linux build command, which we will see later, the image components as described by the linuxkit.yml are downloaded from the Docker hub and afterwards the content of the used images are transformed to OCI compatible root image filesystems. This means, you can use all the Docker hub images directly without worrying about the image format. Neat! Know you know, why you need a Docker environment to build LinuxKit and to build your LinuxKit OS afterwards.

LinuxKit build your image

To build your LinuxKit iso-image, you can use the following command. We have created a separate docker.yml file, to reflect the changes. The resulting iso-image will therefore be named docker.iso automatically.

LinuxKit push image to VMware

After you have build your iso-image, you can push it to a VMware datastore with the following command. Important: In the actual version of LinuxKit is a small problem located. You cannot push to a VMware datacenter if there are multiple hosts located in. Thanks to Dan Finneran @thebsdbox who helped me a lot, there is already a github PR merged which make it possible to push without a problem. The PR will be included in the next version (0.2) of LinuxKit.

Terraform

Sure, at some point in time we will use InfraKit maybe to get our things up and running. But as for today there are hardly alternatives to Terraform. Terraform is Open Source Software but you can purchase an enterprise license if you need. When you start to dig into the parts and pieces of Terrform, you will recognize, that is not the easiest piece of software but incredible powerful.

Just download Terraform from the website and unpack it. The only thing you will see after extraction is just a binaray called terraform.

Before you can do anything with it, you will need a configuration which discribes, what you would like to receive. Due to this concept, Terraform is working with states. You plan a state, you apply a state and you will destroy a state. Terraform will try to keep you resources consistent. Now it comes to the config.

Terraform example

If you run this example with ./terraform plan and terraform, you will get three LinuxKit VMware virtual machines which are saving their persistent state to the hard drive.

Conclusion

You can use the information in from this blog post to build up a basic infrastructure. But to build up a running Docker Swarm cluster with managers and workers there is more work to do. There are a few glitches which makes it a little bit complicated at the moment to use the VMware customization against the LinuxKit running virtual machines for example. In turn, it might be better to go with VMware templates at the moment, because you will have more possibilities. Maybe we will post an update on this soon maybe. Stay tuned!

Mario Kleinsasser on GithubMario Kleinsasser on LinkedinMario Kleinsasser on Twitter
Mario Kleinsasser
Mario Kleinsasser

Doing Linux since 2000 and containers since 2009. Like to hack new and interesting stuff. Containers, Python, DevOps, automation and so on. Interested in science and I like to read (if I found the time). Einstein said “Imagination is more important than knowledge. For knowledge is limited.” – I say “The distance between faith and knowledge is infinite. (c) by me”. Interesting contacts are always welcome – nice to meet you out there – if you like, don’t hesitate and contact me! – M



DevOps Gathering 2018

In February 2018 I will give my first official talk at the DevOps Gathering 2018 at Bochum Germany. Now I would like to write some lines about why I am doing it and I would also like to suggest you this conference to attend to. And now, here comes the story.

Peter Rossbach is one of the founders of the bee42 gmbh and the bee42 gmbh is one of the conference organizers. Bernhard and I know Peter since ten years now. He is one of the people who you met and never forget. We did a lot of work together for our Apache Tomcat installations at work in the past and therefore we always discussed a lot of things beside the work. One of this things was Docker, back then it was 2013. 2013 it was way to early for us to jump on the Docker container train. We already had container (OpenVZ) on premise which we implemented with Kir Kolyshkin.

We tried Docker on a regular yearly basis the next years but there was too much, which was not working as we need it to use it in production. After some years have passed this way, we started a new try this year. And yes, we managed it to reach the point where things began to run smooth. This year we did a lot for our developers, our company, our customers and ourselfs (the Ops) and Docker was our enabler. Therefore my talk is called Docker: Ops unleashed.

But now, for me, it is time to give something back – back to the community! And therefore I’ve decided to give a talk, to share our experience and to share what the motivation could be to make a change. I am going to do this on my own, privately. At the moment it is winter and together with my colleague we founded a Docker Meetup in south Austria and I am proud to be now allowed to call myself “Docker Community Leader”. But there was more. The Docker Con EU 17 was really great. We met a lot of great people and got a lot of insights!

Why I always wrote “we”? We, because you are probably lost without a team and a team is more than the sum of the individual skills. If you have someone to share your thoughts with, you can go even further! You might be motivated to give a talk for example.

Finally, if you can manage it to come to the [DevOps Gathering 2018](https://devops-gathering.io/) at Bochum Germany please come! It will be a great conference and I am sure, there will be a great audience too. And maybe we can meet us there!

Have a lot of fun!

-M

Mario Kleinsasser on GithubMario Kleinsasser on LinkedinMario Kleinsasser on Twitter
Mario Kleinsasser
Mario Kleinsasser

Doing Linux since 2000 and containers since 2009. Like to hack new and interesting stuff. Containers, Python, DevOps, automation and so on. Interested in science and I like to read (if I found the time). Einstein said “Imagination is more important than knowledge. For knowledge is limited.” – I say “The distance between faith and knowledge is infinite. (c) by me”. Interesting contacts are always welcome – nice to meet you out there – if you like, don’t hesitate and contact me! – M



1. Meetup: Allgemeines & Umfrage / General & Survey

We are planning a Docker Meetup in the next months. Therefore we have created a Google forms poll which you can see below. The poll is provided in German language as we are currently expecting only German talking people to come to our first Meetup. The description will only be provided in German language at the moment. If you do not understand German but you would like to attend (if you are from Italy or Slovenia for example), please contact us! If you are interested to hold a lightning talk or if you would like to share your Docker story (approximately 10 minutes) in English, please contact us. You can find our contact information in the left menu or you can head over to our Meetup page.

Wir planen in den nächsten Monaten ein erstes Docker Meetup in Spittal an der Drau. Vorraussichtlich wird das Meetup in den Räumlichkeiten des bfi-Spittal stattfinden. Aufgrund der Platzsituation müssen wir die Teilnehmeranzahl für das Meetup auf 12 Personen begrenzen! Die Organisation des Meetups (Agenda, RSVP, …) wird über unsere Meetup Seite erfolgen (Anmeldung erforderlich). Ein Termin für dieses Meetup steht noch nicht fest, da wir zuerst die Themen sammeln möchten, welche für die Teilnehmer und Teilnehmerinnen von Interesse sind. Aus diesem Grund findet ihr unterhalb dieser Zeilen eine entsprechende Umfrage, mit der Bitte diese auszufüllen.

Die bereits vorgeschlagenen Themen kommen von Bernhard Rausch (CI/CD mit GitLab) und mir (Mario Kleinsasser, Docker 101), da dies Themenbereiche sind, die wir selber aufgrund unserer Erfahrung sehr gut kennen. Solltet ihr weitere Vorschläge haben, so könnt ihr diese gerne bei der Umfrage angeben.

Den Zeitrahmen für das Meetup haben wir mit 2-3 Stunden festgelegt, wobei der Start des Meetups voraussichtlich um 18:30 Uhr sein wird.

Vielen Dank für das Ausfüllen der Umfrage!

Umfrage

Meetup Treffpunkt

Mario Kleinsasser on GithubMario Kleinsasser on LinkedinMario Kleinsasser on Twitter
Mario Kleinsasser
Mario Kleinsasser

Doing Linux since 2000 and containers since 2009. Like to hack new and interesting stuff. Containers, Python, DevOps, automation and so on. Interested in science and I like to read (if I found the time). Einstein said “Imagination is more important than knowledge. For knowledge is limited.” – I say “The distance between faith and knowledge is infinite. (c) by me”. Interesting contacts are always welcome – nice to meet you out there – if you like, don’t hesitate and contact me! – M



Older blog entries...