In February 2018 I will give my first official talk at the DevOps Gathering 2018 at Bochum Germany. Now I would like to write some lines about why I am doing it and I would also like to suggest you this conference to attend to. And now, here comes the story.
Peter Rossbach is one of the founders of the bee42 gmbh and the bee42 gmbh is one of the conference organizers. Bernhard and I know Peter since ten years now. He is one of the people who you met and never forget. We did a lot of work together for our Apache Tomcat installations at work in the past and therefore we always discussed a lot of things beside the work. One of this things was Docker, back then it was 2013. 2013 it was way to early for us to jump on the Docker container train. We already had container (OpenVZ) on premise which we implemented with Kir Kolyshkin.
We tried Docker on a regular yearly basis the next years but there was too much, which was not working as we need it to use it in production. After some years have passed this way, we started a new try this year. And yes, we managed it to reach the point where things began to run smooth. This year we did a lot for our developers, our company, our customers and ourselfs (the Ops) and Docker was our enabler. Therefore my talk is called Docker: Ops unleashed.
But now, for me, it is time to give something back – back to the community! And therefore I’ve decided to give a talk, to share our experience and to share what the motivation could be to make a change. I am going to do this on my own, privately. At the moment it is winter and together with my colleague we founded a Docker Meetup in south Austria and I am proud to be now allowed to call myself “Docker Community Leader”. But there was more. The Docker Con EU 17 was really great. We met a lot of great people and got a lot of insights!
Why I always wrote “we”? We, because you are probably lost without a team and a team is more than the sum of the individual skills. If you have someone to share your thoughts with, you can go even further! You might be motivated to give a talk for example.
Finally, if you can manage it to come to the [DevOps Gathering 2018](https://devops-gathering.io/) at Bochum Germany please come! It will be a great conference and I am sure, there will be a great audience too. And maybe we can meet us there!
Have a lot of fun!
We are planning a Docker Meetup in the next months. Therefore we have created a Google forms poll which you can see below. The poll is provided in German language as we are currently expecting only German talking people to come to our first Meetup. The description will only be provided in German language at the moment. If you do not understand German but you would like to attend (if you are from Italy or Slovenia for example), please contact us! If you are interested to hold a lightning talk or if you would like to share your Docker story (approximately 10 minutes) in English, please contact us. You can find our contact information in the left menu or you can head over to our Meetup page.
Wir planen in den nächsten Monaten ein erstes Docker Meetup in Spittal an der Drau. Vorraussichtlich wird das Meetup in den Räumlichkeiten des bfi-Spittal stattfinden. Aufgrund der Platzsituation müssen wir die Teilnehmeranzahl für das Meetup auf 12 Personen begrenzen! Die Organisation des Meetups (Agenda, RSVP, …) wird über unsere Meetup Seite erfolgen (Anmeldung erforderlich). Ein Termin für dieses Meetup steht noch nicht fest, da wir zuerst die Themen sammeln möchten, welche für die Teilnehmer und Teilnehmerinnen von Interesse sind. Aus diesem Grund findet ihr unterhalb dieser Zeilen eine entsprechende Umfrage, mit der Bitte diese auszufüllen.
Die bereits vorgeschlagenen Themen kommen von Bernhard Rausch (CI/CD mit GitLab) und mir (Mario Kleinsasser, Docker 101), da dies Themenbereiche sind, die wir selber aufgrund unserer Erfahrung sehr gut kennen. Solltet ihr weitere Vorschläge haben, so könnt ihr diese gerne bei der Umfrage angeben.
Den Zeitrahmen für das Meetup haben wir mit 2-3 Stunden festgelegt, wobei der Start des Meetups voraussichtlich um 18:30 Uhr sein wird.
Vielen Dank für das Ausfüllen der Umfrage!
Wird in Kürze bekannt gegeben.
Vim. Yes I am a long term Vim user and I think I still will be a Vim user in a couple of years too. But today I would like to show you, that Microsoft Code is really a great addition for sometimes Golang coders like me. Yes I now, Vim is also a great Golang development IDE but often I need some visualization of a Markdown file for example also.
So lets have a look at Microsoft Code:
The installation is straight forward. Head over to the official download page and download and install the appropriate package for your operating system. In my case, this is Linux (Ubuntu to be precise). After the installation, you can startup the editor via the startup menu of your operating system or you can just type in code in your console.
If you do this, and you are using a remote graphical session like X2GO or XRDP or something similar, nothing will happen, because there is currently a little problem with this setup. But, you can solve this. Just read this issue and at the bottom of it, you can find the (Ubuntu) solution.
After Microsoft Code opens, you should change some user settings. You can open the settings screen through the menu or you can press crtl+shift+p which opens the quick command palette. Not start typing, for example “open settings” and select the correct entry to edit the user settings. Here are the settings, which I have overwritten in my environment.
First, I disabled auto save. In my opinion, this is a little bit annoying if you are writing code, as every time you type, the code is parsed immediately which can slowdown the editor performance a lot. Second, I lower the font size. The default of 14 is to huge for me. I like it to see more code on screen to follow the flow of a code easier. go.toolsGopath is important, as it tells the Microsoft Code golang-plugin – more on this later – where to install the plugin dependencies. go.inferGopath is also important because it tells golang-plugin, to use the actual opened folder as GOPATH variable – this is really useful.
Open the extension browser (crtl+string+p – “extension install”), search for the golang extension and install it. The installation will also install all golang extension dependencies automatically, for example go-lint. All the dependencies will be installed in the path which is defined by the go.toolsGopath.
govendor is a simple golang solution to resolve your project dependencies. You can get it from this Github project page. It is pretty easy to install and use. Just read through the documentation and follow the given steps :-).
If you have setup all correctly, you will get a nice and useful editor for graphical desktop environments. Microsoft Code is no replacement for VIM or Emacs or the editor of your choice but it is a useful and powerful addition.
If you make a mistake and do not correct it, this is called a mistake. [Confucius]
Today we faced a problem with our Docker Swarm which was caused by a permanently restarting service. The focused service was Prometheus, which we use for the monitoring of our Docker environment.
The story starts in the middle of the last week, as the new Prometheus (version 2) was released. In the configuration of our docker-swarm.yml which we use for the Prometheus service, we stupidly still used prometheus:latest. Did you noticed the latest? We have been warned (at Docker Con) to not use this. Yes there are a lot of examples on the internet which are exactly using this, but it is a very bad idea. latest literally means unknown because, you will not now, which image is referenced by the latest tag. latest is only a convention, not a guarantee! Therefore, pin the version of the image which you really want to use by pin-pointing it, eg. prometheus:1.1.1.
In our case, caused by an unplanned service update, the Prometheus image was freshly pulled (you now latest) and corrupted the Prometheus database. Furthermore, the configuration of the Prometheus changed between the version which in turn caused a permanent restart of the service. That happened on the weekend, which wouldn’t be bad, but it caused the container engine to get stressed.
This is documented in this Github issue. The result of this bug is, that the syslog get spammed up with a lot of pointless messages. However this will fill up your log partition after some time (maybe hours, but it will get filled up).
At this point it gets icky. In a default Ubuntu setup for example, the /var partition will contain the log directory and of course the lib/docker directory too. If the /var partition of the system is full, also Docker cannot write its Raft data anymore and your Docker Swarm will be nearly dead.
In our case we had a configuration mistake, because we used four Docker Swarm manager nodes, not three and not five. Now we come to the ugly level. Bad luck, the filled up /var partition killed two of our Docker Swarm managers. The containers continued to work, but the cluster orchestration was messed up, because two out of four nodes where dead. No quorum anymore, no consensus, no manageability.
But, no panic, there are ways to bring back online all services with some Linux voodoo (truncating syslog files, …). To sum it up, what are the lessons learned?
- Watch out for the correct number of Docker Swarm managers (there has to be an odd number of it, 3, 5, 7, …)
- Never ever use the latest tag, if you are not the maintainer of it!
- Restrict syslog to not fill up your partition. Place the syslog logs on a separate partition, or disable the syslog forwarding in the systemd journald config ForwardToSyslog=false. journald’s default configuration is to use a maximum of 10% of the diskspace for the log-data.
- Use LinuxKit maybe. It is made out of containers, which can be restricted to not use all system resources. If you ever asked yourself why you should have a look at it, read number two. The Docker host is not a general purpose server, likely you do not need default syslog and much more. This is what LinuxKit is designed for.
Thats all for today.
(Image by Walter Grassroot Wikipedia)
Ever asked yourself where to find the information what aufs diff folder maps to a running container? It’s pretty simple if you just poke around a little bit in the /var/lib/docker folder. But lets start at the beginning.
In the default Docker engine configuration, aufs is the storage driver which is used for images and container diffs. You can read on about the aufs driver at the official Docker aufs documentation here.
To sum it up, the base entity of every running container is the underlying image. This insight will direct us to the /var/lib/docker/image folder, more exactly to the /var/lib/docker/image/aufs/layerdb/mounts/ folder.
Insight the mounts folder, you can find directories named along your running containers. inside this containers, you will find a file called mount-id. This file contains the information about the aufs id which will help you to find the correct aufs diff directory of the running container in the /var/lib/docker/aufs/diff/ folder.
Here is a full example:
[10:03 /var/lib/docker/image/aufs/layerdb/mounts]# ls
[10:03 /var/lib/docker/image/aufs/layerdb/mounts]# cd 1f500742f31bc22dab85b83805ae88afae23a713cc4c2857fb624eedfcef3c7e/
[10:03 /var/lib/docker/image/aufs/layerdb/mounts/1f500742f31bc22dab85b83805ae88afae23a713cc4c2857fb624eedfcef3c7e]# ls
init-id mount-id parent
[10:03 /var/lib/docker/image/aufs/layerdb/mounts/1f500742f31bc22dab85b83805ae88afae23a713cc4c2857fb624eedfcef3c7e]# cat mount-id
[10:03 /var/lib/docker/image/aufs/layerdb/mounts/1f500742f31bc22dab85b83805ae88afae23a713cc4c2857fb624eedfcef3c7e]# cd /var/lib/docker/aufs/diff/9307f7abdcdca4657002a5084bae91745ecd5b5e6ac11448b00185dce016a54c/
[10:28 /var/lib/docker/aufs/diff/9307f7abdcdca4657002a5084bae91745ecd5b5e6ac11448b00185dce016a54c]# ls
adasd root run var
Happy hacking – M
(Featured image taken from the official Docker aufs documentation here)
First of all
Mario & I where really excited going to DockerCon EU 2017 and didn’t know what was waiting for us. Monday started at about 04:00 AM by driving to Klagenfurt (Austria) and flying to Copenhagen via Vienna. After arriving at Copenhagen airport we managed to buy a metro/bus/train ticket so we could ride to our Hotel near the Bella Center. As we arrived at our Hotel we stored our luggage, connected our Laptops via WiFi and had a short look if everything was fine @work. I had some battery problems with my laptop the whole week (it always needs the power cable to startup) and also with my mobile phone lacking battery power, therefore I was forced to load my mobile via my laptop during the day… But everything went fine in the end and I had enough battery power to stay connected. 🙂
Arriving at Bella Center
Afterwards we went to the Bella Center to get registered and get our name badges. In our opinion the location was really big and some work was going on to finalize it for the public crowd. After registration we called an old friend of us to meet – Peter Rossbach. We met him and his team (Niclas Mietz,…) shortly after registration. It was nice to see them and we started to chat about what we were all waiting for. In the evening the welcome registration took place in the expo hall where we got our T-Shirts and also got a sneak peak of what was waiting for us the next days. And for me that was the start for collecting some Docker Pins… 🙂
We also had some nice chats with Peter Rossbach & Niclas Mietz who we met before and also with Docker Captain Dieter Reuter we haven’t met before but knew he was working closely with Peter.
Day one – the keynote
The next day started with the first keynote. But wait, there also was a breakfast before that and yes, that breakfast definitively needs to be mentioned here. 🙂 I haven’t seen such a big conference with such a great catering during the whole week! Starting from the free coffee from the espresso coffee machines going further on to the really good meals/dishes throughout the whole conference.
But what about the keynote? In my opinion the keynote started very well. The speakers gave an overview about the upcoming new features and what the whole keynote tenor (MTA, Modernizing Traditional Applications) was. The demos where a bit odd but funny to watch and the announcement that Kubernetes will get integrated into the Docker ecosystem beside the Docker Swarm was really interesting.
Day two – the keynote
The keynote of day two wasn’t that good in my opinion because the speakers focused too much on the tele prompter for their talks and so the talks dragged on at length… It felt that everyone in the audience was waiting for it to end to start another great day of more meaningful talks. Also, for me, the MetLife plot has been used too much – everything was just “MetLife”… I understand that this is a good partner for Docker to showcase what can be done but it’s not always about enterprise.
Also revealing that IBM is getting another platinum partner with its bluemix so that the customer can now choose from some of the biggest IT companies to assist him in migrating to a Docker environment hasn’t been that announcement the audience was waiting for…
Here comes a description of the tracks which I (Mario) attended including a short recap of them.
Play with Docker (PWD): Inside Out (Community Track)
That was the first track which I attended and it was a great one. Unfortunately there are no slides online but you can watch the video. The track was about the latest changes on the Play with Docker online application and for me, one of the highlights was, that there will be the possibility to use a direct attached ssh console to communicate with the PWD instance. Play with Docker (PWD): Inside Out
Learning Docker from Square One (Using Docker Track)
This one was held by Chloe Condon and as the topic states, it was a basic track about Docker. But it was a very cool track because Chloe is a great, fun and refreshing person. I really liked her art of presentation!
Experience the Swarm API in Virtual Reality (Community Track)
My second community track and it was a little bit a strange one but not a bad one. The speaker created a 3D virtual reality view of the Docker API! Interesting and unusual.
LinuxKit Deep Dive (Black Belt Track)
Wow, this was one of the heavy tracks. In 30 minutes the audience and I got a really taff and very interesting inside view of what happened to and with LinuxKit in the past 6 month. For me, this was great because with the additional information about the Kubernetes integration into Docker the steps made really made sense.
A Story of Cultural Change: PayPal’s 2 Year Journey to 150,000 Containers with Docker (Transform Track)
One of my personal favorites from the Docker Con EU 17! Why? I am not a pure technician, I have to organize a lot of management stuff too but of course I am also a visionary engineer. Therefore I really liked the talk from Meghdoot Bhattacharya about the transition from the classic virtual machine driven infrastructure to the (not perfect at this time) Docker container infrastructure. We never made it this way, because we already started with containers in 2006 (OpenVZ), and as such we are happy to have missed the interim stage of having hundreds or thousands of virtual machines to manage. For me it was a confirmation that we did it right in the past, we do it right in the present and probably we are going to do it the right way in the future.
Tuesday afternoon meeting with Michael Dielman (Docker Booth)
On Tuesday afternoon we had a short meeting with Michael Dielman who is the account executive of Docker for the DACH region. Michael is a really cool person and we had a great inspiring talk. During the meeting we came in contact with to other Docker employees as well, namely Andreas Lambrecht and Andreas Wilke – was nice to meet you guys! Together with Michael we discussed a lot of topics, including the Docker EE version and why we are currently not going to use the Docker EE version on-prem. He did not force us (our company) to switch from the Docker CE version to the Docker EE version, on the contrary he and his collegues were really interested in how me managed it on-prem. One question we were always facing with from nearly all people we met was, if we are using Docker and more specific Docker Swarm in the productive environment. Our answer was, of course yes, why not, followed by a longer explanation how we manage it.
Eureka! The Open Solution to Solving Storage Persistence (Ecosystem B Track)
This was a short talk about the progress of RexRay held by Chris Duchesne which was very interesting too because it addresses one of the main problems in the Docker ecosphere currently – persistent container storage. It was a impressive short talk but it was also very informative and we were able to get a lot of benefits and ideas out of it.
My Journey To Go (Transform Track)
The last track of the first day for me, held by Ashley McNamara and it was my favorite talk number two. The presentation of Ashley was very brisk and inspiring. Hopefully a lot of people will follow her to find a way to contribute and to come in touch with technology.
Tips and Tricks of the Docker Captains (Using Docker Track)
First track of Wednesday, presented by Adrian Mouat. Some of the tips we knew already but there was a bunch of new things inside the presentation for us. A very good and informative track.
Docker to the rescue of an Ops Team (Community Track)
Help by Rachid Zarouali, this was a classic transition track. Interesting insights on how they moved from the “old” fashioned deployment way to the much more modern CI/CD way. Too bad that they did not get further until today in terms of CI/CD, because managing configuration with Puppetmaster for Docker volume mounts is a little bit odd. But maybe, they haven’t finished it until today, so maybe we will hear from him the next time.
Practical Design Patterns in Docker Networking (Using Docker Track)
Another great talk and one of my favorites. Don Finneran gave a lot of information about Docker networking to the audience and yes, there were some things I did not recognize until this talk.
Gordon’s Secret Session: Kubernetes on Docker (Black Belt Track)
Brilliant! A huge talk about the Kubernetes integration on Docker. A lot of information and insights where given to the audience in a very short time.
In the following you can read through my (Bernhard) recaps of the talks I attended.
Play with Docker (PWD): Inside Out (Community Track)
The first one I attended – the talk was really good and we got some insights on the future of PWD! The guys are really working hard on enhancing the features of PWD including a direct attachable SSH console, get rid of the ReCaptcha and also remove the 4 hours limit. Some really nice features for the next releases! It was a really good talk, watch it!
Play with Docker (PWD): Inside Out
What’s New in Docker (Best Practices Track)
Nice talk from Vivek Saraswat getting some more detailed informations on the insights of the newest Docker features. For the Docker CE, which we use in production, Overlay2 over aufs as the new preferred filesystem & adding “–chown” to ADD/COPY in the Dockerfile where the highlights for me in this talk.
What’s New in Docker
Experience the Swarm API in Virtual Reality (Community Track)
Some fancy talk about visualizing the Docker API so you can “walk” through it in virtual reality. Not that important for production use cases but fun to watch.
Experience the Swarm API in Virtual Reality
Tales of Training: Scaling CodeLabs with Swarm Mode and Docker-Compose (Community Track)
The talk was about getting a robust training environment which is scalable, reproducible & simply usable by other trainers. Not that usable for us but nice to see what others implement with Docker.
Tales of Training: Scaling CodeLabs with Swarm Mode and Docker-Compose
Docker?!?! But I’m a SysAdmin (Using Docker Track)
At the first step it’s all about what you specifically need, that sums up what this talk was all about. Ask yourself some questions to take the right decisions for your needs. The best part for me in this talk was that Mike Coleman offered to write him if you have something important to share with the Docker community! Thx for that, we’ll really come back to him!
Docker?!?! But I’m a SysAdmin
Creating Effective Images (Using Docker Track)
Learning about what you can do by yourself to get simple & small container images which only have tools installed needed by the running application.
Creating Effective Images
Tuesday afternoon meeting with Michael Dielman (Docker Booth)
For me it was really cool to meet Michael Dielman and some guys (Andreas Wilke) from Docker who work in europe and which you maybe can contact in the future if there is the need for it. I pretty much enjoyed the chat with Andreas Lambrecht at the after party because we had things in common in our IT history regarding the company branches and so he was understanding why we are working on the things like we do it today.
Eureka! The Open Solution to Solving Storage Persistence (Ecosystem Track)
Sad that whether the video nor the slides are online by now, hopefully they will be uploaded! It was a really great talk by Chris Duchesne getting some informations about RexRay and how persistent container storage could be implemented with it. We are looking forward testing this in our environment!
Back to the Future: Containerize Legacy Applications (Use Case Track)
Modernizing Traditional Applications (MTA) was the tenor of this DockerCon and this talk was about modernizing the infrastructure of your legacy application. After that use this new infrastructure to modernize the application itself by using the new and in any ways more flexible delivery methods to, in the end, deploy your application faster. For me the talk underlined what we are going through in our environment but for us without the tool set for “converting” the application stack to our Docker environment. I think this tool set will help people out there who won’t get into the details of Docker in the first step but have the need to make the step forward to a containerized world.
Back to the Future: Containerize Legacy Applications
Tips and Tricks of the Docker Captains (Using Docker Track)
A good talk to start the second day. We got some tips and tricks around the docker environment. Some of them we already knew & some others we haven’t known already. Everyone using Docker should have a look at the video & slides, I think everyone can take something out of this cool talk by Adrian Mouat!
Tips and Tricks of the Docker Captains
Empowering Docker with Linked Data Principles (Community Track)
Running your applications in the Docker environment gives you masses of metadata around your running application stack. So what if you want to search for some metadata some of your applications have in common? Semantic data maybe is the key. Really nice talk about what you maybe will need in the future…
Empowering Docker with Linked Data Principles
Monitoring Containers: Follow the Data (Ecosystem Track)
Hmm, what should I say. I expected a more technical talk about monitoring containers in a Docker environment rather than a business talk about Datadog…
Container Orchestration from Theory to Practice (Black Belt Track)
Docker to the Rescue of an Ops Team (Community Track)
A talk about the transition to Docker with CI/CD but as Mario wrote above to this talk, I also don’t understand why they use Puppet rather than migrating the whole thing to more CI/CD including configurations… We also use Puppet for our Docker hosts (and of course our non Docker environments) but no further. Maybe we will see more from him in one of the future DockerCon’s tracks. Would be interesting to see their further development.
Docker to the Rescue of an Ops Team
Taking Docker to Production: What You Need to Know and Decide (Using Docker Track)
Cool, what a talk to end DockerCon day two. I’m also a gaming kid of the 80’s/90’s, so following that talk by Bret Fisher was really cool. Not only because of the games but that gave the talk an ease following and concentrating at the end of the day! Summing up the talk I would say “start using docker” the simplest way you are comfortable with, don’t start using Docker and make CI/CD a show stoping requirement and also don’t start with a persistent application like a database container. So, really cool talk, have a look at it, you won’t regret it!
Taking Docker to Production: What You Need to Know and Decide
Bernhard’s track summary
In this short summary I want to outline four of the tracks I attended because they gave really much information which we will be needing and using in our environment.
Favorite Nr 1 -> Mike Coleman – Docker?!?! But I’m a SysAdmin
Favorite Nr 2 -> Chris Duchesne – Eureka! The Open Solution to Solving Storage Persistence (Ecosystem Track)
Favorite Nr 3 -> Adrian Mouat – Tips and Tricks of the Docker Captains
Favorite Nr 4 -> Bret Fisher – Taking Docker to Production: What You Need to Know and Decide
Shuttle service via busses to the DockerCon “after day one” party. Really cool that we hadn’t to look at metro/bus/train plans to find our way to the after party. Everything was organized that everyone is feeling comfortable by making a shuttle service which brought us to the after party and back to the Bella Center after the after party. 😉
Arriving there we went into that old train workshop hall where everyone could drop his/her jackets/backpack so that you could fully enjoy the party! Everything was organized – drinks, meals/dishes, music, arcade games, life sized Asteroids and Bricks & Pong where you could challenge others. I think most of the DockerCon attendees where at the party and enjoyed that evening. There was enough time to meet new people you have already met the first day but also to meet new people and chat with them about their journey to Docker in their environments.
It was a really cool and great organized event and I hope that this wasn’t the last one we’ll attend in the future.
The last day (for us) at DockerCon started a bit bad for me because I forgot to select the “Join Moby Summit” radio button some weeks before the DockerCon so I thought I wouldn’t be able to attend the summit at that day. Mario and I went to the Bella Center and to the breakfast and Mario got into the summit because he checked that early enough before DockerCon.
So i went to the open place outside the Expo area and had a look for a free desk including a power line plug to get my laptop started, you know, my battery problems… So after a while sitting there and trying some things out with Portainer, reading a bit through the RexRay documentation and Portus installation a woman came out of the Expo area and said that there are some free places at the Moby summit. Yes, I thought, and after that I found myself fumbling my equipment back in my backpack in a hurry and literally running to the Moby summit, where Mario was already sitting with a free chair beside him. So I also had the opportunity to listen to the talks from about 09:45. There where some interesting talks about Moby itself, InfraKit & LinuxKit.
After that we unfortunately had to leave to get to our flight back to Austria… A good end for the DockerCon EU 2017!