We are planning a Docker Meetup in the next months. Therefore we have created a Google forms poll which you can see below. The poll is provided in German language as we are currently expecting only German talking people to come to our first Meetup. The description will only be provided in German language at the moment. If you do not understand German but you would like to attend (if you are from Italy or Slovenia for example), please contact us! If you are interested to hold a lightning talk or if you would like to share your Docker story (approximately 10 minutes) in English, please contact us. You can find our contact information in the left menu or you can head over to our Meetup page.
Wir planen in den nächsten Monaten ein erstes Docker Meetup in Spittal an der Drau. Vorraussichtlich wird das Meetup in den Räumlichkeiten des bfi-Spittal stattfinden. Aufgrund der Platzsituation müssen wir die Teilnehmeranzahl für das Meetup auf 12 Personen begrenzen! Die Organisation des Meetups (Agenda, RSVP, …) wird über unsere Meetup Seite erfolgen (Anmeldung erforderlich). Ein Termin für dieses Meetup steht noch nicht fest, da wir zuerst die Themen sammeln möchten, welche für die Teilnehmer und Teilnehmerinnen von Interesse sind. Aus diesem Grund findet ihr unterhalb dieser Zeilen eine entsprechende Umfrage, mit der Bitte diese auszufüllen.
Die bereits vorgeschlagenen Themen kommen von Bernhard Rausch (CI/CD mit GitLab) und mir (Mario Kleinsasser, Docker 101), da dies Themenbereiche sind, die wir selber aufgrund unserer Erfahrung sehr gut kennen. Solltet ihr weitere Vorschläge haben, so könnt ihr diese gerne bei der Umfrage angeben.
Den Zeitrahmen für das Meetup haben wir mit 2-3 Stunden festgelegt, wobei der Start des Meetups voraussichtlich um 18:30 Uhr sein wird.
Vielen Dank für das Ausfüllen der Umfrage!
Vim. Yes I am a long term Vim user and I think I still will be a Vim user in a couple of years too. But today I would like to show you, that Microsoft Code is really a great addition for sometimes Golang coders like me. Yes I now, Vim is also a great Golang development IDE but often I need some visualization of a Markdown file for example also.
So lets have a look at Microsoft Code:
The installation is straight forward. Head over to the official download page and download and install the appropriate package for your operating system. In my case, this is Linux (Ubuntu to be precise). After the installation, you can startup the editor via the startup menu of your operating system or you can just type in code in your console.
If you do this, and you are using a remote graphical session like X2GO or XRDP or something similar, nothing will happen, because there is currently a little problem with this setup. But, you can solve this. Just read this issue and at the bottom of it, you can find the (Ubuntu) solution.
After Microsoft Code opens, you should change some user settings. You can open the settings screen through the menu or you can press crtl+shift+p which opens the quick command palette. Not start typing, for example “open settings” and select the correct entry to edit the user settings. Here are the settings, which I have overwritten in my environment.
First, I disabled auto save. In my opinion, this is a little bit annoying if you are writing code, as every time you type, the code is parsed immediately which can slowdown the editor performance a lot. Second, I lower the font size. The default of 14 is to huge for me. I like it to see more code on screen to follow the flow of a code easier. go.toolsGopath is important, as it tells the Microsoft Code golang-plugin – more on this later – where to install the plugin dependencies. go.inferGopath is also important because it tells golang-plugin, to use the actual opened folder as GOPATH variable – this is really useful.
Open the extension browser (crtl+string+p – “extension install”), search for the golang extension and install it. The installation will also install all golang extension dependencies automatically, for example go-lint. All the dependencies will be installed in the path which is defined by the go.toolsGopath.
govendor is a simple golang solution to resolve your project dependencies. You can get it from this Github project page. It is pretty easy to install and use. Just read through the documentation and follow the given steps :-).
If you have setup all correctly, you will get a nice and useful editor for graphical desktop environments. Microsoft Code is no replacement for VIM or Emacs or the editor of your choice but it is a useful and powerful addition.
If you make a mistake and do not correct it, this is called a mistake. [Confucius]
Today we faced a problem with our Docker Swarm which was caused by a permanently restarting service. The focused service was Prometheus, which we use for the monitoring of our Docker environment.
The story starts in the middle of the last week, as the new Prometheus (version 2) was released. In the configuration of our docker-swarm.yml which we use for the Prometheus service, we stupidly still used prometheus:latest. Did you noticed the latest? We have been warned (at Docker Con) to not use this. Yes there are a lot of examples on the internet which are exactly using this, but it is a very bad idea. latest literally means unknown because, you will not now, which image is referenced by the latest tag. latest is only a convention, not a guarantee! Therefore, pin the version of the image which you really want to use by pin-pointing it, eg. prometheus:1.1.1.
In our case, caused by an unplanned service update, the Prometheus image was freshly pulled (you now latest) and corrupted the Prometheus database. Furthermore, the configuration of the Prometheus changed between the version which in turn caused a permanent restart of the service. That happened on the weekend, which wouldn’t be bad, but it caused the container engine to get stressed.
This is documented in this Github issue. The result of this bug is, that the syslog get spammed up with a lot of pointless messages. However this will fill up your log partition after some time (maybe hours, but it will get filled up).
At this point it gets icky. In a default Ubuntu setup for example, the /var partition will contain the log directory and of course the lib/docker directory too. If the /var partition of the system is full, also Docker cannot write its Raft data anymore and your Docker Swarm will be nearly dead.
In our case we had a configuration mistake, because we used four Docker Swarm manager nodes, not three and not five. Now we come to the ugly level. Bad luck, the filled up /var partition killed two of our Docker Swarm managers. The containers continued to work, but the cluster orchestration was messed up, because two out of four nodes where dead. No quorum anymore, no consensus, no manageability.
But, no panic, there are ways to bring back online all services with some Linux voodoo (truncating syslog files, …). To sum it up, what are the lessons learned?
- Watch out for the correct number of Docker Swarm managers (there has to be an odd number of it, 3, 5, 7, …)
- Never ever use the latest tag, if you are not the maintainer of it!
- Restrict syslog to not fill up your partition. Place the syslog logs on a separate partition, or disable the syslog forwarding in the systemd journald config ForwardToSyslog=false. journald’s default configuration is to use a maximum of 10% of the diskspace for the log-data.
- Use LinuxKit maybe. It is made out of containers, which can be restricted to not use all system resources. If you ever asked yourself why you should have a look at it, read number two. The Docker host is not a general purpose server, likely you do not need default syslog and much more. This is what LinuxKit is designed for.
Thats all for today.
(Image by Walter Grassroot Wikipedia)