Kubernetes and Weave - a no go per default
Estimated reading time: 5 mins
The default Weave deployment for Kubernetes is not as secure as it should and could be. What does this mean? In fact it does mean, that if you setup a Kubernetes cluster via kubeadm and install Weave Works overlay network via the recommended installation guide, any host that installs Weave manually can join this overlay network!
We are reviewing Kubernetes every couple of month, because there is some (small) kind of uncertainty if Docker Swarm will stay alive or not. Therefore it is always good to know if there are alternatives, like Kubernetes, and how they fit in into an existing environment. Docker Swarm is unmatched simply to manage, we drive more than 2300 containers today, but it is not as hyped as Kubernetes. So we have to stay up-to-date how we can use Kubernetes to replace Docker Swarm if we have to. And every single day when we have a look at it, our review boils down to two main problems, networking and storage. Today we will focus on the network integration, CNI for detail.
There are are lot of Kubernetes CNI plugins out there, but some of them are widely used: Calico, Weave and Flannel. Weave is, of course, a nice solution to get up and running with CNI capabilities inside Kubernetes easily. But there is a major drawback.
If you are following the default installation guide for Weave within Kubernetes there is no password used to protect your overlay network against suspicious network peers!
In the following output, the host atlxkube474 is not part of the Kubernetes cluster. But it can join the created Kubernetes Weave network easily by specifying one of the main peers during
Kubernetes cluster nodes:
[19:52 atlxkube471 ~]# kubectl get nodes NAME STATUS ROLES AGE VERSION atlxkube471 Ready master 12d v1.15.3 atlxkube472 Ready <none> 11d v1.15.3 atlxkube473 Ready <none> 11d v1.15.3
Suspicious Weave host:
19:54 atlxkube474 ~]# kubeclt Command 'kubeclt' not found, did you mean: command 'kubectl' from snap kubectl (1.15.3) See 'snap info <snapname>' for additional versions. [19:55 atlxkube474 ~]# weave launch 10.x.x.1 ... truncated output ... INFO: 2019/09/16 17:55:22.483698 sleeve ->[10.x.x.2:6783|c6:cf:11:33:a1:ca(atlxkube472)]: Effective MTU verified at 1438 [19:56 atlxkube474 ~]# eval $(weave env) [19:56 atlxkube474 ~]# weave status Version: 2.5.2 (up to date; next check at 2019/09/17 01:10:02) Service: router Protocol: weave 1..2 Name: da:09:b8:ee:77:3e(atlxkube474) Encryption: disabled PeerDiscovery: enabled Targets: 1 Connections: 3 (3 established) Peers: 4 (with 12 established connections) TrustedSubnets: none Service: ipam Status: ready Range: 10.32.0.0/12 DefaultSubnet: 10.32.0.0/12 Service: dns Domain: weave.local. Upstream: 10.x.x.50, 10.x.x.51, 10.x.x.52 TTL: 1 Entries: 0 Service: proxy Address: unix:///var/run/weave/weave.sock Service: plugin (legacy) DriverName: weave
And now, everyone can run a container that joins the singe weave overlay network and can do anything that is possible:
[20:00 atlxkube471 ~]# kubectl get pods -n deployment-v1 -o wide NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES nginx-deployment-5754944d6c-z486x 1/1 Running 0 6d9h 10.44.0.1 atlxkube472 <none> <none> [19:59 atlxkube474 ~]# docker run --name a1 -ti weaveworks/ubuntu root@a1:/# ping 10.44.0.1 PING 10.44.0.1 (10.44.0.1) 56(84) bytes of data. 64 bytes from 10.44.0.1: icmp_seq=1 ttl=64 time=2.00 ms 64 bytes from 10.44.0.1: icmp_seq=2 ttl=64 time=0.672 ms ^C --- 10.44.0.1 ping statistics --- 2 packets transmitted, 2 received, 0% packet loss, time 1001ms rtt min/avg/max/mdev = 0.672/1.337/2.002/0.665 ms root@a1:/#
This example shows that any host can really, really easily join the Weave overlay network! This is not a secure by default design!
I know, that it is possible to set a password for Weave, which is used to encrypt the network traffic and to protect unknown host to join the Kubernetes created Weave overlay network. This is described here.
Lets do this for our Kubernetes installation right now. Thanks to summerswallow-whi for opening a KOps issue which is already addressing this. The issue is still open (May 2018), 🙁, but it provides a lot of information how you can iron your Weave setup.
I tried it on my own and the following steps are enough to bring some kind of protection to your Weave overlay.
First, create a password for your Weave overlay and save it to a file:
# < /dev/urandom tr -dc A-Za-z0-9 | head -c16 > weave-password
Now create a Kubernetes secret:
# kubectl create secret -n kube-system generic weave-password --from-file=./weave-password
Add this setting to the Weave Kubernetes daemonset by editing it (under the weave-net container spec):
kubectl create secret -n kube-system generic weave-password --from-file=./weave-password ... template: metadata: creationTimestamp: null labels: name: weave-net spec: containers: - command: - /home/weave/launch.sh env: - name: HOSTNAME valueFrom: fieldRef: apiVersion: v1 fieldPath: spec.nodeName - name: WEAVE_PASSWORD valueFrom: secretKeyRef: key: weave-password name: weave-password image: docker.io/weaveworks/weave-kube:2.5.2
If you now try to join the Weave overlay network, you will see the following failure:
[20:19 atlxkube474 ~]# weave launch 10.x.x.1 24ef2192c30e3c5d9372469eb1fb456e348cdd9efe4b8cda27c3ba1e756ba73c [20:19 atlxkube474 ~]# docker logs weave ... INFO: 2019/09/16 18:19:50.677943 ->[10.x.x.1:6783] connection shutting down due to error during handshake: no password specificed, but peer requested an encrypted connection
Yes! This is want we want! And nothing less!
Weave is using, I my opinion, a non secure default setup. This violates the GDPR article 25 at least. Encryption is a default today! This is one of the points where Docker Swarm is much, much better. Docker swarm creates a VXLAN overlay network for each service per default (not just one single overlay for anything like Weave does)! 😎 Furthermore you cannot(!) join a Docker Swarm without knowing the join token and therefore you cannot infiltrate a existing Docker Swarm overlay network! It is secure by default! No additional screws needed!
It is hard to believe, but even Weave Works itself does not provide a Kubernetes documentation about how to enhance your Kubernetes Weave setup (Kubernetes Secrets, Kubernetes Daemonset,…) Even the actual edition of the book Kubernetes: Up and running, Second Edition does not mention something like this in any way.
I don’t like to know, how many insecure setups are out there - hopefully everyone trust his or her own network. 🙄
Sometimes we think, that the whole industry is just fueling up the whole Kubernetes thing to sell more and more consulting services…
But, lets see…