Motivation

I’m interviewing some European companies for cybersec positions, working towards OSCP and OSWE certifications and started doing bug bounty for a while. For managing CTF boxes, developing bounty hunting automation tools/services I use docker. Awhile back I was contacted by a British fintech company recruiter for a security engineering position. I was asked if I knew docker and kubernetes or not. For the kubernetes part I said that I never used it and very positively was told that “no kubernetes experience” would not work, but the recruiter didn’t close the door on me and told that he could arrange a call if this kubernetes experience issue changes. Although I knew that I would need kubernetes someday, it worked for me without it up until this point and postponed learning kubernetes. So I took this as a challenge and this blog post is the product of a much “postponed” kubernetes learning adventure from a devsecops and security engineering point of view.

What is docker and what problem does it solve?

Let’s say you are developing a web application(surprise surprise). You probably will need applications like a database, a messaging server and a web server. While setting up this application stack you may have compatibility issues with the underlying OS. One of the services maybe incompatible with the chosen OS so that you start checking other OSes. When you solve OS comapatiblity issues you may encounter library dependency issues. DB may depend on one version of a library but the web server may depend on another version of the same library. These issues get real big real quick. This is called “the dependency hell”. There are huge setup costs and you will probably end up with different development-test-production environments which we generally try to avoid while developing applications. All these problems are effectively solved by using containers.

Docker containers

Welcome to docker world. You can run each component in a seperate docker container, independant from the underlying OS and libraries. So this means your DB or other services get its own isolated environment with whatever the libraries they depend on. This is it. You will have a consistent stack for everyone and the “works on my machine” problem will be gone for the good. Now we will happily grow number of docker containers.

Container orchestration

We solved compatibility and dependency issues, started employing growth engineers. The number of users increase steadily, codebase is growing and now we are in need of scaling the web app. What do you do? You run another instance of your app’s decoupled part with just one command: “docker run mybeautifulapp”. Now you have two instances of your dockerized app. If your users keep increasing you now know what to do. Run the command again and voila. You have another instance of your app. If the load increases just run the command again. Watch the load and performance of the instances and when needed run another instance. Also remember to check the health of of your containers. If one dies run another one. But what if the performance of the server is suffering “which is the host for all docker instances”. For solving all these problems related to state performance and health you may pay a dedicated engineer. But when you have large applications consisting of hundreds of containers, paying an engineer to manually do these tasks will not be a practical approach. You may script all these tasks but container orchestration is the industrial solution for just that. Now we are in kubernetestan.

Kubernetes container orchestration architecture and components

In a kubernetes cluster there are three main components. Master node(yes we hope this master/slave terminlogy changes to leader/follower soon), worker node, and CNI(container network interface) which connects these two. There maybe one or more “masters”. On a master node there are several other components like etcd(a key value store), API server, scheduler, controller manager. API server component is the main component which talks to scheduler and controller manager. Another important component of a kubernetes cluster is the CNI(container network interface) which lets worker nodes talk to master inside a kubernetes cluster. When you are installing a kubernetes cluster you first install a master with a control plane and etcd store. After this you will install worker nodes with kubelet components. Kubelet is the main component on a worker node which talk to the API server so that the master knows about the CPU, memory and network usage by kubelets. You will also setup a network to provide a virtual network for your apps on worker nodes.

Architecture

The easiest way to learn about kubernetes concepts on your local PC is a minikube installation.

curl -LO https://storage.googleapis.com/minikube/releases/latest/minikube_latest_amd64.deb
sudo dpkg -i minikube_latest_amd64.deb
minikube start

While installing minikube creates ~/.kube/config file. And from this file kubectl knows how to find the cluster and do things.

Now kubectl installation:

sudo apt-get update

sudo apt-get install -y apt-transport-https curl
curl -s https://packages.cloud.google.com/apt/doc/apt-key.gpg | sudo apt-key add -
cat <<EOF | sudo tee /etc/apt/sources.list.d/kubernetes.list
deb https://apt.kubernetes.io/ kubernetes-xenial main
EOF


sudo apt-get update

sudo apt-get install -y kubelet kubeadm kubectl

sudo apt-mark hold kubelet kubeadm kubectl

As an emacs user I was very happy to find quite a few kubernetes related modes. I tried all of the modes and these are the ones which I find useful: kubernetes, kubernetes-helm, kubernetes-tramp, k8s-mode.

Although these modes are very powerful you sometimes will need kubectl. For those rare moments bash completion for kubectl won’t hurt you. You can get it by issuing this simple command:

source <(kubectl completion bash) 

Your installation should be ready to run. Thats all for part 1.