In this post I’m going to walk through setting up a single-node Kubernetes cluster for personal use. I’ll start from scratch on a newly-provisioned host and end up with a Kubernetes “cluster” with Ingress capabilities. The cluster this process makes is very similar to the cluster I run in my home (see “How this differs from a physical cluster” below).
Why would you want this
A single-node “cluster” (I’m gonna dispense with the scare-quotes from here on) sounds like a heck of a lot of setup when you could just hack together some systemd units and call it a day. Maybe that’s all you need! Some of the reasons I like having this setup are that it’s:
- a place to experiment with Kubernetes outside of work
- a way to deploy personal projects to “real” TLS encrypted URLs with minimal overhead
- the starting point for a lab to experiment with distributed systems
- an environment (outside of work) where I can try out automated infrastructure patterns
To build a cluster, you’ll need:
- a server running Linux: This post’s examples are from a Digital Ocean droplet with 4GB of RAM running Ubuntu 16.04.
$ doctl compute droplet list Name Public IPv4 Memory VCPUs Image Status ubuntu-2gb-nyc3-01 18.104.22.168 4096 2 Ubuntu 16.04.1 x64 active
At home, I’m running the same OS on a physical server. You don’t have to use a totally-fresh server, but it’ll make things easier - we will be taking over port 80 and port 443.
- a domain name: You’ll get the most benefit if you can set a wildcard DNS record pointing to your server. I’ll be using
burnitdown.bizas the domain for my cluster throughout these steps, where I’ve set
*.burnitdown.bizto point to my droplet.
$ dig kube.burnitdown.biz +short 22.214.171.124 $ dig somerandomsubdomain.burnitdown.biz +short 126.96.36.199
If you can’t set wildcard records you’ll get through these steps fine with only
irc.burnitdown.biz pointing to your server.
- an email address: You’ll use this as the contact for your TLS certificates from LetsEncrypt.
I’ll be using
kubeadm, a tool from the Kubernetes project for bootstrapping clusters on arbitrary Linux boxes (as opposed to tools like kops targeted at specific clouds). For the
kubeadm portions of this article I’ll be leaning heavily on its getting started guide.
These instructions will show how to get relevant packages and Kubernetes manifests directly from their source. I have, however, collected up the manifests used in a GitHub repo in case you want to see them all in one place.
1) Install Kubernetes packages: the kubeadm guide’s step 1 includes details for how to set up a Google package mirror that includes all the packages we need:
$ curl -s https://packages.cloud.google.com/apt/doc/apt-key.gpg | apt-key add - OK $ cat <<EOF > /etc/apt/sources.list.d/kubernetes.list > deb http://apt.kubernetes.io/ kubernetes-xenial main > EOF $ apt-get update # ... $ apt-get install -y docker.io # ... $ apt-get install -y kubelet kubeadm kubectl kubernetes-cni # ...
2) Generate cluster configs: the
kubeadm init command generates the cluster’s various configuration files and starts up its services. Before we jump in and run it without options, it’s important to understand some implications of its defaults:
--service-cidr=10.96.0.0/12You should make sure the IP range doesn’t overlap with anything routeable from your host - otherwise you might end up making either your service network unreachable, or masking over other machines you need to use!
10.96.0.0/12is less likely to matter than the default for the pod network - stay tuned.
$ ip route default via 188.8.131.52 dev eth0 onlink 10.17.0.0/16 dev eth0 proto kernel scope link src 10.17.0.5 184.108.40.206/18 dev eth0 proto kernel scope link src 220.127.116.11 172.17.0.0/16 dev docker0 proto kernel scope link src 172.17.0.1 linkdown
My droplet’s network doesn’t overlap with
10.96.0.0/12 - its internal interface is on
10.17.0.0/16, and it additionally listens to a (non-overlapping) public IP. I’ll leave this setting at the default.
--api-external-dns-namesThis option allows us to specify additional CNs for the certificate
kubeadm initgenerates for the
kube-apiserverservice. By default, it tries to detect hostname and IP addresses and make them CNs. That is probably fine on my droplet because
kubeadmcan discover the public IP, but things are trickier on my home cluster (see “Extra considerations for a physical cluster” below). I’ll pass
--api-external-dns-names=kube.burnitdown.bizfor consistency with that setup.
That’s all I’m worried about here. If your setup is meaningfully different from mine, the reference is handy for learning about other options and their implications.
My run of
kubeadm init looks like:
$ kubeadm init --api-external-dns-names=kube.burnitdown.biz [kubeadm] WARNING: kubeadm is in alpha, please do not use it for production clusters. [preflight] Running pre-flight checks [init] Using Kubernetes version: v1.5.3 [tokens] Generated token: "5bd33a.90b60427802097f5" [certificates] Generated Certificate Authority key and certificate. [certificates] Generated API Server key and certificate [certificates] Generated Service Account signing keys [certificates] Created keys and certificates in "/etc/kubernetes/pki" [kubeconfig] Wrote KubeConfig file to disk: "/etc/kubernetes/kubelet.conf" [kubeconfig] Wrote KubeConfig file to disk: "/etc/kubernetes/admin.conf" [apiclient] Created API client, waiting for the control plane to become ready [apiclient] All control plane components are healthy after 38.063960 seconds [apiclient] Waiting for at least one node to register and become ready [apiclient] First node is ready after 4.507374 seconds [apiclient] Creating a test deployment [apiclient] Test deployment succeeded [token-discovery] Created the kube-discovery deployment, waiting for it to become ready [token-discovery] kube-discovery is ready after 14.005653 seconds [addons] Created essential addon: kube-proxy [addons] Created essential addon: kube-dns Your Kubernetes master has initialized successfully! You should now deploy a pod network to the cluster. Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at: http://kubernetes.io/docs/admin/addons/ You can now join any number of machines by running the following on each node: kubeadm join --token=5bd33a.90b60427802097f5 18.104.22.168 # note: yes this is the real token, but no, this cluster doesn't exist anymore
kubeadm has a bunch of features for multi-node clusters (like the join token in the output above!). One of those “features” is that, by default, Kubernetes wont schedule non-system pods on the master node. Since this cluster is only going to be one node, I’ll remove the node taint that causes it to shirk duty:
$ kubectl taint nodes --all dedicated- node "ubuntu-2gb-nyc3-01" tainted
3) Set up a pod network:
kubeadm has no default pod network - we must choose from between the CNI options. There’s a handy list of your options on the Kubernetes website, and a lot of reasons why we might pick one over the other (check out Julia Evans’s awesome post). I like Calico, and this is my blog, so that’s what we’ll use for this cluster. Project Calico provides handy
kubeadm-targeted documentation. We could just blindly apply the manifest they provide, but the defaults have implications.
The pod network that their manifest creates uses the CIDR
192.168.0.0/16. For the droplet, this is totally fine, but on my home network, that’s the network my WiFi router uses! Edit the manifest to use a range that works for you if the default one doesn’t.
On the droplet, this applies like so:
$ wget -q http://docs.projectcalico.org/v2.0/getting-started/kubernetes/installation/hosted/kubeadm/calico.yaml $ kubectl apply -f calico.yaml configmap "calico-config" created daemonset "calico-etcd" created service "calico-etcd" created daemonset "calico-node" created deployment "calico-policy-controller" created job "configure-calico" created
4) breathe! That about wraps up the
kubeadm portion! If all has gone well, you’ve got a single-node cluster running the basic set of Kubernetes controllers. Mine seems to be doing well:
$ kubectl get pods --namespace kube-system NAME READY STATUS RESTARTS AGE calico-etcd-l41vm 1/1 Running 0 1m calico-node-208wc 2/2 Running 0 1m calico-policy-controller-917753764-5l3dh 1/1 Running 0 1m dummy-2088944543-x56wp 1/1 Running 0 3m etcd-ubuntu-2gb-nyc3-01 1/1 Running 0 2m kube-apiserver-ubuntu-2gb-nyc3-01 1/1 Running 0 3m kube-controller-manager-ubuntu-2gb-nyc3-01 1/1 Running 0 3m kube-discovery-1769846148-rgm2z 1/1 Running 0 3m kube-dns-2924299975-s3h9l 4/4 Running 0 2m kube-proxy-vtnvl 1/1 Running 0 2m kube-scheduler-ubuntu-2gb-nyc3-01 1/1 Running 0 2m
Super rad! Now, on to setting up an Ingress Controller so the cluster can easily expose services to the Internet.
5) Install nginx ingress: in order to expose web services on port 80/443 with publicly resolvable domain names, we need some sort of load balancer that is aware of services running on the cluster and able to route between them. Conveniently, that’s exactly what Ingress Controllers do. I’m going to use the nginx-based ingress controller, but if your host is running somewhere with “real” load balancers (like AWS ELBs) it might make sense to check out other options.
The Ingress project has documentation for how to set up on a kubeadm cluster - I’ll follow that:
$ wget -q https://raw.githubusercontent.com/kubernetes/ingress/master/examples/deployment/nginx/kubeadm/nginx-ingress-controller.yaml $ kubectl apply -f nginx-ingress-controller.yaml deployment "default-http-backend" created service "default-http-backend" created deployment "nginx-ingress-controller" created
6) Install kube-lego: this nifty add-on to
nginx-ingress-controller watches for new ingresses and fetches LetsEncrypt certs for the relevant domains! It makes having proper TLS super easy! They provide manifests we can apply here, but you will need to put your own email address in the
kube-lego-configmap.yaml before you apply it.
$ wget -q -O kube-lego-namespace.yaml https://raw.githubusercontent.com/jetstack/kube-lego/master/examples/nginx/lego/00-namespace.yaml $ wget -q -O kube-lego-configmap.yaml https://raw.githubusercontent.com/jetstack/kube-lego/master/examples/nginx/lego/configmap.yaml $ wget -q -O kube-lego-deployment.yaml https://raw.githubusercontent.com/jetstack/kube-lego/master/examples/nginx/lego/deployment.yaml # put my email address in the right place... $ vim kube-lego-configmap.yaml # ...then apply the manifests $ kubectl apply -f kube-lego-namespace.yaml namespace "kube-lego" created $ kubectl apply -f kube-lego-configmap.yaml configmap "kube-lego" created $ kubectl apply -f kube-lego-deployment.yaml deployment "kube-lego" created
7) Run some services! We should be all set! To put the cluster through it’s paces, I’ve set up a few example applications.
$ wget -q https://raw.githubusercontent.com/sophaskins/setting-up-a-personal-kube-cluster/master/glowing-bear.yaml $ kubectl apply -f glowing-bear.yaml # wait a couple of seconds for kube-lego to grab the cert # check it out at https://irc.burnitdown.biz $ curl https://irc.burnitdown.biz <!DOCTYPE html> <html ng-app="weechat" ng-cloak> <head> <meta charset="utf-8"> <meta http-equiv="X-UA-Compatible" content="IE=Edge"> <meta name="viewport" content="width=device-width, initial-scale=1.0, maximum-scale=1.0, user-scalable=no"> <meta name="apple-mobile-web-app-capable" content="yes"> <meta name="mobile-web-app-capable" content="yes"> # ...
In a matter of a few seconds, I just deployed web service, put it behind a (admittedly somewhat lowbrow) load-balancer, and automatically fetched a (browser-valid!) TLS certificate. The future is now!
We can run more than just static JS apps - my second example is a Kubernetes deployment of the super-useful EVE Online tool Evepraisal. It’s a Python Flask app that parses lists of items, looks up their prices in the popular EVE market systems, and tells you how much the list is worth. I put together a Docker wrapper and Kubernetes manifests for it:
$ wget -q https://raw.githubusercontent.com/sophaskins/setting-up-a-personal-kube-cluster/master/evepraisal.yaml $ kubectl apply -f evepraisal.yaml # wait a couple of seconds for kube-lego to grab the cert # check it out at https://evepraisal.burnitdown.biz
Wicked rad! Now I’ve verified that I can serve multiple different websites (with different certificates!) from the same IP. Pretty cool!
Extra considerations for a physical cluster
I run pretty much this exact setup at home on a small PC. The main ways it differs are because my PC doesn’t have an internet-routeable IP address - it connects via NAT to my cable modem. I’ve port-forwarded many ports (including 80 and 443) from my external IP to my server, so for the most part things work transparently. One place it doesn’t is
kubeadm discovery of the IPs to generate certificates for. I like to use
kubectl from other hosts than just directly on my server.
This gets down to why I made sure to pass
--api-external-dns-names=kube.burnitdown.biz to kubeadm earlier. This option adds CNs to the TLS certificate generated for
kube-apiserver - it needs to be valid for whatever name I’m accessing it under via kubectl. Since
kube.burnitdown.biz is a valid name for the server (even if it’s sitting behind NAT, like in my home cluster), kubectl will be happy with it.
Also, as I mentioned in step 3 “Set up a pod network”, the default pod CIDR Calico uses is
192.168.0.0/16, which very much interferes with my home network. Changing it to a non-overlapping RFC 1918 subnet worked great!
Some issues I ran in to while making this post (and maybe you’ll hit too!) are:
kube-dnsconfused - I wrote another blog post about this! The gist is, if your
/etc/resolv.confon the host includes
127.0.0.1, you might have trouble.
- Calico’s etcd doesn’t clean up after itself - if you’re iterating a lot on cluster configuration and
kubeadmoptions, you might use
kubeadm resetto blow away your current configuration and start again. If you do, one thing to note is that the Calico configuration I use runs its own etcd on Kubernetes, which stores its data in the host’s
kubeadm resetcleans up after the Kubernetes etcd cluster (which stores data in the host’s
/var/lib/etcd), it wont clean up the Calico one - you have to do that yourself.
- The “sock shop” example app uses a lot of RAM: the kubeadm getting started guide suggests a sample application to run run on your cluster, the Sock Shop (a basic ecommerce site that sells socks). While this is a pretty neat example app with a lot of depth, it consumes a lot of RAM. I initially was going to use it in step 7 “Run some services!” but it ran my droplet out of memory! I ran it on my (much beefier) home machine - simply starting it up seems to take 4.5 GB of RAM.