Kubernetes the easy way part 01

  Network, System

In this series of articles I will try to describe how to install and configure a kubernetes cluster in high availability and for production use, using an easy approach opposite to hard way, proposed by https://github.com/kelseyhightower/kubernetes-the-hard-way, that, however, is a good way to understand and learn more, inside the scenes, the infrastructure of a kubernetes cluster.

I have chosen, as first easy solution, a ubuntu deployment strategy that use microk8s for deploying the kubernetes clusters in high availability, that, starting from last year, is ready for Production as described here https://microk8s.io/high-availability. In the next article, continuing in the wake of this spirit, I will deep another Ubuntu juju based solution.

With the awareness of the importance of control of an infrastructure in production, I will try to describe how the microk8s architecture works describing limits and benefits.

The reference cluster will be formed by 3 virtual machines, an haproxy behind the cluster, in order to expose the ingress services to standard http and https ports, and a nfs server to provide a network file system to mount and share inside the cluster, all running in Ubuntu 20.04.2 LTS.

Let me start with the microk8s installation.

Microk8s HA Installation

Before starting with the installation, it’s necessary to have the hostnames of the every virtual machine resolved via dns, if it’s not possible, as I made, I suggest to add in each /etc/hosts file the following association:

microkus01# vi /etc/hosts microk8s01 microk8s02 microk8s03

The apiserver, in order to permit to launch commands inside the pods, by exec commands, must connect to kubelet, port 1025, and it needs to resolve the hostname of the pod where is running. Without this, the kubectl exec command doesn’t work if the pod is running in another node different where the command itself is executed.

Moreover, in order to use useful commands for the administration of the cluster, it’s also necessary to set these alias:

microkus01#alias helm='microk8s helm3' alias kubectl='microk8s kubectl'

I remember that helm, even if not used in this article, has become a standard de facto for kubernetes application management.

Let’s go on with the installation of the microk8s – by snap package manager. that is a way to package application, like rpm, installed under the directory /var/snap – in each node of the cluster. The latest stable release will be installed:

root@microk8s01:~# snap install microk8s --classic --channel=1.21/stable
2021-05-21T07:54:19Z INFO Waiting for automatic snapd restart...
microk8s (1.21/stable) v1.21.1 from Canonical✓ installed
root@microk8s01:~# snap info microk8s
name:      microk8s
summary:   Lightweight Kubernetes for workstations and appliances
publisher: Canonical✓
store-url: https://snapcraft.io/microk8s
contact:   https://github.com/ubuntu/microk8s
license:   unset
description: |
  MicroK8s is the smallest, simplest, pure production Kubernetes for clusters, laptops, IoT and
  Edge, on Intel and ARM. One command installs a single-node K8s cluster with carefully selected
  add-ons on Linux, Windows and macOS.  MicroK8s requires no configuration, supports automatic
  updates and GPU acceleration. Use it for offline development, prototyping, testing, to build your
  CI/CD pipeline or your IoT apps.
  - microk8s.add-node
  - microk8s.cilium
  - microk8s.config
  - microk8s.ctr
  - microk8s.dashboard-proxy
  - microk8s.dbctl
  - microk8s.disable
  - microk8s.enable
  - microk8s.helm
  - microk8s.helm3
  - microk8s.inspect
  - microk8s.istioctl
  - microk8s.join
  - microk8s.juju
  - microk8s.kubectl
  - microk8s.leave
  - microk8s.linkerd
  - microk8s
  - microk8s.refresh-certs
  - microk8s.remove-node
  - microk8s.reset
  - microk8s.start
  - microk8s.status
  - microk8s.stop
  microk8s.daemon-apiserver:            simple, enabled, inactive
  microk8s.daemon-apiserver-kicker:     simple, enabled, active
  microk8s.daemon-cluster-agent:        simple, enabled, active
  microk8s.daemon-containerd:           simple, enabled, active
  microk8s.daemon-control-plane-kicker: simple, enabled, inactive
  microk8s.daemon-controller-manager:   simple, enabled, inactive
  microk8s.daemon-etcd:                 simple, enabled, inactive
  microk8s.daemon-flanneld:             simple, enabled, inactive
  microk8s.daemon-kubelet:              simple, enabled, inactive
  microk8s.daemon-kubelite:             simple, enabled, active
  microk8s.daemon-proxy:                simple, enabled, inactive
  microk8s.daemon-scheduler:            simple, enabled, inactive

In this scenario, each node of the cluster is an independent microk8s cluster. The same command below can be executed, with the same output, in any node.

root@microk8s01:~# kubectl get nodes
microk8s01 NotReady 20s v1.21.1-3+1f02fea99e2268
root@microk8s01:~# kubectl get pods -n kube-system
calico-kube-controllers-f7868dd95-zstn4 1/1 Running 0 30s
calico-node-z4kqw 1/1 Running 0 30s

As showed above, the calico plugin is used as container network interface: it is used for assigning the ip addresses to containers and configure the routing for inter pods communication. There are two way to configure the calico plugin: ip2ip and vxlan tunnel. The microk8s, as showed below, uses the vxlan tunnel (for more information about calico you can read my article https://www.securityandit.com/network/kubernetes-network-cluster-architecture/):

microk8s01:~# kubectl get IPPool
default-ipv4-ippool 11m
root@microk8s01:~# kubectl describe IPool default-ipv4-ippool
error: the server doesn't have a resource type "IPool"
root@microk8s01:~# kubectl describe IPool default-ipv4-ippool -n kube-system
error: the server doesn't have a resource type "IPool"
root@microk8s01:~# kubectl describe IPPool default-ipv4-ippool
Name: default-ipv4-ippool
Annotations: projectcalico.org/metadata: {"uid":"12623a0d-f763-4ec8-b45a-51588dda5e4f","creationTimestamp":"2021-05-21T08:38:12Z"}
API Version: crd.projectcalico.org/v1
Kind: IPPool
Creation Timestamp: 2021-05-21T08:38:12Z
Generation: 1
Managed Fields:
API Version: crd.projectcalico.org/v1
Fields Type: FieldsV1
Manager: Go-http-client
Operation: Update
Time: 2021-05-21T08:38:12Z
Resource Version: 370
Self Link: /apis/crd.projectcalico.org/v1/ippools/default-ipv4-ippool
UID: 172a3dc1-4ca5-45f7-83b5-1d395eb974e6
Block Size: 26
Ipip Mode: Never
Nat Outgoing: true
Node Selector: all()
Vxlan Mode: Always

Calico is my best preferred network plugin because it’s stable and, opposite to other cni like flannel, provides the implementation of the network policy important for cluster segregation.

Microk8s has the possibility to enable different modules that are of two types: simple binary installed, like helm3, that must be installed in any nodes, and kubernetes resources that can be installed in one only node of the cluster. Before creating one, I suggest to install the dns module in the first node, microk8s01, that will be used as starting point of the cluster. Theoretically this module could be installed after forming the cluster.

root@microk8s01:~# microk8s enable dns
Enabling DNS
Applying manifest
serviceaccount/coredns created
configmap/coredns created
deployment.apps/coredns created
service/kube-dns created
clusterrole.rbac.authorization.k8s.io/coredns created
clusterrolebinding.rbac.authorization.k8s.io/coredns created
Restarting kubelet
DNS is enabled
root@microk8s01:~# kubectl get pods -n kube-system
coredns-7f9c69c78c-wjxzq 1/1 Running 0 8m9s
calico-kube-controllers-f7868dd95-zstn4 1/1 Running 0 28m
calico-node-z4kqw 1/1 Running 0 28m

The coredns mounts internally a configmap where is specified the DNS forward: it could be useful to change these with internal dns servers:

root@microk8s01:~# kubectl describe configmap coredns -n kube-system|grep forw
forward .

For creating the the cluster. in the first node, it’s enough to get the authentication token:

root@microk8s01:~# microk8s add-node
From the node you wish to join to this cluster, run the following:
microk8s join the node you are adding is not reachable through the default interface you can use one of the following: microk8s join

In the second node, microk8s02, it’s easy to join in this way:

root@microk8s02:~# snap install microk8s --classic --channel=1.21/stable
microk8s (1.21/stable) v1.21.1 from Canonical✓ installed
root@microk8s02:~# microk8s join
Contacting cluster at
Waiting for this node to finish joining the cluster. ..

The approach, used for adding the node, by bootstrap token, is well explained in kubernetes documentation https://kubernetes.io/docs/reference/access-authn-authz/bootstrap-tokens/ After repeating the same process for the third node, we have a cluster formed by three nodes, that is, as explain later, the minimum number of nodes required.

root@microk8s01:~# kubectl get nodes
microk8s01 Ready 55m v1.21.1-3+1f02fea99e2268
microk8s02 Ready 11m v1.21.1-3+1f02fea99e2268
microk8s03 Ready 5m52s v1.21.1-3+1f02fea99e2268
root@microk8s01:~# kubectl get pods
No resources found in default namespace.
root@microk8s01:~# kubectl get pods -n kube-system
coredns-7f9c69c78c-wjxzq 1/1 Running 0 35m
calico-kube-controllers-f7868dd95-zstn4 1/1 Running 0 55m
calico-node-tz7hk 1/1 Running 0 12m
calico-node-gb5dw 1/1 Running 0 12m
calico-node-2kgj5 1/1 Running 0 6m

It’s time to install the rbac authorization (https://kubernetes.io/docs/reference/access-authn-authz/rbac/) that is the kubernetes way for assigning roles to users and groups that have authenticated to the cluster. In microk8s the default user is admin that is authenticated by token (https://kubernetes.io/docs/reference/access-authn-authz/authentication/).

root@microk8s01:~# microk8s enable rbac
Enabling RBAC
Reconfiguring apiserver
Adding argument --authorization-mode to nodes.
Configuring node
Configuring node
Restarting nodes.
Configuring node
Configuring node
RBAC is enabled

The command configure the apiserver of any node of the clusters adding rbac parameter in its startup command.

The next step is to have a way for exposing the kubernetes service outside of the cluster. There are, as I have deepened in this my article https://www.securityandit.com/network/kubernetes-service-nodeport-ingress-and-loadbalancer/, two solutions: one ingress and the other NodePort based. The solution ingress based is more flexible, scalable and secure. It must be present in any production kubernetes cluster.

Ingress Installation in microk8s

The ingress controller is enabled with ‘microk8s enable ingress’, but unfortunately the nginx is listening on localhost – http and https ports – because, despite the presence of hostPort that forces to listen in the host network namespace, there is the nginx publish-status-address parameter set to

In order to expose the controller, I created a NodePort service for the ingress daemon set and by a haproxy, running in an external machine, proxying the http and https ports to NodePort randomly created by kubernetes.

root@microk8s01:~# microk8s enable ingress
Enabling Ingress
ingressclass.networking.k8s.io/public created
namespace/ingress created
serviceaccount/nginx-ingress-microk8s-serviceaccount created
clusterrole.rbac.authorization.k8s.io/nginx-ingress-microk8s-clusterrole created
role.rbac.authorization.k8s.io/nginx-ingress-microk8s-role created
clusterrolebinding.rbac.authorization.k8s.io/nginx-ingress-microk8s created
rolebinding.rbac.authorization.k8s.io/nginx-ingress-microk8s created
configmap/nginx-load-balancer-microk8s-conf created
configmap/nginx-ingress-tcp-microk8s-conf created
configmap/nginx-ingress-udp-microk8s-conf created
daemonset.apps/nginx-ingress-microk8s-controller created
Ingress is enabled
root@microk8s01:~# kubectl get pods -n ingress
nginx-ingress-microk8s-controller-jhtxq 1/1 Running 0 66s
nginx-ingress-microk8s-controller-dh7rh 1/1 Running 0 66s
nginx-ingress-microk8s-controller-qxxhl 1/1 Running 0 66s
root@microk8s01:~# ss -lp |grep http
root@microk8s01:~# ss -lp |grep https
root@microk8s01:~# vi nodeport-service-ingress.yaml
apiVersion: v1
kind: Service
name: nginx-ingress-microk8s
namespace: ingress
type: NodePort
name: nginx-ingress-microk8s
# By default and for convenience, the targetPort is set to the same value as the port field.
- port: 80
name: http
- port: 443
name: https
root@microk8s01:~# kubectl apply -f nodeport-service-ingress.yaml
service/nginx-ingress-microk8s unchanged
root@microk8s01:~# kubectl get service -n ingress
nginx-ingress-microk8s NodePort 80:30982/TCP,443:32264/TCP 42s

After that, let’s go on to configure an haproxy external load balancer running outside of the cluster, that is the entry point for https traffic. The In this way it’s possible to reach our kubernetes service by a url like https://myservice.mydomain.

root@haproxy:#apt install haproxy
root@haproxy:#vi /etc/haproxy/haproxy.cfg
frontend www.mysite.com
bind ssl crt /etc/ssl/certs/mysite.pem
http-request redirect scheme https unless { ssl_fc }
default_backend web_servers
backend web_servers
balance roundrobin
server server1 check maxconn 20 ssl verify none
server server2 check maxconn 20 ssl verify none
server server2 check maxconn 20 ssl verify none

I installed the haproxy for http redirect in https and for balancing all the https traffic to https node port of the ingress. Following all the other configurations done in haproxy using a self signed certificate ignoring the verification by adding verify none to the server.

root@haproxy:/etc/haproxy# openssl req -newkey rsa:2048 -nodes -keyout key.pem -x509 -days 365 -out certificate.pem
root@haproxy:/etc/haproxy# cat certificate.pem >>/etc/ssl/certs/mysite.pem
root@haproxy:/etc/haproxy# cat key.pem >> /etc/ssl/certs/mysite.pem
root@haproxy:/etc/haproxy# systemctl start haproxy
root@haproxy:/etc/haproxy# systemctl status haproxy
root@microk8s01:/etc/haproxy# curl -v https://www.mysite.com
< date: Fri, 21 May 2021 10:16:27 GMT
< content-type: text/html
< content-length: 146
< strict-transport-security: max-age=15724800; includeSubDomains
404 Not Found

The kubernetes dashboard is ready to be installed and reachable by a normal https url.

Kubernetes Dashboard Installation in microk8s

Kubernetes Dashboard is a general purpose, web-based UI for Kubernetes clusters that can be installed following the instructions in the official github address https://github.com/kubernetes/dashboard, but microk8s provides the installation by module that, behind the scenes uses the same yaml files.

root@microk8s01:/etc/# microk8s enable dashboard
Enabling Kubernetes Dashboard
Enabling Metrics-Server
clusterrole.rbac.authorization.k8s.io/system:aggregated-metrics-reader created
clusterrolebinding.rbac.authorization.k8s.io/metrics-server:system:auth-delegator created
rolebinding.rbac.authorization.k8s.io/metrics-server-auth-reader created
Warning: apiregistration.k8s.io/v1beta1 APIService is deprecated in v1.19+, unavailable in v1.22+; use apiregistration.k8s.io/v1 APIService
apiservice.apiregistration.k8s.io/v1beta1.metrics.k8s.io created
serviceaccount/metrics-server created
deployment.apps/metrics-server created
service/metrics-server created
clusterrole.rbac.authorization.k8s.io/system:metrics-server created
clusterrolebinding.rbac.authorization.k8s.io/system:metrics-server created
clusterrolebinding.rbac.authorization.k8s.io/microk8s-admin created
Adding argument --authentication-token-webhook to nodes.
Configuring node
Configuring node
Configuring node
Restarting nodes.
Configuring node
Configuring node
Configuring node
Metrics-Server is enabled
Applying manifest
serviceaccount/kubernetes-dashboard created
service/kubernetes-dashboard created
secret/kubernetes-dashboard-certs created
secret/kubernetes-dashboard-csrf created
secret/kubernetes-dashboard-key-holder created
configmap/kubernetes-dashboard-settings created
role.rbac.authorization.k8s.io/kubernetes-dashboard created
clusterrole.rbac.authorization.k8s.io/kubernetes-dashboard created
rolebinding.rbac.authorization.k8s.io/kubernetes-dashboard created
clusterrolebinding.rbac.authorization.k8s.io/kubernetes-dashboard created
deployment.apps/kubernetes-dashboard created
service/dashboard-metrics-scraper created
deployment.apps/dashboard-metrics-scraper created

As you can see, there is a strange thing: why it’s necessary to add authentication-token-webhook in every node of the cluster? The answer is because the metric server, that collects metrics like CPU or memory consumption for containers or nodes, must authenticate to kubelet and the authentication-token-webhook parameter added to it has the scope to use the service account token for that.

For exposing the kubernetes dashboard, the best way is to use an ingress created in this way:

root@root@microk8s01:#vi ingress-dashboard.yaml
apiVersion: networking.k8s.io/v1
kind: Ingress
name: k8s-dashboard
namespace: kube-system
kubernetes.io/ingress.class: public
nginx.ingress.kubernetes.io/ssl-passthrough: "true"
nginx.ingress.kubernetes.io/backend-protocol: "HTTPS"
host: k8s-micro-dashboard.mydomain
- path: /
pathType: Prefix
name: kubernetes-dashboard
number: 443
root@root@microk8s01:# kubectl apply -f ingress-dashboard.yaml
root@root@microk8s01:# kubectl get ingress -A |grep dash
kube-system k8s-dashboard k8s-micro-dashboard.mydomain 80, 443 2d
cat <<EOF | kubectl apply -f -
apiVersion: v1
kind: ServiceAccount
name: kubernetes-dashboard
namespace: kube-system
root@root@microk8s01:# cat <<EOF | kubectl apply -f -
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
name: kube-system
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: cluster-admin
kind: ServiceAccount
name: kubernetes-dashboard
namespace: kube-system
root@root@microk8s01#kubectl -n kube-system get secret $(kubectl -n kube-system get sa/kubernetes-dashboard -o jsonpath="{.secrets[0].name}") -o go-template="{{.data.token | base64decode}}"

It’s possible to reach the kubernetes dashboard via https://k8s-dashboard.mydomain, where the virtual host is resolved in the ip address of haproxy that proxies to kubernetes NodePort. The authentication to dashboard is implemented via token, that by rbac, has associated the admin cluster role. In a Production environment, it would be better to have a login that, by username and password, gets roles based to principle of least privilege necessary to perform its function.

For implementing a scenario like that, as kubernetes suggested in https://kubernetes.io/docs/reference/access-authn-authz/authentication/, it’s preferable to use openid connect as authentication strategy. I suggest to use dex, as oidc provider service, and gangway to easily enable authentication flows via OIDC for a kubernetes cluster. Unfortunately, as explained below, in microk8s it’s not possible to add oidc parameters to apiserver and it prevents, until now, to use it.

I noted that microk8s provides as module portainer that is a gui created for docker swarm management and reconverted for kubernetes. Despite having the possibility to manage different clusters and support the ldap authentication, I prefer to use kubernetes dashboard because portainer seems to be a dress born for swarms and badly cut for kubernetes.

For having a kubernetes cluster ready to use in Production, we should have a shared storage that can be mounted inside pods for sharing data, config or to centralize the logs.

Storage configuration

In microk8s it’s possible to enable the storage module – that creates a host path storage class for creating dynamic persistent volumes that mount a file or directory from the host node’s filesystem into your Pod. This is a volume that cannot be shared between pods and is not very useful in a Production environment.

In order to to this, it’s enough to enable the module in this way:

root@root@microk8s01:#microk8s enable storage
root@root@microk8s01:#kubectl get sc NAME PROVISIONER RECLAIMPOLICY VOLUMEBINDINGMODE ALLOWVOLUMEEXPANSION AGE microk8s-hostpath (default) microk8s.io/hostpath Delete Immediate false 14d

For production environment, it’s better to have a network file system shared that can be mounted as read-write by many nodes. We have two choices:

  1. Create a nfs persistent volume that will be bound when the persistent volume claim requests it by storage class.
  2. Install a nfs external provisioner that creates the nfs persistent volume dynamically when the persistent volume claim requests it by storage class..

In this case I will install a nfs external provisioner supposing to have a external nfs file server reachable at address:

root@root@microk8s01:#shelm repo add nfs-subdir-external-provisioner https://kubernetes-sigs.github.io/nfs-subdir-external-provisioner/
root@root@microk8s01:#microk8s helm install nfs-subdir-external-provisioner nfs-subdir-external-provisioner/nfs-subdir-external-provisioner \ --set nfs.server=\ --set nfs.path=/data-nfs/microk8s
root@root@microk8s01:#microk8s kubectl get sc NAME PROVISIONER RECLAIMPOLICY VOLUMEBINDINGMODE ALLOWVOLUMEEXPANSION AGE microk8s-hostpath (default) microk8s.io/hostpath Delete Immediate false 14d nfs-client cluster.local/nfs-subdir-external-provisioner Delete Immediate true 14d

The nfs server must have a local directory accessible to all microk8s nodes where the /etc/exports is configured in this way:

/data/microk8s,sync,no_root_squash) /data/microk8s,sync,no_root_squash) /data/microk8s,sync,no_root_squash)
root@root@microk8s01:#exportfs -a

The nfs volumes can be created in this simple way:

root@microk8s01:# cat < apiVersion: v1
> kind: PersistentVolumeClaim
> metadata:
>   name: nfs-volume
> spec:
>   storageClassName: nfs-client
>   accessModes:
>     - ReadWriteMany
>   resources:
>     requests:
>       storage: 5Gi
persistentvolumeclaim/nfs-volume created
root@microk8s01:# kubectl get pv
NAME                                       CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS   CLAIM                 STORAGECLASS        REASON   AGE
pvc-ec5ffba7-20a8-5fab-b974-721a35b9c1f8   5Gi        RWX            Delete           Bound    default/nfs-volume    nfs-client       

The persistent volume nfs-volume is ready to be mounted inside any pod declaring in a deployment file in this way:

root@microk8s01:#cat <<EOF | kubectl apply -f -
apiVersion: apps/v1
kind: Deployment
name: nginx
app: nginx
replicas: 1 # tells deployment to run 2 pods matching the template
app: nginx-nfs
- name: nginx
image: nginx
- containerPort: 80
- mountPath: "/var/log/nginx/"
name: testvolume
restartPolicy: Always
- name: testvolume
claimName: nfs-volume
root@microk8s01:#kubectl get pods NAME READY STATUS RESTARTS AGE nginx-6bd6c9685f-fqtgq 1/1 Running 0 6s root@microk8s01:#
kubectl exec -it nginx-6bd6c9685f-fqtgq -- /bin/bash root@nginx-6bd6c9685f-fqtgq:/#
df -k|grep data 29324176 10976972 16834552 40% /etc/hosts 343811072 217825280 108497920 67% /var/log/nginx

We have a cluster with all important services installed and configured, necessary for production management, like ingress, dashboard and storage. We should also add system monitoring service like a prometheus/grafana stack. This can be added by the ‘microk8s enable prometheus’, but in my case this didn’t work correctly, so I decided to install, but it’s out scope of this article, the prometheus operator directly from https://grafana.com/docs/grafana-cloud/quickstart/prometheus_operator/

The last piece missing, before installing it in Production environment, is its knowledge for having more control of any its part,

Inside the microk8s

The kubernetes cluster is easy to configure, but, in order to manage it in an production environment, is very important to understand how the infrastructure is deployed and configured.

For this scope, as a starting point to deepen the knowledge of the infrastructure, we can take a look at the output of ‘snap info microk8s’ command, where all the services up&running in the cluster are showed

root@microk8s01:/etc/haproxy# snap info microk8s |grep daemon
microk8s.daemon-apiserver: simple, enabled, inactive
microk8s.daemon-apiserver-kicker: simple, enabled, active
microk8s.daemon-cluster-agent: simple, enabled, active
microk8s.daemon-containerd: simple, enabled, active
microk8s.daemon-control-plane-kicker: simple, enabled, inactive
microk8s.daemon-controller-manager: simple, enabled, inactive
microk8s.daemon-etcd: simple, enabled, inactive
microk8s.daemon-flanneld: simple, enabled, inactive
microk8s.daemon-kubelet: simple, enabled, inactive
microk8s.daemon-kubelite: simple, enabled, active
microk8s.daemon-proxy: simple, enabled, inactive
microk8s.daemon-scheduler: simple, enabled, inactive

I was surprised to see controller, scheduler, etcd and proxy services inactives, and my first reaction was: how does the cluster work in this way?

Deepening the situation and reading the microk8s documentation (https://microk8s.io/), I found that, starting from microk8s 1.20 version, all the kubernetes functionalities – apiserver, controller, scheduler, kube-proxy and db- are stored in the kubelite process. Infact the apiserver port, 16443,, and the kubelet port, 10250, are managed by kubelite process.

root@microk8s01:/etc/# kubectl cluster-info
Kubernetes control plane is running at
CoreDNS is running at
Metrics-server is running at
To further debug and diagnose cluster problems, use 'kubectl cluster-info dump'.
root@microk8s01:/etc/# ss -lp |grep 16443
tcp LISTEN 0 4096 *:16443 *:* users:(("kubelite",pid=122424,fd=32))
root@microk8s01:/etc/# ss -lp |grep 10250
tcp LISTEN 0 4096 *:10250 *:* users:(("kubelite",pid=122424,fd=32))
root@microk8s01:/etc/haproxy# ps -afe |grep kubelite
root 122424 1 6 10:26 ? 00:02:23 /snap/microk8s/2210/kubelite --scheduler-args-file=/var/snap/microk8s/2210/args/kube-scheduler --controller-manager-args-file=/var/snap/microk8s/2210/args/kube-controller-manager --proxy-args-file=/var/snap/microk8s/2210/args/kube-proxy --kubelet-args-file=/var/snap/microk8s/2210/args/kubelet --apiserver-args-file=/var/snap/microk8s/2210/args/kube-apiserver --kubeconfig-file=/var/snap/microk8s/2210/credentials/client.config --start-control-plane=true
root 153903 1583 0 11:05 pts/0 00:00:00 grep --color=auto kubelite

The kubelite has, as input parameters, the configuration files used by apiserver, controller, scheduler, kube-proxy and apiserver. The arguments are stored in these files:

  1. Scheduler args: /var/snap/microk8s/2210/args/kube-schedule.
  2. Apiserver args: /var/snap/microk8s/2210/args/kube-apiserver.
  3. Controller args: /var/snap/microk8s/2210/args/kube-controller-manager.
  4. Kubelet args: /var/snap/microk8s/2210/args/kubelet.
  5. Kube-proxy args: /var/snap/microk8s/2210/args/kube-proxy.

Taking at look, for example, at the apiserver configuration we can see the common apiserver parameters.

root@microk8s01:~/# cat /var/snap/microk8s/2210/args/kube-apiserver

The limit of this cluster is that some kubernetes apiserver parameters are not accepted. Infact, I tried to added oidc parameters – well explained in this ask ubuntu https://askubuntu.com/questions/1243695/microk8s-oidc-with-keycloak/1339374#1339374 – and it didn’t work. This doesn’t permit to install an oidc provider, like dex ( https://github.com/dexidp/dex ) with a oidc proxy, like gangway (https://github.com/heptiolabs/gangway), that offer together a GUI, configured by known backend protocols like ldap, for username and password authentication.

The kubelite is running in any node of the cluster and it means that every node of the cluster is also a controller that is not possible to separate it from worker nodes. For restarting kubelite, with all the controllers inside it, it’s possible to use the systemd command:

root@microk8s01:#systemctl restart snap.microk8s.daemon-kubelite.service

As container runtime, running in each node of the cluster so that Pods can run, is used containerd that can be restarted in this way:

root@microk8s01:#systemctl restart snap.microk8s.daemon-containerd.service

Instead of docker, if it’s necessary to troubleshoot container issue, the command ctr, that is installed in each node inside the directory /snap/microk8s/2210/bin/ctr, could be used or by, in a simple way, with ctr natively invoked by microk8s:

root@microk8s01:#kubectl describe pod nginx-6bd6c9685f-fqtgq  |grep Conta
    Container ID:   containerd://cc6b48c5ebbe3914b9e918f6842e0ed356a10bcad684b1f679e8e34ac75b8101
  ContainersReady   True /snap/microk8s/2210/bin/ctr -namespace k8s.io -address 
root@microk8s01:#/snap/microk8s/2210/bin/ctr -namespace k8s.io -address /var/snap/microk8s/common/run/containerd.sock containers ls |grep cc6b48c5ebbe3914b9e918f6842e0ed356a10bcad684b1f679e8e34ac75b8101
cc6b48c5ebbe3914b9e918f6842e0ed356a10bcad684b1f679e8e34ac75b8101 sha256:d1a364dc548d5357f0da3268c888e1971bbdb957ee3f028fe7194f1d61c6fdee io.containerd.runc.v1
root@microk8s01:#microk8s ctr containers ls|grep cc6b48c5ebbe3914b9e918f6842e0ed356a10bcad684b1f679e8e34ac75b8101
cc6b48c5ebbe3914b9e918f6842e0ed356a10bcad684b1f679e8e34ac75b8101 sha256:d1a364dc548d5357f0da3268c888e1971bbdb957ee3f028fe7194f1d61c6fdee io.containerd.runc.v1

As said before, the network plugin used is calico that is formed by a controller, that watches continually tha apiserver, and a daemon set, running in any node of the cluster that contains the daemons for distribute routing information to other nodes and configure routes to local node and the calico configuration expressed by crd resources:

root@microk8s01:# kubectl get pods -A |grep calico
kube-system calico-kube-controllers-f7868dd95-vnbtf 1/1 Running 0 14d
kube-system calico-node-bb4dw 1/1 Running 0 14d
kube-system calico-node-k8h4s 1/1 Running 0 14d
kube-system calico-node-6bd9m 1/1 Running 0 14d
root@microk8s01:# kubectl get daemonset -A |grep calico
kube-system calico-node 3 3 3 3 3 kubernetes.io/os=linux 14d
root@microk8s01:# kubectl get crd -A |grep calico
bgpconfigurations.crd.projectcalico.org 2021-05-19T14:19:41Z
bgppeers.crd.projectcalico.org 2021-05-19T14:19:41Z
blockaffinities.crd.projectcalico.org 2021-05-19T14:19:41Z
clusterinformations.crd.projectcalico.org 2021-05-19T14:19:41Z
felixconfigurations.crd.projectcalico.org 2021-05-19T14:19:41Z
globalnetworkpolicies.crd.projectcalico.org 2021-05-19T14:19:41Z
globalnetworksets.crd.projectcalico.org 2021-05-19T14:19:41Z
hostendpoints.crd.projectcalico.org 2021-05-19T14:19:41Z
ipamblocks.crd.projectcalico.org 2021-05-19T14:19:41Z
ipamconfigs.crd.projectcalico.org 2021-05-19T14:19:41Z
ipamhandles.crd.projectcalico.org 2021-05-19T14:19:41Z
ippools.crd.projectcalico.org 2021-05-19T14:19:41Z
networkpolicies.crd.projectcalico.org 2021-05-19T14:19:41Z
networksets.crd.projectcalico.org 2021-05-19T14:19:41Z

A metric server, that scraps metric from the 1025 port of kubelet, is running and configured and it permits to monitor in real time the cpu and memory usage of pods and nodes:

root@microk8s01:# kubectl get pods -A |grep metric
kube-system metrics-server-8bbfb4bdb-fpg64 1/1 Running 0 12d
root@microk8s01:~# kubectl top nodes
W0603 13:00:15.879802 214384 top_node.go:119] Using json format to get metrics. Next release will switch to protocol-buffers, switch early by passing --use-protocol-buffers flag
microk8s01 250m 6% 4302Mi 54%
microk8s02 188m 4% 3714Mi 47%
microk8s03 127m 3% 3410Mi 43%

As datastore, instead of etcd, is used dqlite. An alternate backing store, running inside the kubelite process. that also uses raft algoritm for voting the leader in the same approach provided by etcd. The minimum number of nodes for creating a cluster in high availability is three, and it explains why a microk8s cluster must be formed by a minimum of three nodes. More information about dqlite utilities, to use for example backup and restore, are explained In https://microk8s.io/docs.


This microk8s kubernetes solution is easy to install and very useful for testing, demo and development environment. Canonical has released it for production, and in this context, it’s better to be awareness about the following features that could become limits to the scalability of the solution:

  1. Every node of the cluster is a controller node where the kubelite process, that contains all the controller part of the cluster, is running. This could have impact to the worker pods; of course, there is no way separate controllers from workers node.
  2. The api server and the controllers could not accept, preventing to custom the cluster according to your needs, all the kubernetes standard arguments. 

Despite all this, the solution is light, flexible and, after enabling all the services necessary in an production environment, I think it is really worth trying it in Production.