Kubernetes install with kubeadm

Network Example

10.10.11.20 kubemaster kubemaster.myhypervisor.ca
10.10.11.30 kube1 kube1.myhypervisor.ca
10.10.11.36 kube2 kube2.myhypervisor.ca

Disable SELinux.

setenforce 0
sed -i --follow-symlinks 's/SELINUX=enforcing/SELINUX=disabled/g' /etc/sysconfig/selinux

Enable the br_netfilter module for cluster communication.

modprobe br_netfilter
echo '1' > /proc/sys/net/bridge/bridge-nf-call-iptables

Disable swap to prevent memory allocation issues.

swapoff -a
vim /etc/fstab
#Remove swap from fstab

Setup NTP

yum install -y ntp
systemctl enable ntpd
systemctl start ntpd

Install Docker CE.

Install the Docker prerequisites.

yum install -y yum-utils device-mapper-persistent-data lvm2

Add the Docker repo and install Docker.

yum-config-manager --add-repo https://download.docker.com/linux/centos/docker-ce.repo
yum install -y docker-ce

Add the Kubernetes repo.

cat << EOF | tee /etc/yum.repos.d/kubernetes.repo
[kubernetes]
name=Kubernetes
baseurl=https://packages.cloud.google.com/yum/repos/kubernetes-el7-x86_64
enabled=1
gpgcheck=0
repo_gpgcheck=0
gpgkey=https://packages.cloud.google.com/yum/doc/yum-key.gpg
       https://packages.cloud.google.com/yum/doc/rpm-package-key.gpg
EOF

Install Kubernetes.

yum install -y kubelet kubeadm kubectl

Reboot.

Enable and start Docker and Kubernetes.

systemctl enable docker
systemctl enable kubelet
systemctl start docker
systemctl start kubelet

Check the group Docker is running in.

docker info | grep -i cgroup

*Note: Complete the following section on the MASTER ONLY!

Initialize the cluster using the IP range for Flannel.

kubeadm init --pod-network-cidr=10.244.0.0/16

Copy the kubeadmin join output.

Create standard user

useradd kubeuser
usermod -aG wheel kubeuser
passwd kubeuser
su kubeuser
mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config

Deploy Flannel.

kubectl apply -f https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml

Check the cluster state.

kubectl get pods --all-namespaces

Note: Complete the following steps on the NODES ONLY!

Run the join command that you copied earlier, then check your nodes from the master.

kubectl get nodes

Create a service account (for kube dashboard

Copy the token in a safe location, you will be able to use that token for services such as the k8s dashboard

Creating a admin / service account user called k

kubectl create serviceaccount k8sadmin -n kube-system

Give the user admin privileges

kubectl create clusterrolebinding k8sadmin --clusterrole=cluster-admin --serviceaccount=kube-system:k8sadmin

Get the token

kubectl -n kube-system describe secret $(sudo kubectl -n kube-system get secret | (grep k8sadmin || echo "$_") | awk '{print $1}') | grep token: | awk '{print $2}'

Installing MetalLB

kubectl apply -f https://raw.githubusercontent.com/google/metallb/v0.8.3/manifests/metallb.yaml

Create a file called metallb.yml

apiVersion: v1
kind: ConfigMap
metadata:
  namespace: metallb-system
  name: config
data:
  config: |
    address-pools:
    - name: default
      protocol: layer2
      addresses:
      - 10.10.11.150-10.10.11.160

kubectl create -f metallb.yml

Installing Dashboard

kubectl apply -f https://raw.githubusercontent.com/kubernetes/dashboard/v2.0.0-beta8/aio/deploy/recommended.yaml

Create a file called kube-dashboard-service.yml

apiVersion: v1
kind: Service
metadata:
  labels:
    app.kubernetes.io/name: load-balancer-dashboard
  name: dashboard-service
  namespace: kubernetes-dashboard
spec:
  ports:
    - port: 8080
      protocol: TCP
      targetPort: 8443
  selector:
    k8s-app: kubernetes-dashboard
  type: LoadBalancer

kubectl create -f kube-dashboard-service.yml
kubectl get all -A

With the command "kubectl get all -A" you will be able to find the external IP provided by metallb, only https is supported, use the token from the previous step to login to the dashboard.

(Heketi Install guide here: https://wiki.myhypervisor.ca/books/linux/page/glusterfs-using-ubuntu1604-c74)

Add Heketi for Dynamic Volumes

Create a file called heketi-storage.yml

apiVersion: v1
kind: Secret
metadata:
  name: heketi-secret
  namespace: default
type: "kubernetes.io/glusterfs"
data:
  # base64 encoded password. E.g.: echo -n "password" | base64
  key: cGFzc3dvcmQ= 
---
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: gluster-vol-default
provisioner: kubernetes.io/glusterfs
parameters:
  resturl: "http://10.10.11.161:8080"
  restuser: "admin"
  secretNamespace: "default"
  secretName: "heketi-secret"
  clusterid: "eba4f23d2eb41f894590cbe3ee05e51e"
allowVolumeExpansion: true
---
apiVersion: v1
kind: Endpoints
metadata:
  name: glusterfs-cluster
subsets:
- addresses:
  - ip: 10.10.11.200
  ports:
  - port: 5000
- addresses:
  - ip: 10.10.11.201
  ports:
  - port: 5000
---
apiVersion: v1
kind: Service
metadata:
  name: glusterfs-cluster
spec:
  ports:
  - port: 5000

Install Glusterfs-Client, The version needs to be the same as the Glusterfs Server.

apt-get install software-properties-common
add-apt-repository ppa:gluster/glusterfs-7
apt install glusterfs-client

Add a PVC to the storage (Example at the bottom is for a pihole deployment)

---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
 name: pihole-dnsmasq-volume
 annotations:
   volume.beta.kubernetes.io/storage-class: gluster-vol-default
spec:
 accessModes:
  - ReadWriteMany
 resources:
   requests:
     storage: 1Gi
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
 name: pihole-etc-volume
 annotations:
   volume.beta.kubernetes.io/storage-class: gluster-vol-default
spec:
 accessModes:
  - ReadWriteMany
 resources:
   requests:
     storage: 1Gi

Deploy container

Deploy your application (example below is pihole)

---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: pihole
  labels:
    app: pihole
spec:
  replicas: 1 
  selector:
    matchLabels:
      app: pihole
  template:
    metadata:
      labels:
        app: pihole
        name: pihole
    spec:
      containers:
      - name: pihole
        image: pihole/pihole:latest
        imagePullPolicy: Always
        env:
        - name: TZ
          value: "America/New_York"
        - name: WEBPASSWORD
          value: "secret"
        volumeMounts:
        - name: pihole-etc-volume
          mountPath: "/etc/pihole"
        - name: pihole-dnsmasq-volume
          mountPath: "/etc/dnsmasq.d"
      volumes:
      - name: pihole-etc-volume
        persistentVolumeClaim:
          claimName: pihole-etc-volume
      - name: pihole-dnsmasq-volume
        persistentVolumeClaim:
          claimName: pihole-dnsmasq-volume
---
apiVersion: v1
kind: Service
metadata:
  name: pihole-udp
spec:
  type: LoadBalancer
  ports:
    - port: 53
      targetPort: 53
      protocol: UDP
      name: dns-udp
  selector:
    app: pihole
---
apiVersion: v1
kind: Service
metadata:
  name: pihole-tcp
spec:
  type: LoadBalancer
  ports:
    - port: 53
      targetPort: 53
      protocol: TCP
      name: dns-tcp
    - protocol: TCP
      name: web
      port: 80
  selector:
    app: pihole

Install Helm Chart

curl https://raw.githubusercontent.com/helm/helm/master/scripts/get-helm-3 > get_helm.sh
chmod 700 get_helm.sh
./get_helm.sh

helm repo add stable https://kubernetes-charts.storage.googleapis.com/
helm repo update

traefik

apiVersion: v1
kind: ServiceAccount
metadata:
  namespace: default
  name: traefik-ingress-controller
---
kind: Deployment
apiVersion: apps/v1
metadata:
  namespace: default
  name: traefik
  labels:
    app: traefik
spec:
  replicas: 1
  selector:
    matchLabels:
      app: traefik
  template:
    metadata:
      labels:
        app: traefik
    spec:
      serviceAccountName: traefik-ingress-controller
      containers:
        - name: traefik
          image: traefik:v2.0
          args:
            - --api.insecure
            - --accesslog
            - --entrypoints.web.Address=:80
            - --providers.kubernetescrd
          ports:
            - name: web
              containerPort: 80
            - name: admin
              containerPort: 8080
---
kind: Deployment
apiVersion: apps/v1
metadata:
  namespace: default
  name: whoami
  labels:
    app: whoami
spec:
  replicas: 2
  selector:
    matchLabels:
      app: whoami
  template:
    metadata:
      labels:
        app: whoami
    spec:
      containers:
        - name: whoami
          image: containous/whoami
          ports:
            - name: web
              containerPort: 80

apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:
  name: ingressroutes.traefik.containo.us

spec:
  group: traefik.containo.us
  version: v1alpha1
  names:
    kind: IngressRoute
    plural: ingressroutes
    singular: ingressroute
  scope: Namespaced

---
apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:
  name: ingressroutetcps.traefik.containo.us

spec:
  group: traefik.containo.us
  version: v1alpha1
  names:
    kind: IngressRouteTCP
    plural: ingressroutetcps
    singular: ingressroutetcp
  scope: Namespaced

---
apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:
  name: middlewares.traefik.containo.us

spec:
  group: traefik.containo.us
  version: v1alpha1
  names:
    kind: Middleware
    plural: middlewares
    singular: middleware
  scope: Namespaced

---
apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:
  name: tlsoptions.traefik.containo.us

spec:
  group: traefik.containo.us
  version: v1alpha1
  names:
    kind: TLSOption
    plural: tlsoptions
    singular: tlsoption
  scope: Namespaced

---
apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:
  name: traefikservices.traefik.containo.us

spec:
  group: traefik.containo.us
  version: v1alpha1
  names:
    kind: TraefikService
    plural: traefikservices
    singular: traefikservice
  scope: Namespaced

---
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1beta1
metadata:
  name: traefik-ingress-controller

rules:
  - apiGroups:
      - ""
    resources:
      - services
      - endpoints
      - secrets
    verbs:
      - get
      - list
      - watch
  - apiGroups:
      - extensions
    resources:
      - ingresses
    verbs:
      - get
      - list
      - watch
  - apiGroups:
      - extensions
    resources:
      - ingresses/status
    verbs:
      - update
  - apiGroups:
      - traefik.containo.us
    resources:
      - middlewares
    verbs:
      - get
      - list
      - watch
  - apiGroups:
      - traefik.containo.us
    resources:
      - ingressroutes
    verbs:
      - get
      - list
      - watch
  - apiGroups:
      - traefik.containo.us
    resources:
      - ingressroutetcps
    verbs:
      - get
      - list
      - watch
  - apiGroups:
      - traefik.containo.us
    resources:
      - tlsoptions
    verbs:
      - get
      - list
      - watch
  - apiGroups:
      - traefik.containo.us
    resources:
      - traefikservices
    verbs:
      - get
      - list
      - watch

apiVersion: traefik.containo.us/v1alpha1
kind: IngressRoute
metadata:
  name: simpleingressroute
  namespace: default
spec:
  entryPoints:
    - web
  routes:
  - match: Host(`pihole.myhypervisor.ca`) && PathPrefix(`/`)
    kind: Rule
    services:
    - name: pihole-web
      port: 80