k8s 1.17.5高可用部署

k8s 1.17.5高可用部署

环境准备

K8S版本: 1.17.5

机器初始化(所有节点操作)

• 使用aliyun debian源
• 部署docker 18.09.9 (https://docs.docker.com/engine/install/debian/)
• 部署kubectl kubeadm kubelet
• 系统初始化, 关闭swap、使能ip foward等

#!/bin/bash

curl -fsSL https://mirrors.aliyun.com/kubernetes/apt/doc/apt-key.gpg | apt-key add -

cat <<EOF >/etc/apt/sources.list.d/kubernetes.list
deb http://mirrors.aliyun.com/kubernetes/apt/ kubernetes-xenial main
EOF

apt-get update
apt-get install -y kubelet=1.17.5-00 kubeadm=1.17.5-00 kubectl=1.17.5-00
modprobe br_netfilter
echo '1' > /proc/sys/net/bridge/bridge-nf-call-iptables
echo '1'  > /proc/sys/net/ipv4/ip_forward
swapoff -a

master节点初始化

• kubeadm 初始化配置文件kubeadm-config.yaml如下:

apiVersion: kubeadm.k8s.io/v1beta2
kind: ClusterConfiguration
imageRepository: registry.cn-hangzhou.aliyuncs.com/google_containers
kubernetesVersion: v1.17.5    # k8s版本
apiServer:
  certSANs:                   # apiserver的节点ip、vip
  - 192.168.0.2
  - 192.168.0.5
  - 192.168.0.6
  - 59.111.230.7
controlPlaneEndpoint: "59.111.230.7:6443"
networking:
  podSubnet: "10.244.0.0/16"

• kubeadm 初始化集群

kubeadm init –config=kubeadm-config.yaml|tee kubeadim-init.log

• 记录输出master和node加入命令

You can now join any number of control-plane nodes by copying certificate authorities
and service account keys on each node and then running the following as root:

  kubeadm join 59.111.230.7:6443 --token 876e54.j6so6f38paqd8wq2 \
    --discovery-token-ca-cert-hash sha256:262e95150dc375ec0992f7cd8297dc6e7019af14175039eabbd3d7e631ec594e \
    --control-plane

Then you can join any number of worker nodes by running the following on each as root:

kubeadm join 59.111.230.7:6443 --token 876e54.j6so6f38paqd8wq2 \
    --discovery-token-ca-cert-hash sha256:262e95150dc375ec0992f7cd8297dc6e7019af14175039eabbd3d7e631ec594e

网络安装(选flannel安装)

• flannel

kubectl apply -f https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml

• calico

kubectl create -f https://docs.projectcalico.org/manifests/tigera-operator.yaml
kubectl create -f https://docs.projectcalico.org/manifests/custom-resources.yaml

参考 https://docs.projectcalico.org/getting-started/kubernetes/quickstart

master、node节点加入

• master节点加入需先拷贝证书

master_hosts
192.168.0.3
192.168.0.4
---
pscp -p 20 -h master_hosts -O "StrictHostKeyChecking=no" /etc/kubernetes/pki/ca.crt /etc/kubernetes/pki/ca.crt
pscp -p 20 -h master_hosts -O "StrictHostKeyChecking=no" /etc/kubernetes/pki/ca.key /etc/kubernetes/pki/ca.key
pscp -p 20 -h master_hosts -O "StrictHostKeyChecking=no" /etc/kubernetes/pki/sa.key /etc/kubernetes/pki/sa.key
pscp -p 20 -h master_hosts -O "StrictHostKeyChecking=no" /etc/kubernetes/pki/sa.pub /etc/kubernetes/pki/sa.pub
pscp -p 20 -h master_hosts -O "StrictHostKeyChecking=no" /etc/kubernetes/pki/front-proxy-ca.crt /etc/kubernetes/pki/front-proxy-ca.crt
pscp -p 20 -h master_hosts -O "StrictHostKeyChecking=no" /etc/kubernetes/pki/front-proxy-ca.key /etc/kubernetes/pki/front-proxy-ca.key
pscp -p 20 -h master_hosts -O "StrictHostKeyChecking=no" /etc/kubernetes/pki/etcd/ca.crt /etc/kubernetes/pki/etcd/ca.crt
pscp -p 20 -h master_hosts -O "StrictHostKeyChecking=no" /etc/kubernetes/pki/etcd/ca.key /etc/kubernetes/pki/etcd/ca.key


kubeadm join 59.111.230.7:6443 --token 876e54.j6so6f38paqd8wq2 \
    --discovery-token-ca-cert-hash sha256:262e95150dc375ec0992f7cd8297dc6e7019af14175039eabbd3d7e631ec594e \
    --control-plane    

• node节点可直接加入

kubeadm join 59.111.230.7:6443 --token 876e54.j6so6f38paqd8wq2 \
    --discovery-token-ca-cert-hash sha256:262e95150dc375ec0992f7cd8297dc6e7019af14175039eabbd3d7e631ec594e

问题记录

• DNS域名解析失败(k8s+flannel DNS解析失败)

hostNetwork: true
dnsPolicy: ClusterFirstWithHostNet

原因是: DNS请求的UDP报文checksum error被丢弃导致的

深层原因: 内核一个bug https://github.com/torvalds/linux/commit/ea64d8d6c675c0bb712689b13810301de9d8f77a

解决方案1

关闭tx checksum offload

ethtool -K flannel.1 tx-checksum-ip-generic off

解决方案2:

更新kube-proxy版本可以规避这个问题: kube-proxy-amd64:v1.17.9

新的规则：

-A POSTROUTING -m comment --comment "kubernetes postrouting rules" -j KUBE-POSTROUTING
-A KUBE-POSTROUTING -m mark ! --mark 0x4000/0x4000 -j RETURN
-A KUBE-POSTROUTING -j MARK --set-xmark 0x4000/0x0
-A KUBE-POSTROUTING -m comment --comment "kubernetes service traffic requiring SNAT" -j MASQUERADE