k8s 1.17.5高可用部署

k8s 1.17.5高可用部署

环境准备

K8S版本: 1.17.5

机器初始化(所有节点操作)

• 使用aliyun debian源
• 部署docker 18.09.9 (https://docs.docker.com/engine/install/debian/)
• 部署kubectl kubeadm kubelet

• 系统初始化, 关闭swap、使能ip foward等

  #!/bin/bash
  
  curl -fsSL https://mirrors.aliyun.com/kubernetes/apt/doc/apt-key.gpg | apt-key add -
  
  cat <<EOF >/etc/apt/sources.list.d/kubernetes.list
  deb http://mirrors.aliyun.com/kubernetes/apt/ kubernetes-xenial main
  EOF
  
  apt-get update
  apt-get install -y kubelet=1.17.5-00 kubeadm=1.17.5-00 kubectl=1.17.5-00
  modprobe br_netfilter
  echo '1' > /proc/sys/net/bridge/bridge-nf-call-iptables
  echo '1'  > /proc/sys/net/ipv4/ip_forward
  swapoff -a
  

master节点初始化

• kubeadm 初始化配置文件kubeadm-config.yaml如下:

  apiVersion: kubeadm.k8s.io/v1beta2
  kind: ClusterConfiguration
  imageRepository: registry.cn-hangzhou.aliyuncs.com/google_containers
  kubernetesVersion: v1.17.5    # k8s版本
  apiServer:
  certSANs:                   # apiserver的节点ip、vip
  - 192.168.0.2
  - 192.168.0.5
  - 192.168.0.6
  - 59.111.230.7
  controlPlaneEndpoint: "59.111.230.7:6443"
  networking:
  podSubnet: "10.244.0.0/16"
  

• kubeadm 初始化集群

kubeadm init –config=kubeadm-config.yaml|tee kubeadim-init.log

• 记录输出master和node加入命令

  You can now join any number of control-plane nodes by copying certificate authorities
  and service account keys on each node and then running the following as root:
  
  kubeadm join 59.111.230.7:6443 --token 876e54.j6so6f38paqd8wq2 \
  --discovery-token-ca-cert-hash sha256:262e95150dc375ec0992f7cd8297dc6e7019af14175039eabbd3d7e631ec594e \
  --control-plane
  
  Then you can join any number of worker nodes by running the following on each as root:
  
  kubeadm join 59.111.230.7:6443 --token 876e54.j6so6f38paqd8wq2 \
  --discovery-token-ca-cert-hash sha256:262e95150dc375ec0992f7cd8297dc6e7019af14175039eabbd3d7e631ec594e
  

网络安装(选flannel安装)

• flannel

  kubectl apply -f https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml
  

• calico

  kubectl create -f https://docs.projectcalico.org/manifests/tigera-operator.yaml
  kubectl create -f https://docs.projectcalico.org/manifests/custom-resources.yaml
  
  参考 https://docs.projectcalico.org/getting-started/kubernetes/quickstart
  

master、node节点加入

• master节点加入需先拷贝证书

  master_hosts
  192.168.0.3
  192.168.0.4
  ---
  pscp -p 20 -h master_hosts -O "StrictHostKeyChecking=no" /etc/kubernetes/pki/ca.crt /etc/kubernetes/pki/ca.crt
  pscp -p 20 -h master_hosts -O "StrictHostKeyChecking=no" /etc/kubernetes/pki/ca.key /etc/kubernetes/pki/ca.key
  pscp -p 20 -h master_hosts -O "StrictHostKeyChecking=no" /etc/kubernetes/pki/sa.key /etc/kubernetes/pki/sa.key
  pscp -p 20 -h master_hosts -O "StrictHostKeyChecking=no" /etc/kubernetes/pki/sa.pub /etc/kubernetes/pki/sa.pub
  pscp -p 20 -h master_hosts -O "StrictHostKeyChecking=no" /etc/kubernetes/pki/front-proxy-ca.crt /etc/kubernetes/pki/front-proxy-ca.crt
  pscp -p 20 -h master_hosts -O "StrictHostKeyChecking=no" /etc/kubernetes/pki/front-proxy-ca.key /etc/kubernetes/pki/front-proxy-ca.key
  pscp -p 20 -h master_hosts -O "StrictHostKeyChecking=no" /etc/kubernetes/pki/etcd/ca.crt /etc/kubernetes/pki/etcd/ca.crt
  pscp -p 20 -h master_hosts -O "StrictHostKeyChecking=no" /etc/kubernetes/pki/etcd/ca.key /etc/kubernetes/pki/etcd/ca.key
  
  
  kubeadm join 59.111.230.7:6443 --token 876e54.j6so6f38paqd8wq2 \
  --discovery-token-ca-cert-hash sha256:262e95150dc375ec0992f7cd8297dc6e7019af14175039eabbd3d7e631ec594e \
  --control-plane    
  

• node节点可直接加入

  kubeadm join 59.111.230.7:6443 --token 876e54.j6so6f38paqd8wq2 \
  --discovery-token-ca-cert-hash sha256:262e95150dc375ec0992f7cd8297dc6e7019af14175039eabbd3d7e631ec594e
  

问题记录

• DNS域名解析失败(k8s+flannel DNS解析失败)

  hostNetwork: true
  dnsPolicy: ClusterFirstWithHostNet
  

原因是: DNS请求的UDP报文checksum error被丢弃导致的

深层原因: 内核一个bug https://github.com/torvalds/linux/commit/ea64d8d6c675c0bb712689b13810301de9d8f77a

解决方案1

关闭tx checksum offload

ethtool -K flannel.1 tx-checksum-ip-generic off

解决方案2:

更新kube-proxy版本可以规避这个问题: kube-proxy-amd64:v1.17.9

新的规则：

-A POSTROUTING -m comment --comment "kubernetes postrouting rules" -j KUBE-POSTROUTING
-A KUBE-POSTROUTING -m mark ! --mark 0x4000/0x4000 -j RETURN
-A KUBE-POSTROUTING -j MARK --set-xmark 0x4000/0x0
-A KUBE-POSTROUTING -m comment --comment "kubernetes service traffic requiring SNAT" -j MASQUERADE