1. 环境准备与基础配置在开始构建高可用Kubernetes集群之前我们需要做好充分的环境准备。我建议使用3台及以上配置相同的CentOS 7/8或Ubuntu 20.04 LTS服务器每台至少4核CPU、8GB内存和50GB磁盘空间。在实际项目中我曾遇到过内存不足导致kubelet频繁崩溃的情况所以资源预留很关键。首先在所有节点上执行基础系统配置# 关闭防火墙 systemctl stop firewalld systemctl disable firewalld # 关闭SELinux setenforce 0 sed -i s/SELINUXenforcing/SELINUXdisabled/ /etc/selinux/config # 关闭swap swapoff -a sed -i / swap / s/^\(.*\)$/#\1/g /etc/fstab # 设置时区 timedatectl set-timezone Asia/Shanghai # 配置hosts cat /etc/hosts EOF 192.168.1.101 k8s-master1 192.168.1.102 k8s-master2 192.168.1.103 k8s-master3 192.168.1.201 k8s-node1 192.168.1.202 k8s-node2 EOF内核参数调优是很多教程会忽略的部分但这些配置对集群稳定性至关重要cat /etc/sysctl.d/k8s.conf EOF net.bridge.bridge-nf-call-ip6tables 1 net.bridge.bridge-nf-call-iptables 1 net.ipv4.ip_forward 1 vm.swappiness 0 vm.overcommit_memory 1 vm.panic_on_oom 0 fs.inotify.max_user_instances 8192 fs.inotify.max_user_watches 1048576 fs.file-max 52706963 fs.nr_open 52706963 net.ipv6.conf.all.disable_ipv6 1 net.netfilter.nf_conntrack_max 2310720 EOF sysctl -p /etc/sysctl.d/k8s.conf2. 证书体系设计与签发Kubernetes重度依赖TLS证书进行组件间通信加密手动部署时需要特别注意证书的合理规划。我采用Cloudflare的cfssl工具链来生成证书相比openssl更易用。首先创建CA证书在第一个Master节点操作mkdir -p /etc/kubernetes/pki cd /etc/kubernetes/pki cat ca-config.json EOF { signing: { default: { expiry: 87600h }, profiles: { kubernetes: { usages: [signing, key encipherment, server auth, client auth], expiry: 87600h } } } } EOF cat ca-csr.json EOF { CN: Kubernetes, key: { algo: rsa, size: 2048 }, names: [ { C: CN, ST: Shanghai, L: Shanghai, O: Kubernetes, OU: CA } ] } EOF cfssl gencert -initca ca-csr.json | cfssljson -bare ca接下来为API Server生成证书特别注意要包含所有可能的访问方式cat kube-apiserver-csr.json EOF { CN: kube-apiserver, key: { algo: rsa, size: 2048 }, names: [ { C: CN, ST: Shanghai, L: Shanghai, O: Kubernetes, OU: Kubernetes } ] } EOF cfssl gencert \ -caca.pem \ -ca-keyca-key.pem \ -configca-config.json \ -hostname10.96.0.1,192.168.1.101,192.168.1.102,192.168.1.103,127.0.0.1,kubernetes,kubernetes.default,kubernetes.default.svc,kubernetes.default.svc.cluster.local \ -profilekubernetes \ kube-apiserver-csr.json | cfssljson -bare kube-apiserver其他组件证书类似方式生成后需要将CA证书和生成的证书同步到所有节点for node in k8s-master2 k8s-master3 k8s-node1 k8s-node2; do rsync -avz /etc/kubernetes/pki/ca* $node:/etc/kubernetes/pki/ rsync -avz /etc/kubernetes/pki/kube-apiserver* $node:/etc/kubernetes/pki/ # 同步其他必要证书... done3. 核心组件部署与配置3.1 etcd集群部署etcd作为Kubernetes的后端存储必须确保高可用。我采用3节点集群部署# 下载etcd二进制包 ETCD_VERv3.5.0 wget https://github.com/etcd-io/etcd/releases/download/${ETCD_VER}/etcd-${ETCD_VER}-linux-amd64.tar.gz tar xvf etcd-${ETCD_VER}-linux-amd64.tar.gz mv etcd-${ETCD_VER}-linux-amd64/{etcd,etcdctl} /usr/local/bin/ # 创建systemd服务文件 cat /etc/systemd/system/etcd.service EOF [Unit] Descriptionetcd Documentationhttps://github.com/coreos/etcd [Service] Typenotify ExecStart/usr/local/bin/etcd \\ --namek8s-master1 \\ --data-dir/var/lib/etcd \\ --initial-advertise-peer-urlshttps://192.168.1.101:2380 \\ --listen-peer-urlshttps://192.168.1.101:2380 \\ --listen-client-urlshttps://192.168.1.101:2379,https://127.0.0.1:2379 \\ --advertise-client-urlshttps://192.168.1.101:2379 \\ --initial-cluster-tokenetcd-cluster \\ --initial-clusterk8s-master1https://192.168.1.101:2380,k8s-master2https://192.168.1.102:2380,k8s-master3https://192.168.1.103:2380 \\ --initial-cluster-statenew \\ --client-cert-auth \\ --trusted-ca-file/etc/kubernetes/pki/ca.pem \\ --cert-file/etc/kubernetes/pki/etcd.pem \\ --key-file/etc/kubernetes/pki/etcd-key.pem \\ --peer-client-cert-auth \\ --peer-trusted-ca-file/etc/kubernetes/pki/ca.pem \\ --peer-cert-file/etc/kubernetes/pki/etcd.pem \\ --peer-key-file/etc/kubernetes/pki/etcd-key.pem Restarton-failure RestartSec5 [Install] WantedBymulti-user.target EOF启动etcd后验证集群健康状态ETCDCTL_API3 etcdctl \ --endpointshttps://192.168.1.101:2379 \ --cacert/etc/kubernetes/pki/ca.pem \ --cert/etc/kubernetes/pki/etcd.pem \ --key/etc/kubernetes/pki/etcd-key.pem \ endpoint health3.2 控制平面组件部署Kubernetes控制平面包含kube-apiserver、kube-controller-manager和kube-scheduler三个核心组件。我采用二进制方式部署# 下载Kubernetes二进制文件 K8S_VERv1.25.0 wget https://dl.k8s.io/${K8S_VER}/kubernetes-server-linux-amd64.tar.gz tar xvf kubernetes-server-linux-amd64.tar.gz cd kubernetes/server/bin cp kube-apiserver kube-controller-manager kube-scheduler kubectl /usr/local/bin/ # 配置kube-apiserver cat /etc/systemd/system/kube-apiserver.service EOF [Unit] DescriptionKubernetes API Server Documentationhttps://kubernetes.io/docs/concepts/overview/ [Service] ExecStart/usr/local/bin/kube-apiserver \\ --advertise-address192.168.1.101 \\ --allow-privilegedtrue \\ --apiserver-count3 \\ --audit-log-maxage30 \\ --audit-log-maxbackup3 \\ --audit-log-maxsize100 \\ --audit-log-path/var/log/kubernetes/audit.log \\ --authorization-modeNode,RBAC \\ --bind-address0.0.0.0 \\ --client-ca-file/etc/kubernetes/pki/ca.pem \\ --enable-admission-pluginsNodeRestriction \\ --enable-bootstrap-token-authtrue \\ --etcd-cafile/etc/kubernetes/pki/ca.pem \\ --etcd-certfile/etc/kubernetes/pki/apiserver-etcd-client.pem \\ --etcd-keyfile/etc/kubernetes/pki/apiserver-etcd-client-key.pem \\ --etcd-servershttps://192.168.1.101:2379,https://192.168.1.102:2379,https://192.168.1.103:2379 \\ --kubelet-client-certificate/etc/kubernetes/pki/apiserver-kubelet-client.pem \\ --kubelet-client-key/etc/kubernetes/pki/apiserver-kubelet-client-key.pem \\ --service-account-key-file/etc/kubernetes/pki/sa.pub \\ --service-account-signing-key-file/etc/kubernetes/pki/sa.key \\ --service-account-issuerhttps://kubernetes.default.svc.cluster.local \\ --service-cluster-ip-range10.96.0.0/12 \\ --tls-cert-file/etc/kubernetes/pki/kube-apiserver.pem \\ --tls-private-key-file/etc/kubernetes/pki/kube-apiserver-key.pem \\ --requestheader-client-ca-file/etc/kubernetes/pki/front-proxy-ca.pem \\ --proxy-client-cert-file/etc/kubernetes/pki/front-proxy-client.pem \\ --proxy-client-key-file/etc/kubernetes/pki/front-proxy-client-key.pem \\ --requestheader-allowed-namesfront-proxy-client \\ --requestheader-extra-headers-prefixX-Remote-Extra- \\ --requestheader-group-headersX-Remote-Group \\ --requestheader-username-headersX-Remote-User Restarton-failure RestartSec5 [Install] WantedBymulti-user.target EOF其他控制平面组件配置类似启动后可以通过kubectl检查状态kubectl get componentstatuses4. 高可用与负载均衡实现生产环境必须确保控制平面的高可用性。我采用HAProxyKeepalived方案实现API Server的负载均衡# 安装HAProxy yum install -y haproxy # 配置HAProxy cat /etc/haproxy/haproxy.cfg EOF global log /dev/log local0 log /dev/log local1 notice chroot /var/lib/haproxy stats socket /run/haproxy/admin.sock mode 660 level admin stats timeout 30s user haproxy group haproxy daemon defaults log global mode http option httplog option dontlognull timeout connect 5000 timeout client 50000 timeout server 50000 frontend k8s-api bind *:6443 mode tcp default_backend k8s-api backend k8s-api mode tcp balance roundrobin server k8s-master1 192.168.1.101:6443 check server k8s-master2 192.168.1.102:6443 check server k8s-master3 192.168.1.103:6443 check EOF # 配置Keepalived cat /etc/keepalived/keepalived.conf EOF vrrp_script chk_haproxy { script killall -0 haproxy interval 2 weight 2 } vrrp_instance VI_1 { interface eth0 state MASTER virtual_router_id 51 priority 101 virtual_ipaddress { 192.168.1.100/24 } track_script { chk_haproxy } } EOF5. 工作节点与网络插件部署工作节点需要部署kubelet和kube-proxy并安装容器运行时这里以Docker为例# 安装Docker yum install -y yum-utils yum-config-manager --add-repo https://download.docker.com/linux/centos/docker-ce.repo yum install -y docker-ce docker-ce-cli containerd.io systemctl enable --now docker # 配置kubelet cat /etc/systemd/system/kubelet.service EOF [Unit] DescriptionKubernetes Kubelet Documentationhttps://kubernetes.io/docs/concepts/overview/ [Service] ExecStart/usr/local/bin/kubelet \\ --bootstrap-kubeconfig/etc/kubernetes/bootstrap-kubelet.conf \\ --kubeconfig/etc/kubernetes/kubelet.conf \\ --config/var/lib/kubelet/config.yaml \\ --container-runtimeremote \\ --container-runtime-endpointunix:///run/containerd/containerd.sock \\ --pod-infra-container-imageregistry.aliyuncs.com/google_containers/pause:3.7 \\ --node-labelsnode-role.kubernetes.io/node Restarton-failure RestartSec5 [Install] WantedBymulti-user.target EOF网络插件选择Calico它支持网络策略和IP地址管理kubectl apply -f https://docs.projectcalico.org/manifests/calico.yaml6. 集群验证与排错部署完成后需要进行全面验证# 检查节点状态 kubectl get nodes -o wide # 检查核心组件状态 kubectl get pods -n kube-system # 部署测试应用 kubectl create deployment nginx --imagenginx kubectl expose deployment nginx --port80 --typeNodePort # 测试网络连通性 kubectl run busybox --rm -it --imagebusybox -- sh wget -qO- nginx常见问题排查技巧kubelet无法启动检查证书路径和权限Pod网络不通检查Calico日志和路由表API Server不可用检查etcd集群状态和HAProxy日志证书过期使用cfssl重新生成并分发证书
