k8s v1.28.15部署（kubeadm方式）

k8s部署（kubeadm方式）

部署环境及版本

系统版本：CentOS Linux release 7.9.2009
k8s版本：v1.28.15
docker版本：26.1.4
containerd版本：1.6.33
calico版本：v3.25.0

准备

主机ip	主机名	角色	配置
192.168.1.131	vm131	master	2核4G，50G
192.168.1.132	vm132	node1	2核4G，50G
192.168.1.133	vm133	node2	2核4G，50G

1、修改主机名

#master
hostnamectl set-hostname vm131
#node1
hostnamectl set-hostname vm132
#node2
hostnamectl set-hostname vm133#修改hosts文件
vim /etc/hosts
192.168.1.131 vm131
192.168.1.132 vm132
192.168.1.133 vm133

2、关闭自带防火墙

# 关闭SELinux
setenforce 0 
sed -i 's/SELINUX=enforcing/SELINUX=disabled/g' /etc/selinux/config# 关闭Firewalld并禁止自启动
systemctl stop firewalld
systemctl disable firewalld# 设置iptables规则
iptables -F && iptables -X && iptables -F -t nat && iptables -X -t nat && iptables -P FORWARD ACCEPT# 设置iptables，解决iptables而导致流量无法正确路由的问题
cat <<EOF >  /etc/sysctl.d/k8s.conf
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
net.ipv4.ip_nonlocal_bind = 1
net.ipv4.ip_forward = 1
vm.swappiness = 0
vm.overcommit_memory = 1
EOF
sysctl --system# 关闭swap
swapoff -a && free –h
sed -i 's@/dev/mapper/centos-swap@#/dev/mapper/centos-swap@g' /etc/fstab

3、时间同步

yum -y install ntp
systemctl enable ntpd
systemctl start ntpd
timedatectl set-timezone Asia/Shanghai
ntpdate -u time.nist.gov
date#如果遇到yum源的问题，可以通过以下方式解决
curl -o /etc/yum.repos.d/CentOS-Base.repo http://mirrors.aliyun.com/repo/Centos-7.repo
yum clean all #清除缓存
yum makecache #生成缓存

一、安装docker

#安装编译器环境
yum -y install gcc gcc-c++
#卸载之前下载的docker（防止和之前安装过的docker出现冲突）yum remove docker \docker-client \docker-client-latest \docker-common \docker-latest \docker-latest-logrotate \docker-logrotate \docker-engine
#安装yum-utils
yum install -y yum-utils
#将docker-ce.repo添加到/etc/yum.repos.d/
yum-config-manager --add-repo http://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo
#提升yum安装的速度
yum makecache fast
#安装docker
yum install docker-ce docker-ce-cli containerd.io -y
#启动docker
systemctl start docker
#开机自启动
systemctl enable docker

二、安装containerd

yum install containerd -y#配置/etc/containerd/config.toml 
#如果你不打算使用跟踪功能，可以在 containerd 配置文件中禁用该插件。vim /etc/containerd/config.toml
#disabled_plugins = ["cri"]
[plugins."io.containerd.grpc.v1.cri"]
disable = false#配置完成后，重启 containerd：
systemctl restart containerd
#如果你需要启用 tracing，可以配置跟踪端点。在 containerd 的配置文件中，提供正确的 tracing 端点配置：
#[plugins."io.containerd.tracing.processor.v1.otlp"]
#  endpoint = "your-tracing-endpoint:port"
#根据你所使用的 tracing 服务（如 OpenTelemetry Collector）来提供适当的端点信息。
#配置完成后，重启 containerd：
#systemctl restart containerd

三、Master节点安装kubeadm

1、安装kubelet 和kubeadm以及kubectl

yum install -y kubelet kubeadm kubectl --disableexcludes=kubernetes
systemctl enable --now kubelet

2、下载所需要的镜像

#查看需要下载的镜像
kubeadm config images list
#查看结果
I0306 11:27:12.886721   21939 version.go:256] remote version is much newer: v1.32.2; falling back to: stable-1.28
registry.k8s.io/kube-apiserver:v1.28.15
registry.k8s.io/kube-controller-manager:v1.28.15
registry.k8s.io/kube-scheduler:v1.28.15
registry.k8s.io/kube-proxy:v1.28.15
registry.k8s.io/pause:3.9
registry.k8s.io/etcd:3.5.9-0
registry.k8s.io/coredns/coredns:v1.10.1
#下载镜像（注意：namespace是k8s.io）
ctr -n k8s.io images pull registry.cn-hangzhou.aliyuncs.com/google_containers/kube-apiserver:v1.28.15
ctr -n k8s.io images pull registry.cn-hangzhou.aliyuncs.com/google_containers/kube-controller-manager:v1.28.15
ctr -n k8s.io images pull registry.cn-hangzhou.aliyuncs.com/google_containers/kube-scheduler:v1.28.15
ctr -n k8s.io images pull registry.cn-hangzhou.aliyuncs.com/google_containers/kube-proxy:v1.28.15
ctr -n k8s.io images pull registry.cn-hangzhou.aliyuncs.com/google_containers/pause:3.9
ctr -n k8s.io images pull registry.cn-hangzhou.aliyuncs.com/google_containers/etcd:3.5.9-0
ctr -n k8s.io images pull registry.cn-hangzhou.aliyuncs.com/google_containers/coredns:v1.10.1
ctr -n k8s.io images pull registry.cn-hangzhou.aliyuncs.com/google_containers/pause:3.6
#重新打标签
ctr -n k8s.io images tag registry.cn-hangzhou.aliyuncs.com/google_containers/kube-apiserver:v1.28.15 registry.k8s.io/kube-apiserver:v1.28.15
ctr -n k8s.io images tag registry.cn-hangzhou.aliyuncs.com/google_containers/kube-controller-manager:v1.28.15 registry.k8s.io/kube-controller-manager:v1.28.15
ctr -n k8s.io images tag registry.cn-hangzhou.aliyuncs.com/google_containers/kube-scheduler:v1.28.15 registry.k8s.io/kube-scheduler:v1.28.15
ctr -n k8s.io images tag registry.cn-hangzhou.aliyuncs.com/google_containers/kube-proxy:v1.28.15 registry.k8s.io/kube-proxy:v1.28.15
ctr -n k8s.io images tag registry.cn-hangzhou.aliyuncs.com/google_containers/pause:3.9 registry.k8s.io/pause:3.9
ctr -n k8s.io images tag registry.cn-hangzhou.aliyuncs.com/google_containers/pause:3.6 registry.k8s.io/pause:3.6
ctr -n k8s.io images tag registry.cn-hangzhou.aliyuncs.com/google_containers/etcd:3.5.9-0 registry.k8s.io/etcd:3.5.9-0
ctr -n k8s.io images tag registry.cn-hangzhou.aliyuncs.com/google_containers/coredns:v1.10.1 registry.k8s.io/coredns/coredns:v1.10.1
#删除aliyun镜像
for item in `ctr -n k8s.io images list | awk '$1 ~ /registry.cn-hangzhou.aliyuncs.com/ {print $1}'`;do ctr -n k8s.io images rm ${item};done

3、更改kubelet参数

vim /etc/sysconfig/kubelet#改为如下参数
KUBELET_EXTRA_ARGS=--cgroup-driver=systemd

4、kubeadm初始化

kubeadm init
#出现Your Kubernetes control-plane has initialized successfully!说明初始化成功
#根据提示执行如下命令
mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config

四、Node节点安装kubeadm

1、安装kubeadm kubelet

yum -y install kubeadm kubelet
systemctl enable kubelet.service

2、下载所需要的镜像

#查看需要下载的镜像
kubeadm config images list
#查看结果
I0306 11:27:12.886721   21939 version.go:256] remote version is much newer: v1.32.2; falling back to: stable-1.28
registry.k8s.io/kube-apiserver:v1.28.15
registry.k8s.io/kube-controller-manager:v1.28.15
registry.k8s.io/kube-scheduler:v1.28.15
registry.k8s.io/kube-proxy:v1.28.15
registry.k8s.io/pause:3.9
registry.k8s.io/etcd:3.5.9-0
registry.k8s.io/coredns/coredns:v1.10.1
#下载镜像（注意：namespace是k8s.io）
ctr -n k8s.io images pull registry.cn-hangzhou.aliyuncs.com/google_containers/kube-apiserver:v1.28.15
ctr -n k8s.io images pull registry.cn-hangzhou.aliyuncs.com/google_containers/kube-controller-manager:v1.28.15
ctr -n k8s.io images pull registry.cn-hangzhou.aliyuncs.com/google_containers/kube-scheduler:v1.28.15
ctr -n k8s.io images pull registry.cn-hangzhou.aliyuncs.com/google_containers/kube-proxy:v1.28.15
ctr -n k8s.io images pull registry.cn-hangzhou.aliyuncs.com/google_containers/pause:3.9
ctr -n k8s.io images pull registry.cn-hangzhou.aliyuncs.com/google_containers/etcd:3.5.9-0
ctr -n k8s.io images pull registry.cn-hangzhou.aliyuncs.com/google_containers/coredns:v1.10.1
ctr -n k8s.io images pull registry.cn-hangzhou.aliyuncs.com/google_containers/pause:3.6
#重新打标签
ctr -n k8s.io images tag registry.cn-hangzhou.aliyuncs.com/google_containers/kube-apiserver:v1.28.15 registry.k8s.io/kube-apiserver:v1.28.15
ctr -n k8s.io images tag registry.cn-hangzhou.aliyuncs.com/google_containers/kube-controller-manager:v1.28.15 registry.k8s.io/kube-controller-manager:v1.28.15
ctr -n k8s.io images tag registry.cn-hangzhou.aliyuncs.com/google_containers/kube-scheduler:v1.28.15 registry.k8s.io/kube-scheduler:v1.28.15
ctr -n k8s.io images tag registry.cn-hangzhou.aliyuncs.com/google_containers/kube-proxy:v1.28.15 registry.k8s.io/kube-proxy:v1.28.15
ctr -n k8s.io images tag registry.cn-hangzhou.aliyuncs.com/google_containers/pause:3.9 registry.k8s.io/pause:3.9
ctr -n k8s.io images tag registry.cn-hangzhou.aliyuncs.com/google_containers/pause:3.6 registry.k8s.io/pause:3.6
ctr -n k8s.io images tag registry.cn-hangzhou.aliyuncs.com/google_containers/etcd:3.5.9-0 registry.k8s.io/etcd:3.5.9-0
ctr -n k8s.io images tag registry.cn-hangzhou.aliyuncs.com/google_containers/coredns:v1.10.1 registry.k8s.io/coredns/coredns:v1.10.1
#删除aliyun镜像
for item in `ctr -n k8s.io images list | awk '$1 ~ /registry.cn-hangzhou.aliyuncs.com/ {print $1}'`;do ctr -n k8s.io images rm ${item};done

3、更改kubelet参数

vim /etc/sysconfig/kubelet#改为如下参数
KUBELET_EXTRA_ARGS=--cgroup-driver=systemd

4、加入Master

#token来自master节点执行kubeinit的结果
kubeadm join 192.168.1.131:6443 --token 51lpgy.tue7o5rnwi3h62ql \
> --discovery-token-ca-cert-hash sha256:200ad4e8f13649b3cd14db97cdaf842d97f4dbbca0cbec123c06b2a3c687ede1

五、安装网络插件

1、在Master节点拉取并修改calico.yml文件

#下载calico的yaml文件（Master节点操作）
yum install wget -y
mkdir -pv /data/yaml && cd /data/yaml
wget https://docs.tigera.io/archive/v3.25/manifests/calico.yaml
#修改calico的yaml文件
- name: CALICO_IPV4POOL_CIDRvalue: "10.244.0.0/16"
#在CLUSTER_TYPE那行，下面添加（interface根据自己的主机网卡名修改）
- name: IP_AUTODETECTION_METHODvalue: "interface=ens192"

2、下载所需要的镜像（Master和Node都要操作）

mkdir -pv /data/tar && cd /data/tar
#需要先用docker下载，用ctr测试下载不下来
docker pull calico/cni:v3.25.0
docker pull calico/node:v3.25.0
docker pull calico/kube-controllers:v3.25.0
#导出docker镜像
docker save -o cni.tar calico/cni:v3.25.0
docker save -o node.tar calico/node:v3.25.0
docker save -o kube-controllers.tar calico/kube-controllers:v3.25.0
#用ctr导入镜像（namespace是k8s.io）（前面两步可以在一台服务器上操作，最后这步都要操作）
ctr -n k8s.io images import cni.tar
ctr -n k8s.io images import node.tar
ctr -n k8s.io images import kube-controllers.tar
#验证是否导入成功
ctr -n k8s.io images list|grep calico

3、安装calico插件

#只在Master节点操作即可
cd /data/yaml
kubectl apply -f calico.yaml

[root@vm131 ~]# kubectl get pods -A
NAMESPACE     NAME                                       READY   STATUS    RESTARTS       AGE
kube-system   calico-kube-controllers-658d97c59c-fc46q   1/1     Running   0              46m
kube-system   calico-node-4cc9q                          1/1     Running   0              46m
kube-system   calico-node-l6ch2                          1/1     Running   0              46m
kube-system   calico-node-r27jj                          1/1     Running   0              46m
kube-system   coredns-5dd5756b68-84cww                   0/1     Running   0              4h
kube-system   coredns-5dd5756b68-nd9v8                   0/1     Running   0              4h
kube-system   etcd-vm131                                 1/1     Running   0              4h1m
kube-system   kube-apiserver-vm131                       1/1     Running   0              4h1m
kube-system   kube-controller-manager-vm131              1/1     Running   1 (3h9m ago)   4h1m
kube-system   kube-proxy-26qwm                           1/1     Running   0              3h44m
kube-system   kube-proxy-bj5xg                           1/1     Running   0              4h
kube-system   kube-proxy-wgfnz                           1/1     Running   0              3h43m
kube-system   kube-scheduler-vm131                       1/1     Running   0              4h1m#出现上面情况可以尝试，重启coreDNS
#plugin/kubernetes: Kubernetes API connection failure: Get "https://10.96.0.1:443/version": dial tcp 10.96.0.1:443: i/o timeout
kubectl rollout restart deployment coredns -n kube-system

六、验证

[root@vm131 ~]# kubectl get pods -A
NAMESPACE     NAME                                       READY   STATUS    RESTARTS        AGE
kube-system   calico-kube-controllers-658d97c59c-t678p   1/1     Running   0               5m26s
kube-system   calico-node-5zj2n                          1/1     Running   0               5m26s
kube-system   calico-node-hjdcm                          1/1     Running   0               5m26s
kube-system   calico-node-hrwh4                          1/1     Running   0               5m26s
kube-system   coredns-967d5bb69-sb9xx                    1/1     Running   0               31m
kube-system   coredns-967d5bb69-sngxg                    1/1     Running   0               31m
kube-system   etcd-vm131                                 1/1     Running   0               4h34m
kube-system   kube-apiserver-vm131                       1/1     Running   0               4h34m
kube-system   kube-controller-manager-vm131              1/1     Running   1 (3h43m ago)   4h34m
kube-system   kube-proxy-26qwm                           1/1     Running   0               4h17m
kube-system   kube-proxy-bj5xg                           1/1     Running   0               4h34m
kube-system   kube-proxy-wgfnz                           1/1     Running   0               4h16m
kube-system   kube-scheduler-vm131                       1/1     Running   0               4h34m
[root@vm131 ~]# kubectl get node
NAME    STATUS   ROLES           AGE     VERSION
vm131   Ready    control-plane   4h34m   v1.28.2
vm132   Ready    <none>          4h16m   v1.28.2
vm133   Ready    <none>          4h17m   v1.28.2

七、问题记录

crictl images 报错：
WARN[0000] image connect using default endpoints: [unix:///var/run/dockershim.sock unix:///run/containerd/containerd.sock unix:///run/crio/crio.sock unix:///var/run/cri-dockerd.sock]. As the default settings are now deprecated, you should set the endpoint instead. 
E0305 16:42:46.922252   22812 remote_image.go:119] "ListImages with filter from image service failed" err="rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing dial unix /var/run/dockershim.sock: connect: no such file or directory\"" filter="&ImageFilter{Image:&ImageSpec{Image:,Annotations:map[string]string{},},}"
FATA[0000] listing images: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing dial unix /var/run/dockershim.sock: connect: no such file or directory"
解决：
export CONTAINER_RUNTIME_ENDPOINT=unix:///run/containerd/containerd.sock
export IMAGE_SERVICE_ENDPOINT=unix:///run/containerd/containerd.sock
配置 crictl 默认连接 containerd
编辑 crictl 配置文件：
mkdir -p /etc/crictl
echo 'runtime-endpoint: unix:///run/containerd/containerd.sock' > /etc/crictl.yamlvim /etc/containerd/config.toml
#disabled_plugins = ["cri"]
[plugins."io.containerd.grpc.v1.cri"]
disable = false禁用跟踪插件： 如果你不打算使用跟踪功能，可以在 containerd 配置文件中禁用该插件。在 /etc/containerd/config.toml 文件中，查找并禁用跟踪插件：
[plugins."io.containerd.tracing.processor.v1.otlp"]enabled = false
然后重启 containerd：
sudo systemctl restart containerd
配置 tracing endpoint： 如果你需要启用 tracing，可以配置跟踪端点。在 containerd 的配置文件中，提供正确的 tracing 端点配置：[plugins."io.containerd.tracing.processor.v1.otlp"]endpoint = "your-tracing-endpoint:port"
根据你所使用的 tracing 服务（如 OpenTelemetry Collector）来提供适当的端点信息。配置完成后，重启 containerd：
sudo systemctl restart containerd
总结：
如果不使用跟踪功能，可以简单地忽略这个信息或禁用该插件。
如果需要启用跟踪功能，确保配置了正确的 tracing endpoint