二进制部署kubernetes高可用集群

一、单节点部署

1、集群节点规划（均是24位掩码）

负载均衡节点	Master节点	Node节点	Harbor私有仓库节点
nginx1=10.4.7.23	master1=10.4.7.11	node1=10.4.7.21	10.4.7.200
nginx2=10.4.7.24	master2=10.4.7.12	node2=10.4.7.22

2、基本环境准备（所有节点上操作）

# 关闭防火墙
systemctl stop firewalld
systemctl disable firewalld
#关闭核心防护
find_key="SELINUX="
sed -ri "/^$find_key/c${find_key}disabled" /etc/selinux/config

更改主机名（每个主机不一样）
hostnamectl set-hostname master1
su

3、部署etcd

3.1,准备文件

mkdir k8s 
cd k8s/
ls		 ##从宿主机将所需要的个文件在这，两个脚本，一个目录
etcd-cert  etcd-cert.sh  etcd.sh
cd etcd-cert/
chmod +x cfssl cfssl-certinfo cfssljson
mv cfssl cfssl-certinfo cfssljson /usr/local/bin/
ls /usr/local/bin/	##查看下有有没有下面三个文件
cfssl  cfssl-certinfo  cfssljson
#cfssl：生成证书工具
#cfssl-certinfo：查看证书信息
#cfssljson：通过传入json文件生成证书

3.2、制作CA证书

1、创建ca证书的配置文件
vim ca-config.json
{"signing": {"default": {   ##系统设定值"expiry": "87600h"  ##证书的有效期10年},"profiles": {   ##配置"www": {   ##台头是www"expiry": "87600h",  ##时效 87600h"usages": [   ##常见用法（标签）"signing",   ##签字签名"key encipherment",  ##加密"server auth",  ##服务端验证"client auth"   ##客户端验证]}}}
}
2、创建ca证书的签名证书  ##看到csr这个就是签名的证书
vim ca-csr.json
{"CN":"etcd CA",   ##CN中国"key": {    ##秘钥"algo":"rsa",   ##秘钥类型采用rsa非对称秘钥"size":2048   ##秘钥长度},"names": [   ##证书的名称{"C":"CN",   ##来自于中国"L":"Beijing",  ##地区"ST":"Beijing"  ##时区}]
}3、用ca签名证书生成ca证书，得到ca-key.pem ca.pem
cfssl gencert -initca ca-csr.json | cfssljson -bare ca -
ls	##能看到下面六个文件
ca-config.json  ca.csr  ca-csr.json  ca-key.pem  ca.pem  etcd-cert.sh

上面指令3的部分解释。
gencert -initca读取
ca-csr.json初始化用的
cfssljson -bare基本信息的读取时-bare
ca-key.pem ca.pem这两个是所获得的证书
ca-config.json做server要用到的

3.3、制作etcd节点通讯json，生产秘钥文件

vim server-csr.json    ##通讯验证证书
{"CN": "etcd","hosts": [   ##指明host的文件地址"10.4.7.11","10.4.7.21","10.4.7.22"],"key": {"algo": "rsa",   ##给你ca相匹配的秘钥证明"size": 2048},"names": [{"C": "CN","L": "Beijing","ST": "Beijing"}]
}cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=www server-csr.json | cfssljson -bare serverls	##看一下会有下面八个证书#其中server-key.pem  server.pem是最重要的
ca-config.json  ca-csr.json  ca.pem        server.csr       server-key.pem
ca.csr          ca-key.pem   etcd-cert.sh  server-csr.json  server.pem

3.4、使用etcd，证书搭建etcd集群

1、上传一个生成ETCD配置文件的脚本etcd.sh到 /root/k8s 目录下，脚本内容如下：
cd ..
tar zxf etcd-v3.3.10-linux-amd64.tar.gz
2、解压后到etcd-v3.3.10-linux-amd64下会生成两个文件etcd 和etcdctl是很重要的。
mkdir /opt/etcd/{cfg,bin,ssl} -p ##在opt目录中创建etcd集群的工作目录
ls /opt/etcd/
#bin  cfg  ssl   		##看是否创建成功
cd /root/k8s/etcd-v3.3.10-linux-amd64/
mv etcd etcdctl /opt/etcd/bin/ ##将解压后的文件放在/opt/etcd/bin 目录下
ls /opt/etcd/bin/   		##查看一下有两个文件了etcd  etcdctl
cd /root/k8s/etcd-cert/
cp *.pem /opt/etcd/ssl/  	##将所有的证书放在/opt/etcd/ssl中
ls /opt/etcd/ssl/
#ca-key.pem  ca.pem  server-key.pem  server.pem       ##再去查看下有没有
cd /root/k8s/  ##将这个三个压缩包gz结尾的从宿主机放在它下面
#cfssl.sh   etcd-v3.3.10-linux-amd64            kubernetes-server-linux-amd64.tar.gz
#etcd-cert  etcd-v3.3.10-linux-amd64.tar.gz
#etcd.sh    flannel-v0.10.0-linux-amd64.tar.gz
3、执行 etcd.sh 脚本产生etcd集群的配置脚本和服务启动脚本，进入卡住状态等待其他节点加入
#注意：修改成自己的ip地址《vim /root/k8s/etcd.sh这个文件中能看你配置的群集信息》
#注意：修改成自己的ip地址《vim /root/k8s/etcd.sh这个文件中能看你配置的群集信息》
[root@localhost k8s]# bash etcd.sh etcd01 10.4.7.11 etcd02=https://10.4.7.21:2380,etcd03=https://10.4.7.22:23804、分发文件到node节点
scp -r /opt/etcd/ root@10.4.7.21:/opt/
scp -r /opt/etcd/ root@10.4.7.22:/opt/
#将所有的配置文件远程拷贝到对方的/opt目录下，记得一定是这个目录，不然你要重新改配置文件。
scp /usr/lib/systemd/system/etcd.service root@10.4.7.21:/usr/lib/systemd/system/
scp /usr/lib/systemd/system/etcd.service root@10.4.7.22:/usr/lib/systemd/system/
#将启动脚本远程拷贝到node节点的启动脚本的位置5、在node上面进行相关的更改节点node2的node1差不对这里不多解释
cd /opt/etcd/cfg       #
[root@node2 cfg]# vim etcd   ##修改集群的名称和ip#[Member]    ##成员
ETCD_NAME="etcd02"    ##修改etcd节点的名称
ETCD_DATA_DIR="/var/lib/etcd/default.etcd"  ##服务运行数据保存的路径(设置数据保存的目录)
ETCD_LISTEN_PEER_URLS="https://20.0.0.11:2380" ##监听的同伴通信的地址,比如http://ip:2380,如果有多个,使用逗号分隔。需要所有节点都能够访问,所以不要使用 localhost!(用于监听其他etcd member的url (统一资源定位器，有叫网页地址))
ETCD_LISTEN_CLIENT_URLS="https://20.0.0.11:2379" ##监听的客户端服务地址(对外提供服务的地址（CLIENT客户机）)#[Clustering]     #使成群（cluster的现在分词）或聚类
ETCD_INITIAL_ADVERTISE_PEER_URLS="https://20.0.0.11:2380" ##对外公告的该节点同伴监听地址,这个值会告诉集群中其他节点
ETCD_ADVERTISE_CLIENT_URLS="https://20.0.0.11:2379"  ##对外公告的该节点客户端监听地址,这个值会告诉集群中其他节点。
ETCD_INITIAL_CLUSTER="etcd01=https://20.0.0.10:2380,etcd02=https://20.0.0.11:2380,etcd03=https://20.0.0.12:2380"    ##集群中所有节点的信息,格式为
ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster"  ##集群的ID,多个集群的时候,每个集群的ID必须保持唯一
ETCD_INITIAL_CLUSTER_STATE="new"   ##新建集群的时候,这个值为 new;假如加入已经存在的集群,这个值为existing。这里是具体的要更改的说明
vim etcd
#[Member]
ETCD_NAME="etcd02" ##改
ETCD_DATA_DIR="/var/lib/etcd/default.etcd"
ETCD_LISTEN_PEER_URLS="https://20.0.0.11:2380" ##改
ETCD_LISTEN_CLIENT_URLS="https://20.0.0.11:2379" ##改
##
#[Clustering]
ETCD_INITIAL_ADVERTISE_PEER_URLS="https://20.0.0.11:2380" ##改
ETCD_ADVERTISE_CLIENT_URLS="https://20.0.0.11:2379"  ##改
ETCD_INITIAL_CLUSTER="etcd01=https://20.0.0.10:2380,etcd02=https://20.0.0.11:2380,etcd03=https://20.0.0.12:2380"
ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster"
ETCD_INITIAL_CLUSTER_STATE="new"6、两个node节点都改过后可以去master节点上将集群打开，，，原来开的应该会超时而自动退出，所有要重新去开启，
[root@localhost k8s]# bash etcd.sh etcd01 10.4.7.11 etcd02=https://10.4.7.21:2380,etcd03=https://10.4.7.22:23807、然后回到node节点开启etcd
[root@localhost ~]# systemctl start etcd
[root@localhost ~]# systemctl status etcd
若是running状态证明是正常的8、最后到master去执行集群健康检查
cd /root/k8s/etcd-cert
/opt/etcd/bin/etcdctl --ca-file=ca.pem --cert-file=server.pem --key-file=server-key.pem --endpoints="https://10.4.7.11:2379,https://10.4.7.21:2379,https://10.4.7.22:2379" cluster-healthmember 988139385f78284 is healthy: got healthy result from https://10.4.7.22:2379
member 1ba3960d0c371211 is healthy: got healthy result from https://10.4.7.11:2379
member 5a0ef2a004fc4349 is healthy: got healthy result from https://10.4.7.21:2379
cluster is healthy出现cluster is healthy说明正常

4、部署docker

4.1、安装依赖包

 yum -y install yum-utils device-mapper-persistent-data lvm2yum-utils 提供了yum-config-manager
device 毛片儿存储驱动程序需要device-mapperpersistent-data和lvm2
device mapper 是linux2.6 内核中支持逻辑管理的通用设备映射机制
它为实现用于存储资源管理的块设备驱动提供了 一个高度模块化的内核架构

4.2、设置阿里云镜像源

yum-config-manager --add-repo https://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo

4.3、安装Docker-ce

yum -y install docker-ce

systemctl start docker

systemctl enable docker

4.4、设置镜像加速器

tee /etc/docker/daemon.json <<-'EOF'
{"registry-mirrors":["https://05vz3np5.mirror.aliyuncs.com"]
}
EOFdocker daemon-reload
systemctl restart docker

4.5、网络优化

vim /etc/sysctl.confnet.ipv4.ip_forward=1sysctl -p
service network restart
systemctl restart docker

4.6、查看命令

查看docker 版本
docker version
搜索nginx镜像（公有仓库）
docker search nginx

5、flannel网络配置

5.1、写入分配的子网段etcd中供flannel使用

master1上执行

cd /root/k8s/etcd-cert
/opt/etcd/bin/etcdctl --ca-file=ca.pem --cert-file=server.pem --key-file=server-key.pem --endpoints="https://10.4.7.11:2379,https://10.4.7.21:2379,https://10.4.7.22:2379" set /coreos.com/network/config '{"Network": "172.17.0.0/16","Backend": {"Type":"vxlan"}}'

5.2、将flannel放到node节点

tar zxvf flannel-v0.10.0-linux-amd64.tar.gz
mkdir /opt/kubernetes/{cfg,bin,ssl} -p
mv mk-docker-opts.sh flanneld /opt/kubernetes/bin

5.3、将flannel放到node上，执行安装

bash flannel.sh https://10.4.7.11:2379,https://10.4.7.21:2379,https://10.4.7.22:2379

5.4、flannel网络配置docker

vim /usr/lib/systemd/system/docker.service13 # for containers run by docker14 EnvironmentFile=/run/flannel/subnet.env  ##新增15 ExecStart=/usr/bin/dockerd $DOCKER_NETWORK_OPTIONS -H fd:// --containerd=/run/containerd/containerd.sock  ##添加部分内容## 查看获取ip地址的范围
cat /run/flannel/subnet.env 
DOCKER_OPT_BIP="--bip=172.17.54.1/24"
DOCKER_OPT_IPMASQ="--ip-masq=false"
DOCKER_OPT_MTU="--mtu=1450"
DOCKER_NETWORK_OPTIONS=" --bip=172.17.54.1/24 --ip-masq=false --mtu=1450"[root@node1 ~]# systemctl daemon-reload
[root@node1 ~]# systemctl restart docker

5.5、下载一个镜像进程测试下(下载centos)

7、1进入容器后直接看有没有IP地址
yum -y install net-tools
ifconfig                ##这个时候会随机出现ip地址给你如
eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1450inet 172.17.54.2  netmask 255.255.255.0  broadcast 172.17.54.255ether 02:42:ac:11:36:02  txqueuelen 0  (Ethernet)RX packets 13840  bytes 11482584 (10.9 MiB)RX errors 0  dropped 0  overruns 0  frame 0TX packets 7536  bytes 411117 (401.4 KiB)TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0lo: flags=73<UP,LOOPBACK,RUNNING>  mtu 65536inet 127.0.0.1  netmask 255.0.0.0loop  txqueuelen 1  (Local Loopback)RX packets 0  bytes 0 (0.0 B)RX errors 0  dropped 0  overruns 0  frame 0TX packets 0  bytes 0 (0.0 B)TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

6、部署master组件

6.1、在master，api-server 生成证书

上传master.zip 到master1节点
unzip master.zip
mkdir /opt/kubernetes/{cfg,bin,ssl} -p
mkdir k8s-cert
cd k8s-cert/添加相关的文件
cat > ca-config.json <<EOF
{"signing": {"default": {"expiry": "87600h"},"profiles": {"kubernetes": {"expiry": "87600h","usages": ["signing","key encipherment","server auth","client auth"]}}}
}
EOFcat > ca-csr.json <<EOF
{"CN": "kubernetes","key": {"algo": "rsa","size": 2048},"names": [{"C": "CN","L": "Beijing","ST": "Beijing","O": "k8s","OU": "System"}]
}
EOFcfssl gencert -initca ca-csr.json | cfssljson -bare ca -cat > server-csr.json <<EOF
{"CN": "kubernetes","hosts": ["10.0.0.1","127.0.0.1","10.4.7.11","10.4.7.12","10.4.7.100","10.4.7.23","10.4.7.24","kubernetes","kubernetes.default","kubernetes.default.svc","kubernetes.default.svc.cluster","kubernetes.default.svc.cluster.local"],"key": {"algo": "rsa","size": 2048},"names": [{"C": "CN","L": "BeiJing","ST": "BeiJing","O": "k8s","OU": "System"}]
}
EOFcfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=kubernetes server-csr.json | cfssljson -bare servercat > admin-csr.json <<EOF
{"CN": "admin","hosts": [],"key": {"algo": "rsa","size": 2048},"names": [{"C": "CN","L": "BeiJing","ST": "BeiJing","O": "system:masters","OU": "System"}]
}
EOFcfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=kubernetes admin-csr.json | cfssljson -bare admincat > kube-proxy-csr.json <<EOF
{"CN": "system:kube-proxy","hosts": [],"key": {"algo": "rsa","size": 2048},"names": [{"C": "CN","L": "BeiJing","ST": "BeiJing","O": "k8s","OU": "System"}]
}
EOFcfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=kubernetes kube-proxy-csr.json | cfssljson -bare kube-proxy查看生成的k8s证书[root@master k8s-cert]# ls *pem
admin-key.pem  ca-key.pem  kube-proxy-key.pem  server-key.pem
admin.pem      ca.pem      kube-proxy.pem      server.pemcp ca*pem server*pem /opt/kubernetes/ssl/
cd /root/k8s
tar zxvf kubernetes-server-linux-amd64.tar.gz
cd /root/k8s/kubernetes/server/bin
cp kube-apiserver kubectl kube-controller-manager kube-scheduler /opt/kubernetes/bin/
cd /root/k8s生成令牌
cat > /opt/kubernetes/cfg/token.csv << EOF
`head -c 16 /dev/urandom | od -An -t x | tr -d ' '`,kubelet-bootstrap,10001,"system:kubelet-bootstrap"
EOF开启apiserver
bash apiserver.sh 10.4.7.11 https://10.4.7.21:2379,https://10.4.7.21:2379,https://10.4.7.22:2379启动scheduler服务
./scheduler.sh 127.0.0.1启动controller-manager
chmod +x controller-manager.sh
./controller-manager.sh 127.0.0.1查看master 节点状态
cp /opt/kubernetes/bin/kubectl /usr/local/bin
[root@master1 k8s-cert]# kubectl get cs
NAME                 STATUS    MESSAGE             ERROR
scheduler            Healthy   ok
controller-manager   Healthy   ok
etcd-0               Healthy   {"health":"true"}
etcd-1               Healthy   {"health":"true"}
etcd-2               Healthy   {"health":"true"}把 kubelet、kube-proxy拷贝到node节点上去
cd /root/k8s/kubernetes/server/bin
scp kubelet kube-proxy 10.4.7.21:/opt/kubernetes/bin/
scp kubelet kube-proxy 10.4.7.22:/opt/kubernetes/bin/cd /root/k8s
mkdir kubeconfig
cd kubeconfig/拷贝kubeconfig.sh文件进行重命名
mv kubeconfig.sh kubeconfig查看令牌的id号是什么
cat /opt/kubernetes/cfg/token.csv
更改kubeconfig的相关参数# 创建 TLS Bootstrapping Token#BOOTSTRAP_TOKEN=$(head -c 16 /dev/urandom | od -An -t x | tr -d ' ')BOOTSTRAP_TOKEN=0fb61c46f8991b718eb38d27b605b008    ## 将序列号更换为刚才查到的设置环境变量（可以写入到/etc/profile中）
export PATH=$PATH:/opt/kubernetes/bin/生成配置文件
bash kubeconfig 10.4.7.11 /root/k8s/k8s-cert/将生成的文件拷贝到node节点
scp bootstrap.kubeconfig kube-proxy.kubeconfig 10.4.7.21:/opt/kubernetes/cfg/
scp bootstrap.kubeconfig kube-proxy.kubeconfig 10.4.7.22:/opt/kubernetes/cfg/创建bootstrap角色赋予权限用于连接apiserver请求签名
kubectl create clusterrolebinding kubelet-bootstrap --clusterrole=system:node-bootstrapper --user=kubelet-bootstrap

6.2、node节点上操作

nod01节点操作（复制node.zip到/root目录下再解压）
unzip node.zip 
bash kubelet.sh 10.4.7.21检查kubelet服务启动
ps aux | grep kubenod02节点操作（复制node.zip到/root目录下再解压）
unzip node.zip 
bash kubelet.sh 10.4.7.22检查kubelet服务启动
ps aux | grep kube

6.3、master1节点上添加授权（node1和node2类同）

[root@localhost kubeconfig]# kubectl get csr
NAME                                                   AGE     REQUESTOR           CONDITION
node-csr-NOI-9vufTLIqJgMWq4fHPNPHKbjCXlDGHptj7FqTa8A   4m27s   kubelet-bootstrap   Pending（等待集群给该节点颁发证书）[root@localhost kubeconfig]# kubectl certificate approve node-csr-NOI-9vufTLIqJgMWq4fHPNPHKbjCXlDGHptj7FqTa8A
certificatesigningrequest.certificates.k8s.io/node-csr-NOI-9vufTLIqJgMWq4fHPNPHKbjCXlDGHptj7FqTa8A approved
//继续查看证书状态
[root@localhost kubeconfig]# kubectl get csr
NAME                                                   AGE     REQUESTOR           CONDITION
node-csr-NOI-9vufTLIqJgMWq4fHPNPHKbjCXlDGHptj7FqTa8A   8m56s   kubelet-bootstrap   Approved,Issued（已经被允许加入群集）
//查看群集节点，成功加入node01节点
[root@localhost kubeconfig]# kubectl get node
NAME              STATUS   ROLES    AGE    VERSION
10.4.7.21   Ready    <none>   118s   v1.12.32、在node01节点操作，启动proxy服务
[root@localhost ~]# bash proxy.sh 10.4.7.21
Created symlink from /etc/systemd/system/multi-user.target.wants/kube-proxy.service to /usr/lib/systemd/system/kube-proxy.service.
[root@localhost ~]# systemctl status kube-proxy.service 

6.4、master节点上查看下部署是否正常了

[root@master1 ~]# kubectl get node
NAME        STATUS   ROLES    AGE   VERSION
10.4.7.21   Ready    <none>   11h   v1.12.3
10.4.7.22   Ready    <none>   11h   v1.12.3
[root@master1 ~]#

二、部署多节点集群之添加master2节点

1、初始化master2环境信息

# 关闭防火墙
systemctl stop firewalld
systemctl disable firewalld
#关闭核心防护
find_key="SELINUX="
sed -ri "/^$find_key/c${find_key}disabled" /etc/selinux/config更改主机名（每个主机不一样）
hostnamectl set-hostname master2
su

2、将基本信息从master1拷贝到master2

scp -r /opt/kubernetes 10.4.7.12:/opt
scp /usr/lib/systemd/system/{kube-apiserver,kube-controller-manager,kube-scheduler}.service 10.4.7.12:/usr/lib/systemd/system
scp -r /opt/etcd 10.4.7.12:/opt

3、到master2上修改相关的参数信息

cd /opt/kubernetes/cfg
vim kube-apiserver
[root@master2 cfg]# vim kube-apiserver
[root@master2 cfg]# cat  kube-apiserverKUBE_APISERVER_OPTS="--logtostderr=true \
--v=4 \
--etcd-servers=https://10.4.7.11:2379,https://10.4.7.21:2379,https://10.4.7.22:2379 \
--bind-address=10.4.7.12 \      ## 更改
--secure-port=6443 \
--advertise-address=10.4.7.12 \   ## 更改

4、在master2上开启apiserver，controller-manager，scheduler服务

systemctl start kube-apiserver.service && systemctl start kube-scheduler.service && systemctl start kube-controller-manager.service
systemctl status kube-apiserver.service && systemctl status kube-scheduler.service && systemctl status kube-controller-manager.servicecp /opt/kubernetes/bin/kubectl /usr/local/bin
[root@master2 cfg]# kubectl get node
NAME        STATUS   ROLES    AGE   VERSION
10.4.7.21   Ready    <none>   13h   v1.12.3
10.4.7.22   Ready    <none>   13h   v1.12.3

这时候其实master2问题还是很大的。。。master1挂了，但是2还是不能直接顶替它干活的，虽然能看它的节点信息，但是node指的地址都是master1，所有的都指向的是master1

我们可以在任意一个node上看下kubelet的配置kubelet.kubeconfig

三、负载均衡集群

1、在开两台做nginx负载均衡，23,24 初始化环境信息

负载均衡做nginx还要做反向代理，有个VIP要是给外面提供的。你的VIP地址不能消失，消失了node就会是no 连接
# 关闭防火墙
systemctl stop firewalld
systemctl disable firewalld
#关闭核心防护
find_key="SELINUX="
sed -ri "/^$find_key/c${find_key}disabled" /etc/selinux/config

2、安装nginx（两台操作一样）

cat > /etc/yum.repos.d/nginx.repo << EOF
[nginx-stable]
name=nginx stable repo
baseurl=http://nginx.org/packages/centos/$releasever/$basearch/
gpgcheck=1
enabled=1
gpgkey=https://nginx.org/keys/nginx_signing.key
module_hotfixes=true
EOFyum -y install nginx

3、对nginx做基本配置（两台操作一样）

四层转发的内容需要添加在配置脚本上
vim /etc/nginx/nginx.confuser  nginx;
worker_processes  auto;error_log  /var/log/nginx/error.log notice;
pid        /var/run/nginx.pid;events {worker_connections  1024;
}stream {log_format main '$remote_addr $upstream_addr - [$time_local] $status $upstream_bytes_sent';access_log /var/log/nginx/k8s-access.log main;upstream k8s-apiserver {server 10.4.7.11:6443;server 10.4.7.12:6443;}server {listen 6443;proxy_pass k8s-apiserver;}
}http {include       /etc/nginx/mime.types;default_type  application/octet-stream;log_format  main  '$remote_addr - $remote_user [$time_local] "$request" ''$status $body_bytes_sent "$http_referer" ''"$http_user_agent" "$http_x_forwarded_for"';access_log  /var/log/nginx/access.log  main;sendfile        on;#tcp_nopush     on;keepalive_timeout  65;#gzip  on;include /etc/nginx/conf.d/*.conf;
}

4、启动nginx（两台操作一样）

systemctl start nginx

systemctl enable nginx

5、安装keepalived 服务(并制作配置文件)

yum -y install keepalivedcat > /root/keepalived.conf <<EOF
! Configuration File for keepalivedglobal_defs {# 接收邮件地址notification_email {acassen@firewall.locfailover@firewall.locsysadmin@firewall.loc}# 邮件发送地址notification_email_from Alexandre.Cassen@firewall.locsmtp_server 127.0.0.1smtp_connect_timeout 30router_id NGINX_MASTER
}vrrp_script check_nginx {script "/etc/nginx/check_nginx.sh"
}vrrp_instance VI_1 {state MASTERinterface ens33virtual_router_id 51 # VRRP 路由 ID实例，每个实例是唯一的priority 100    # 优先级，备服务器设置 90advert_int 1    # 指定VRRP 心跳包通告间隔时间，默认1秒authentication {auth_type PASSauth_pass 1111}virtual_ipaddress {10.4.7.100/24}track_script {check_nginx}
}
EOFcat > /etc/nginx/check_nginx.sh << EOF         ## 这个有个问题就是$符号所以需要查看下
#!/bin/bash
chect_nginx_start (){count=$(ps -ef |grep keepalived |egrep -cv "grep |$$")if [ "$count" -ne 3 ];thensystemctl start keepalivedfi}test (){count=$(ps -ef |grep nginx |egrep -cv "grep |$$")if [ "$count" -eq 0 ];thensystemctl stop keepalivedelsechect_nginx_startfi}
test
EOF chmod +x /etc/nginx/check_nginx.sh最好是添加一个计划任务
crontab -e
* * * * * sh /etc/nginx/check_nginx.sh

6、启动keepalived

systemctl start keepalived   
ip a            # 启动后需要查看下是否有漂移ip地址了说明下：： 当两台nginx服务都启动后，keepalived服务在启动的时候不会伴随nginx服务的启动而启动，但是会伴随nginx的关闭而关闭，你可以做个测试，关掉一个nginx服务，看下漂移地址会不会到另外一个nginx的节点上去。

7、下面做VIP的负载均衡让node1和node2认master1和master2

7.1、要改3个文件（两个node节点操作）

cd /opt/kubernetes/cfg/
bootstrap.kubeconfig    将里面的10.4.7.11 改为 10.4.7.100
kubelet.kubeconfig      将里面的10.4.7.11 改为 10.4.7.100
kube-proxy.kubeconfig   将里面的10.4.7.11 改为 10.4.7.100

7.2、重启kubelet和kube-proxy服务

systemctl restart kubelet.service
systemctl restart kube-proxy.service

8、创建一个deployment查看下是否正常

kubectl run nginx --image=nginx