OpenStack HA高可用集群Train版-0集群环境准备

OpenStack HA高可用集群Train版-0集群环境准备

主机配置

1.主机名

hostnamectl set-hostname controller01
exec bashhostnamectl set-hostname controller02
exec bashhostnamectl set-hostname controller03
exec bashhostnamectl set-hostname compute01
exec bashhostnamectl set-hostname compute02
exec bashhostnamectl set-hostname compute03
exec bash

2.网卡配置

网卡UUID

nmcli con show
uuidgen ens33

cd /etc/sysconfig/network-scripts/
ls

reboot

nmcli con show
cat /etc/sysconfig/network-scripts/ifcfg-ens33
cat /etc/sysconfig/network-scripts/ifcfg-ens37
cat /etc/networks

ifconfig

配置主机名解析

echo 10.0.0.41 controller01 >> /etc/hosts
echo 10.0.0.42 controller02 >> /etc/hosts
echo 10.0.0.43 controller03 >> /etc/hosts
echo 10.0.0.51 compute01 >> /etc/hosts
echo 10.0.0.52 compute02 >> /etc/hosts
echo 10.0.0.53 compute03 >> /etc/hosts

cat /etc/hosts

ping -c 4 controller01
ping -c 4 controller02
ping -c 4 controller03
ping -c 4 compute01
ping -c 4 compute02
ping -c 4 compute03
ping -c 4 qq.com

高可用OpenStack（Queen版）集群-1. 集群环境

配置免密登录

在controller01生成密钥

ssh-keygen -t rsa

ssh-copy-id controller01 

ssh-copy-id controller02
scp -r .ssh controller02:~/

ssh-copy-id controller03
scp -r .ssh controller03:~/

ssh-copy-id compute01
scp -r .ssh compute01:~/

ssh-copy-id compute02
scp -r .ssh compute02:~/

ssh-copy-id compute03
scp -r .ssh compute03:~/

快照拍一个

防火墙相关配置

systemctl stop firewalld.service
systemctl disable firewalld.service
systemctl status firewalld.service

setenforce 0
getenforce
sed -i 's#SELINUX=enforcing#SELINUX=disabled#g' /etc/sysconfig/selinux
grep SELINUX=disabled /etc/sysconfig/selinux
cat /etc/sysconfig/selinux

reboot

sestatus

时间同步配置

每个节点

yum install chrony -y 

cp /etc/chrony.conf /etc/chrony.conf.bak

设计vim的部分都可以进行优化，要么sed改，要么cat EOF进行文件替换

vim /etc/chrony.conf

egrep -v "^$|^#" /etc/chrony.conf 

[root@controller03 ~]# egrep -v "^$|^#" /etc/chrony.conf 
server ntp1.aliyun.com iburst
server controller01 iburst
server controller02 iburst
server controller03 iburst
driftfile /var/lib/chrony/drift
makestep 1.0 3
rtcsync
allow 10.0.0.2/24
logdir /var/log/chrony
[root@controller03 ~]# 

systemctl enable chronyd.service
systemctl restart chronyd.service
systemctl status chronyd.service

timedatectl set-timezone Asia/Shanghai
chronyc sources -v
timedatectl status
date

当前节点为controller03

scp -r /etc/chrony.conf root@controller01:/etc/chrony.conf
scp -r /etc/chrony.conf root@controller02:/etc/chrony.conf
scp -r /etc/chrony.conf root@controller03:/etc/chrony.conf
scp -r /etc/chrony.conf root@compute01:/etc/chrony.conf
scp -r /etc/chrony.conf root@compute02:/etc/chrony.conf
scp -r /etc/chrony.conf root@compute03:/etc/chrony.conf

确认yum源，可以更换为本地源

cat > /etc/yum.repos.d/CentOS-PrivateLocal.repo <<EOF [base]
name=CentOS-$releasever - Base
baseurl=http://192.168.2.103/yumrepository/base/
gpgcheck=0
enabled=1[updates]
name=CentOS-$releasever - Updates
baseurl=http://192.168.2.103/yumrepository/updates/
gpgcheck=0
enabled=1[extras]
name=CentOS-$releasever - Extras
baseurl=http://192.168.2.103/yumrepository/extras/
gpgcheck=0
enabled=1[centos-openstack-train]
name=CentOS-7 - OpenStack train
baseurl=http://192.168.2.103/yumrepository/centos-openstack-train/
gpgcheck=0
enabled=1[centos-qemu-ev]
name=CentOS-$releasever - QEMU EV
baseurl=http://192.168.2.103/yumrepository/centos-qemu-ev/
gpgcheck=0
enabled=1[centos-ceph-nautilus]
name=CentOS-7 - Ceph Nautilus
baseurl=http://192.168.2.103/yumrepository/centos-ceph-nautilus/
gpgcheck=0
enabled=1[centos-nfs-ganesha28]
name=CentOS-7 - NFS Ganesha 2.8
baseurl=http://192.168.2.103/yumrepository/centos-nfs-ganesha28/
gpgcheck=0
enabled=1
EOF

ls /etc/yum.repos.d/CentOS-PrivateLocal.repo

yum install centos-release-openstack-train -y
yum clean all
yum makecache

yum install python-openstackclient openstack-selinux -y
yum install openstack-utils -y 
yum install -y lsof vim net-tools wget git 

poweroff

所有控制和计算节点，拍快照

配置完成
一环境准备

https://www.cnblogs.com/netonline/p/9180299.html

https://blog.csdn.net/u013469753/article/details/106274225

二、基础软件安装

https://www.cnblogs.com/netonline/p/9184691.html

https://blog.csdn.net/u013469753/article/details/106274225

数据库

在全部controller节点安装mariadb，以controller01节点为例

所有控制节点

在全部controller节点安装mariadb，以controller01节点为例
安装galera相关插件，利用galera搭建集群

yum install mariadb mariadb-server MySQL-python python2-PyMySQL -y
yum install mariadb-server-galera mariadb-galera-common galera xinetd rsync -y

在全部控制节点初始化数据库密码，以controller01节点为例；
root初始密码为空

systemctl restart mariadb.service
systemctl status mariadb.service

mysql_secure_installation <<EOFy
mysql_pass
mysql_pass
y
n
y
y
EOF

mysql -uroot -pmysql_pass
exit

或者手动设置

mysql_secure_installation

[root@controller01 ~]# mysql_secure_installation
Enter current password for root (enter for none): #直接按回车
Set root password? [Y/n] y
New password:               #设置root用户密码为：mysql_pass
Re-enter new password: 
Remove anonymous users? [Y/n] y
Disallow root login remotely? [Y/n] n #注意允许
Remove test database and access to it? [Y/n] y
Reload privilege tables now? [Y/n] y

所有控制节点创建OpenStack的配置文件，IP地址根据实际所在节点进行修改

#touch /etc/my.cnf.d/openstack.cnf

# [mysqld]
# bind-address = 0.0.0.0
# default-storage-engine = innodb
# innodb_file_per_table = on
# max_connections = 4096
# collation-server = utf8_general_ci
# character-set-server = utf8
# init-connect = 'SET NAMES utf8'

在全部控制节点/etc/my.cnf.d/目录下新增openstack.cnf配置文件，主要设置集群同步相关参数.

以controller01节点为例，个别涉及ip地址/host名等参数根据实际情况修改

控制节点1

cat > /etc/my.cnf.d/openstack.cnf <<EOF
[mysqld]
binlog_format = ROW
bind-address = 0.0.0.0
default-storage-engine = innodb
innodb_file_per_table = on
max_connections = 4096
collation-server = utf8_general_ci
character-set-server = utf8
init-connect = 'SET NAMES utf8'[galera]
bind-address = 10.0.0.41
wsrep_provider = /usr/lib64/galera/libgalera_smm.so
wsrep_cluster_address ="gcomm://controller01,controller02,controller03"
wsrep_cluster_name = openstack-cluster-01
wsrep_node_name = controller01
wsrep_node_address = 10.0.0.41
wsrep_on=ON
wsrep_slave_threads=4
wsrep_sst_method=rsync
default_storage_engine=InnoDB
[embedded]
[mariadb]
[mariadb-10.1]
EOF

控制节点2

cat > /etc/my.cnf.d/openstack.cnf <<EOF
[mysqld]
binlog_format = ROW
bind-address = 0.0.0.0
default-storage-engine = innodb
innodb_file_per_table = on
max_connections = 4096
collation-server = utf8_general_ci
character-set-server = utf8
init-connect = 'SET NAMES utf8'[galera]
bind-address = 10.0.0.42
wsrep_provider = /usr/lib64/galera/libgalera_smm.so
wsrep_cluster_address ="gcomm://controller01,controller02,controller03"
wsrep_cluster_name = openstack-cluster-01
wsrep_node_name = controller02
wsrep_node_address = 10.0.0.42
wsrep_on=ON
wsrep_slave_threads=4
wsrep_sst_method=rsync
default_storage_engine=InnoDB
[embedded]
[mariadb]
[mariadb-10.1]
EOF

控制节点3

cat > /etc/my.cnf.d/openstack.cnf <<EOF
[mysqld]
binlog_format = ROW
bind-address = 0.0.0.0
default-storage-engine = innodb
innodb_file_per_table = on
max_connections = 4096
collation-server = utf8_general_ci
character-set-server = utf8
init-connect = 'SET NAMES utf8'[galera]
bind-address = 10.0.0.43
wsrep_provider = /usr/lib64/galera/libgalera_smm.so
wsrep_cluster_address ="gcomm://controller01,controller02,controller03"
wsrep_cluster_name = openstack-cluster-01
wsrep_node_name = controller03
wsrep_node_address = 10.0.0.43
wsrep_on=ON
wsrep_slave_threads=4
wsrep_sst_method=rsync
default_storage_engine=InnoDB
[embedded]
[mariadb]
[mariadb-10.1]
EOF

cat /etc/my.cnf.d/openstack.cnf

所有控制节点先拍个快照

https://www.cnblogs.com/netonline/p/9184691.html

构建数据库集群

停止全部控制节点的mariadb服务，以controller01节点为例

systemctl stop mariadb.service
systemctl status mariadb.service

拍个快照，后面服务status异常，未能启动成功

任选1个控制节点以如下方式启动mariadb服务，这里选择controller01节点

/usr/libexec/mysqld --wsrep-new-cluster --user=root &
systemctl status mariadb.service

其他控制节点加入mariadb集群，以controller02节点为例，启动后加入集群，通过查看service的status，可以看到controller02节点从controller01节点同步数据，也可同步查看mariadb日志/var/log/mariadb/mariadb.log

systemctl start mariadb.service
systemctl status mariadb.service

控制节点1

[root@controller01 ~]# /usr/libexec/mysqld --wsrep-new-cluster --user=root &
[1] 1229
[root@controller01 ~]# 2022-01-13  9:37:20 0 [Note] /usr/libexec/mysqld (mysqld 10.3.20-MariaDB) starting as process 1229 ...[root@controller01 ~]# systemctl status mariadb.service
● mariadb.service - MariaDB 10.3 database serverLoaded: loaded (/usr/lib/systemd/system/mariadb.service; disabled; vendor preset: disabled)Active: inactive (dead)Docs: man:mysqld(8)https://mariadb.com/kb/en/library/systemd/
[root@controller01 ~]# 

控制节点2

[root@controller02 ~]# systemctl start mariadb.service
[root@controller02 ~]# 
[root@controller02 ~]# systemctl status mariadb.service
● mariadb.service - MariaDB 10.3 database serverLoaded: loaded (/usr/lib/systemd/system/mariadb.service; disabled; vendor preset: disabled)Active: active (running) since Thu 2022-01-13 09:37:57 CST; 4min 14s agoDocs: man:mysqld(8)https://mariadb.com/kb/en/library/systemd/Process: 1445 ExecStartPost=/usr/libexec/mysql-check-upgrade (code=exited, status=0/SUCCESS)Process: 1259 ExecStartPre=/usr/libexec/mysql-prepare-db-dir %n (code=exited, status=0/SUCCESS)Process: 1235 ExecStartPre=/usr/libexec/mysql-check-socket (code=exited, status=0/SUCCESS)Main PID: 1297 (mysqld)Status: "Taking your SQL requests now..."CGroup: /system.slice/mariadb.service└─1297 /usr/libexec/mysqld --basedir=/usrJan 13 09:37:55 controller02 rsyncd[1385]: connect from controller01 (10.0.0.41)
Jan 13 09:37:55 controller02 rsyncd[1385]: rsync to rsync_sst/./mysql from controller01 (10.0.0.41)
Jan 13 09:37:55 controller02 rsyncd[1385]: receiving file list
Jan 13 09:37:55 controller02 rsyncd[1391]: connect from controller01 (10.0.0.41)
Jan 13 09:37:55 controller02 rsyncd[1391]: rsync to rsync_sst/./performance_schema from controller01 (10.0.0.41)
Jan 13 09:37:55 controller02 rsyncd[1391]: receiving file list
Jan 13 09:37:55 controller02 rsyncd[1393]: connect from controller01 (10.0.0.41)
Jan 13 09:37:55 controller02 rsyncd[1393]: rsync to rsync_sst/ from controller01 (10.0.0.41)
Jan 13 09:37:55 controller02 rsyncd[1393]: receiving file list
Jan 13 09:37:57 controller02 systemd[1]: Started MariaDB 10.3 database server.
[root@controller02 ~]# 

控制节点3

[root@controller03 ~]# systemctl start mariadb.service
[root@controller03 ~]# 
[root@controller03 ~]# systemctl status mariadb.service
● mariadb.service - MariaDB 10.3 database serverLoaded: loaded (/usr/lib/systemd/system/mariadb.service; disabled; vendor preset: disabled)Active: active (running) since Thu 2022-01-13 09:42:39 CST; 2s agoDocs: man:mysqld(8)https://mariadb.com/kb/en/library/systemd/Process: 1442 ExecStartPost=/usr/libexec/mysql-check-upgrade (code=exited, status=0/SUCCESS)Process: 1260 ExecStartPre=/usr/libexec/mysql-prepare-db-dir %n (code=exited, status=0/SUCCESS)Process: 1236 ExecStartPre=/usr/libexec/mysql-check-socket (code=exited, status=0/SUCCESS)Main PID: 1298 (mysqld)Status: "Taking your SQL requests now..."CGroup: /system.slice/mariadb.service└─1298 /usr/libexec/mysqld --basedir=/usrJan 13 09:42:37 controller03 rsyncd[1386]: connect from controller01 (10.0.0.41)
Jan 13 09:42:38 controller03 rsyncd[1386]: rsync to rsync_sst/./mysql from controller01 (10.0.0.41)
Jan 13 09:42:38 controller03 rsyncd[1386]: receiving file list
Jan 13 09:42:38 controller03 rsyncd[1388]: connect from controller01 (10.0.0.41)
Jan 13 09:42:38 controller03 rsyncd[1388]: rsync to rsync_sst/./performance_schema from controller01 (10.0.0.41)
Jan 13 09:42:38 controller03 rsyncd[1388]: receiving file list
Jan 13 09:42:38 controller03 rsyncd[1390]: connect from controller01 (10.0.0.41)
Jan 13 09:42:38 controller03 rsyncd[1390]: rsync to rsync_sst/ from controller01 (10.0.0.41)
Jan 13 09:42:38 controller03 rsyncd[1390]: receiving file list
Jan 13 09:42:39 controller03 systemd[1]: Started MariaDB 10.3 database server.
[root@controller03 ~]# 

重新启动controller01节点，启动前删除contrller01节点的数据

pkill -9 mysql
rm -rf /var/lib/mysql/*

[root@controller01 ~]# pkill -9 mysql
[root@controller01 ~]# rm -rf /var/lib/mysql/*

控制节点以system unit方式启动mariadb服务时的权限配置

chown mysql:mysql /var/run/mariadb/mariadb.pid
reboot

[root@controller01 ~]# chown mysql:mysql /var/run/mariadb/mariadb.pid
[root@controller01 ~]# reboot

启动后查看节点所在服务状态，controller01节点从controller02节点同步数据

[root@controller01 ~]# systemctl start mariadb.service
[root@controller01 ~]# systemctl status mariadb.service
[root@controller01 ~]# systemctl status mariadb.service
● mariadb.service - MariaDB 10.3 database serverLoaded: loaded (/usr/lib/systemd/system/mariadb.service; disabled; vendor preset: disabled)Active: active (running) since Fri 2022-01-07 11:50:16 CST; 10min agoDocs: man:mysqld(8)https://mariadb.com/kb/en/library/systemd/Process: 1508 ExecStartPost=/usr/libexec/mysql-check-upgrade (code=exited, status=0/SUCCESS)Process: 1261 ExecStartPre=/usr/libexec/mysql-prepare-db-dir %n (code=exited, status=0/SUCCESS)Process: 1237 ExecStartPre=/usr/libexec/mysql-check-socket (code=exited, status=0/SUCCESS)Main PID: 1364 (mysqld)Status: "Taking your SQL requests now..."CGroup: /system.slice/mariadb.service└─1364 /usr/libexec/mysqld --basedir=/usrJan 07 11:50:15 controller01 rsyncd[1452]: connect from controller03 (10.0.0.43)
Jan 07 11:50:15 controller01 rsyncd[1452]: rsync to rsync_sst/./mysql from controller03 (10.0.0.43)
Jan 07 11:50:15 controller01 rsyncd[1452]: receiving file list
Jan 07 11:50:15 controller01 rsyncd[1454]: connect from controller03 (10.0.0.43)
Jan 07 11:50:15 controller01 rsyncd[1454]: rsync to rsync_sst/./performance_schema from controller03 (10.0.0.43)
Jan 07 11:50:15 controller01 rsyncd[1454]: receiving file list
Jan 07 11:50:15 controller01 rsyncd[1456]: connect from controller03 (10.0.0.43)
Jan 07 11:50:15 controller01 rsyncd[1456]: rsync to rsync_sst/ from controller03 (10.0.0.43)
Jan 07 11:50:15 controller01 rsyncd[1456]: receiving file list
Jan 07 11:50:16 controller01 systemd[1]: Started MariaDB 10.3 database server.

查看集群状态

mysql -uroot -pmysql_pass

[root@controller01 ~]# mysql -uroot -pmysql_pass
Welcome to the MariaDB monitor.  Commands end with ; or \g.
Your MariaDB connection id is 13
Server version: 10.3.20-MariaDB MariaDB ServerCopyright (c) 2000, 2018, Oracle, MariaDB Corporation Ab and others.Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.MariaDB [(none)]> show status like "wsrep_cluster_size";
+--------------------+-------+
| Variable_name      | Value |
+--------------------+-------+
| wsrep_cluster_size | 3     |
+--------------------+-------+
1 row in set (0.001 sec)MariaDB [(none)]> show status LIKE 'wsrep_ready';
+---------------+-------+
| Variable_name | Value |
+---------------+-------+
| wsrep_ready   | ON    |
+---------------+-------+
1 row in set (0.002 sec)MariaDB [(none)]> exit

设置心跳检测clustercheck

准备脚本

wget https://raw.githubusercontent.com/olafz/percona-clustercheck/master/clustercheck

[root@controller01 ~]# wget https://raw.githubusercontent.com/olafz/percona-clustercheck/master/clustercheck
--2022-01-07 13:46:43--  https://raw.githubusercontent.com/olafz/percona-clustercheck/master/clustercheck
Resolving raw.githubusercontent.com (raw.githubusercontent.com)... 185.199.109.133, 185.199.108.133, 185.199.111.133, ...
Connecting to raw.githubusercontent.com (raw.githubusercontent.com)|185.199.109.133|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 3882 (3.8K) [text/plain]
Saving to: ‘clustercheck’100%[===========================================================================================================================================>] 3,882       --.-K/s   in 0s      2022-01-07 13:46:44 (115 MB/s) - ‘clustercheck’ saved [3882/3882][root@controller01 ~]# 

chmod +x clustercheck
cp ~/clustercheck /usr/bin/

创建心跳检测用户,(任意控制节点)

在任意控制节点创建clustercheck_user用户并赋权；
注意账号/密码与脚本中的账号/密码对应，这里采用的是脚本默认的账号/密码，否则需要修改clustercheck脚本文件

mysql -uroot -pmysql_pass

GRANT PROCESS ON *.* TO 'clustercheckuser'@'localhost' IDENTIFIED BY 'clustercheckpassword!';
FLUSH PRIVILEGES;
exit

检测配置文件

在全部控制节点新增心跳检测服务配置文件/etc/xinetd.d/mysqlchk，以controller01节点为例

全部控制节点

cat > /etc/xinetd.d/mysqlchk <<EOF
# default: on
# description: mysqlchk
service mysqlchk
{port = 9200disable = nosocket_type = streamprotocol = tcpwait = nouser = rootgroup = rootgroups = yesserver = /usr/bin/clusterchecktype = UNLISTEDper_source = UNLIMITEDlog_on_success =log_on_failure = HOSTflags = REUSE
} 
EOF

每个控制节点启动心跳检测服务

每个控制节点，修改/etc/services，变更tcp9200端口用途，以controller01节点为例

vim /etc/services +10101

[root@controller01 ~]# vim /etc/services +10101
............
#wap-wsp         9200/tcp                # WAP connectionless session service
mysqlchk        9200/tcp                # mysqlchk
............

每个控制节点，启动xinetd服务，以controller01节点为例

systemctl daemon-reload
systemctl enable xinetd
systemctl start xinetd
systemctl status xinetd

[root@controller01 ~]# systemctl daemon-reload
[root@controller01 ~]# systemctl enable xinetd
[root@controller01 ~]# systemctl start xinetd

测试心跳检测脚本

在全部控制节点验证，以controller01节点为例

/usr/bin/clustercheck

[root@controller01 ~]# /usr/bin/clustercheck

https://www.cnblogs.com/netonline/p/9184691.html

设置心跳检测clustercheck

测试心跳检测脚本

!!! 心跳脚本测试失败，先关机，拍快照

poweroff

[root@controller01 ~]# systemctl daemon-reload
[root@controller01 ~]# systemctl enable xinetd
[root@controller01 ~]# systemctl start xinetd
[root@controller01 ~]# systemctl status xinetd
● xinetd.service - Xinetd A Powerful Replacement For InetdLoaded: loaded (/usr/lib/systemd/system/xinetd.service; enabled; vendor preset: enabled)Active: active (running) since Fri 2022-01-07 17:03:57 CST; 11min agoMain PID: 951 (xinetd)CGroup: /system.slice/xinetd.service└─951 /usr/sbin/xinetd -stayalive -pidfile /var/run/xinetd.pidJan 07 17:03:57 controller01 xinetd[951]: removing daytime
Jan 07 17:03:57 controller01 xinetd[951]: removing discard
Jan 07 17:03:57 controller01 xinetd[951]: removing discard
Jan 07 17:03:57 controller01 xinetd[951]: removing echo
Jan 07 17:03:57 controller01 xinetd[951]: removing echo
Jan 07 17:03:57 controller01 xinetd[951]: removing tcpmux
Jan 07 17:03:57 controller01 xinetd[951]: removing time
Jan 07 17:03:57 controller01 xinetd[951]: removing time
Jan 07 17:03:57 controller01 xinetd[951]: xinetd Version 2.3.15 started with libwrap ...n.
Jan 07 17:03:57 controller01 xinetd[951]: Started working: 0 available services
Hint: Some lines were ellipsized, use -l to show in full.
[root@controller01 ~]# /usr/bin/clustercheck
HTTP/1.1 503 Service Unavailable
Content-Type: text/plain
Connection: close
Content-Length: 44Percona XtraDB Cluster Node is not synced.
[root@controller01 ~]#