kolla-ansible ussuri 集群扩展计算节点
一 摘要
本文是基于上文[openstack ussuri版本基于内网三台物理机集群kolla-ansible部署及与ceph 集群 集成]（https://www.cnblogs.com/weiwei2021/p/14200722.html）。
研究如何扩展一台计算节点。
二 环境信息
（一）openstack 版本
ussuri 版
（二）硬件信息
| 品牌 | 型号 | 网卡 | IP | 配置 |
|---|---|---|---|---|
| 浪潮 | SA5212M5 | eth0 | 10.3.176.46 | 5118 * 2/240G * 2/4T * 2/32G * 24/T4 |
三 部署规划
| 机器名 | 网卡 | IP | VIP | 角色 |
|---|---|---|---|---|
| ussuritest004.cloud.chouniu.fun | eth0 | 10.3.176.46 | 计算节点 |
四 实施
(一)准备工作
4.1.1 安装操作系统
安装centos8.1.1911 版本，具体安装过程略。
4.1.2 插GPU卡
这台机器采购时不带gpu卡，断电装上自购的T4卡，然后通电重启，检查gpu卡是否识别。
[root@localhost ~]# lspci | grep -i nvidia
af:00.0 3D controller: NVIDIA Corporation TU104GL [Tesla T4] (rev a1)
[root@localhost ~]#
4.1.3 网卡修改为eth0
请参考centos7 centos8 修改网卡名称为eth0
4.1.4 配置dns
[dev@10-3-170-32 base]$ ansible-playbook modifydns.yml
4.1.4 配置内网 yum 源
[dev@10-3-170-32 base]$ ansible-playbook upateyum.yml
4.1.5配置内网时间服务器
[dev@10-3-170-32 base]$ ansible-playbook modifychronyclient.yml
4.1.6 关闭防火墙和selinux
dev@10-3-170-32 base]$ ansible-playbook closefirewalldandselinux.yml
4.1.7配置dns及修改机器名
首先在内网dns 系统里新增域名 ussuritest004.cloud.kxdigit.com 解析到10.3.176.46
然后在修改主机名
[root@localhost ~]# hostnamectl set-hostname ussuritest004.cloud.kxdigit.com
最后重启下机器
4.1.8 安装docker
安装docker且
新增 /etc/systemd/system/docker.service.d/kolla.conf
参考[openstack ussuri版本基于内网三台物理机集群kolla-ansible部署及与ceph 集群 集成]（https://www.cnblogs.com/weiwei2021/p/14200722.html）
4.1.9 禁用禁用libvirt
[root@ussuritest004 docker.service.d]# systemctl stop libvirtd.service && systemctl disable libvirtd.service && systemctl status libvirtd.service
Removed /etc/systemd/system/multi-user.target.wants/libvirtd.service.
Removed /etc/systemd/system/sockets.target.wants/virtlogd.socket.
Removed /etc/systemd/system/sockets.target.wants/virtlockd.socket.
● libvirtd.service - Virtualization daemon
Loaded: loaded (/usr/lib/systemd/system/libvirtd.service; disabled; vendor preset: enabled)
Active: inactive (dead) since Tue 2021-01-19 15:57:40 CST; 115ms ago
Docs: man:libvirtd(8)
https://libvirt.org
Main PID: 6712 (code=exited, status=0/SUCCESS)
CGroup: /system.slice/libvirtd.service
├─7047 /usr/sbin/dnsmasq --conf-file=/var/lib/libvirt/dnsmasq/default.conf --leasefile-ro --dhcp-script=/usr/libexec/libvirt_leases>
└─7048 /usr/sbin/dnsmasq --conf-file=/var/lib/libvirt/dnsmasq/default.conf --leasefile-ro --dhcp-script=/usr/libexec/libvirt_leases>
Jan 19 15:44:04 ussuritest004.cloud.kxdigit.com dnsmasq[7047]: compile time options: IPv6 GNU-getopt DBus no-i18n IDN2 DHCP DHCPv6 no-Lua TFTP>
Jan 19 15:44:04 ussuritest004.cloud.kxdigit.com dnsmasq-dhcp[7047]: DHCP, IP range 192.168.122.2 -- 192.168.122.254, lease time 1h
Jan 19 15:44:04 ussuritest004.cloud.kxdigit.com dnsmasq-dhcp[7047]: DHCP, sockets bound exclusively to interface virbr0
Jan 19 15:44:04 ussuritest004.cloud.kxdigit.com dnsmasq[7047]: reading /etc/resolv.conf
Jan 19 15:44:04 ussuritest004.cloud.kxdigit.com dnsmasq[7047]: using nameserver 10.3.157.201#53
Jan 19 15:44:04 ussuritest004.cloud.kxdigit.com dnsmasq[7047]: read /etc/hosts - 2 addresses
Jan 19 15:44:04 ussuritest004.cloud.kxdigit.com dnsmasq[7047]: read /var/lib/libvirt/dnsmasq/default.addnhosts - 0 addresses
Jan 19 15:44:04 ussuritest004.cloud.kxdigit.com dnsmasq-dhcp[7047]: read /var/lib/libvirt/dnsmasq/default.hostsfile
Jan 19 15:57:40 ussuritest004.cloud.kxdigit.com systemd[1]: Stopping Virtualization daemon...
Jan 19 15:57:40 ussuritest004.cloud.kxdigit.com systemd[1]: Stopped Virtualization daemon.
lines 1-20/20 (END)
4.1.10 激活第二张网卡
该网卡不需要配置地址，但需要激活
新增配置
[root@ussuritest004 network-scripts]# nmcli connection add type ethernet con-name eth0 ifname eth1
Warning: There is another connection with the name 'eth0'. Reference the connection by its uuid 'ecdc0f79-013a-4779-9721-ea1d04d11720'
Connection 'eth0' (ecdc0f79-013a-4779-9721-ea1d04d11720) successfully added.
[root@ussuritest004 network-scripts]#
修改配置
主要是修改
BOOTPROTO=none
ONBOOT=yes
TYPE=Ethernet
PROXY_METHOD=none
BROWSER_ONLY=no
BOOTPROTO=none
DEFROUTE=yes
IPV4_FAILURE_FATAL=no
IPV6INIT=yes
IPV6_AUTOCONF=yes
IPV6_DEFROUTE=yes
IPV6_FAILURE_FATAL=no
IPV6_ADDR_GEN_MODE=stable-privacy
NAME=eth1
UUID=71894ec7-bcec-439a-81a0-1bffab76a307
DEVICE=eth1
ONBOOT=yes
激活
root@ussuritest004 network-scripts]# nmcli c reload /etc/sysconfig/network-scripts/ifcfg-eth1
[root@ussuritest004 network-scripts]#
验证
eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet 10.3.176.46 netmask 255.255.252.0 broadcast 10.3.179.255
inet6 fe80::575d:fe74:b231:ad4b prefixlen 64 scopeid 0x20<link>
ether 6c:92:bf:a2:2b:d0 txqueuelen 1000 (Ethernet)
RX packets 61447 bytes 48065896 (45.8 MiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 4885 bytes 477622 (466.4 KiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
eth1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet6 fe80::10f3:1b61:a13f:19e8 prefixlen 64 scopeid 0x20<link>
ether 6c:92:bf:a2:2b:d1 txqueuelen 1000 (Ethernet)
RX packets 8787 bytes 607717 (593.4 KiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 173 bytes 26703 (26.0 KiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
eth2: flags=4099<UP,BROADCAST,MULTICAST> mtu 1500
ether 6c:92:bf:a2:2b:d2 txqueuelen 1000 (Ethernet)
RX packets 0 bytes 0 (0.0 B)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 0 bytes 0 (0.0 B)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
好像是 eth1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
RUNNING 代表网卡激活
4.1.11 部署节点免密登录到该计算节点
部署节点是10.3.176.2 在上面做免密登录到10.3.176.46
[dev@10-3-170-32 base]$ ssh root@10.3.176.2
Activate the web console with: systemctl enable --now cockpit.socket
Last login: Tue Jan 19 14:17:57 2021 from 10.3.170.32
[root@ussuritest001 ~]# ssh-copy-id root@10.3.176.46
/usr/bin/ssh-copy-id: INFO: Source of key(s) to be installed: "/root/.ssh/id_rsa.pub"
The authenticity of host '10.3.176.46 (10.3.176.46)' can't be established.
ECDSA key fingerprint is SHA256:RLr9ud5uf1PjzOA8wjJJA2X1mtbE072rhwFPaBprOVo.
Are you sure you want to continue connecting (yes/no/[fingerprint])? yes
/usr/bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter out any that are already installed
/usr/bin/ssh-copy-id: INFO: 1 key(s) remain to be installed -- if you are prompted now it is to install the new keys
root@10.3.176.46's password:
Number of key(s) added: 1
Now try logging into the machine, with: "ssh 'root@10.3.176.46'"
and check to make sure that only the key(s) you wanted were added.
[root@ussuritest001 ~]#
4.1.11 安装pip3
[root@ussuritest004 ~]# whereis pip3.6
pip3: /usr/bin/pip3 /usr/bin/pip3.6 /usr/share/man/man1/pip3.1.gz
[root@ussuritest004 ~]# ln -s /usr/bin/pip3.6 /usr/bin/pip
[root@ussuritest004 ~]# pip -V
pip 9.0.3 from /usr/lib/python3.6/site-packages (python 3.6)
[root@ussuritest004 ~]#
4.1.12 安装基础依赖
[root@ussuritest004 dependencies]# dnf install python3-devel libffi-devel gcc openssl-devel python3-libselinux git vim bash-completion net-tools
4.1.13 安装基础依赖docker-4.2.1-py2.py3-none-any.whl
该包不安装 做prechecks 会报错。
[root@ussuritest004 kollapip]# pip install docker-4.2.1-py2.py3-none-any.whl --no-index --find-links .
WARNING: Running pip install with root privileges is generally not a good idea. Try `pip install --user` instead.
Processing ./docker-4.2.1-py2.py3-none-any.whl
Requirement already satisfied: six>=1.4.0 in /usr/lib/python3.6/site-packages (from docker==4.2.1)
Requirement already satisfied: requests!=2.18.0,>=2.14.2 in /usr/lib/python3.6/site-packages (from docker==4.2.1)
Collecting websocket-client>=0.32.0 (from docker==4.2.1)
Requirement already satisfied: chardet<3.1.0,>=3.0.2 in /usr/lib/python3.6/site-packages (from requests!=2.18.0,>=2.14.2->docker==4.2.1)
Requirement already satisfied: idna<2.8,>=2.5 in /usr/lib/python3.6/site-packages (from requests!=2.18.0,>=2.14.2->docker==4.2.1)
Requirement already satisfied: urllib3<1.25,>=1.21.1 in /usr/lib/python3.6/site-packages (from requests!=2.18.0,>=2.14.2->docker==4.2.1)
Installing collected packages: websocket-client, docker
Successfully installed docker-4.2.1 websocket-client-0.57.0
[root@ussuritest004 kollapip]# pwd
/root/software/ussuri/kollapip
[root@ussuritest004 kollapip]#
（二）安装openstack
4.2.1 修改multinode
备份原文件，在计算配置下 新增ussuritest004.cloud.kxdigit.com
[root@ussuritest001 ansible]# cp multinode multinode.bak.20210119allinthree
[root@ussuritest001 ansible]# vim multinode
[root@ussuritest001 ansible]# pwd
/root/ansible
[root@ussuritest001 ansible]#
[compute]
#compute01
ussuritest001.cloud.kxdigit.com
ussuritest002.cloud.kxdigit.com
ussuritest003.cloud.kxdigit.com
ussuritest004.cloud.kxdigit.com
4.2.2 预检查
[root@ussuritest001 software]# kolla-ansible -v -i /root/ansible/multinode prechecks
检查结果
PLAY RECAP ************************************************************************************************************************************
localhost : ok=15 changed=0 unreachable=0 failed=0 skipped=11 rescued=0 ignored=0
ussuritest001.cloud.kxdigit.com : ok=57 changed=0 unreachable=0 failed=0 skipped=238 rescued=0 ignored=0
ussuritest002.cloud.kxdigit.com : ok=55 changed=0 unreachable=0 failed=0 skipped=226 rescued=0 ignored=0
ussuritest003.cloud.kxdigit.com : ok=55 changed=0 unreachable=0 failed=0 skipped=226 rescued=0 ignored=0
ussuritest004.cloud.kxdigit.com : ok=31 changed=0 unreachable=0 failed=0 skipped=30 rescued=0 ignored=0
4.2.3 拉取镜像
[root@ussuritest001 software]# kolla-ansible -v -i /root/ansible/multinode pull
Pulling Docker images : ansible-playbook -i /root/ansible/multinode -e @/etc/kolla/globals.yml -e @/etc/kolla/passwords.yml -e CONFIG_DIR=/etc/kolla -e kolla_action=pull /usr/local/share/kolla-ansible/ansible/site.yml --verbose
PLAY RECAP ************************************************************************************************************************************
localhost : ok=4 changed=0 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0
ussuritest001.cloud.kxdigit.com : ok=37 changed=0 unreachable=0 failed=0 skipped=97 rescued=0 ignored=0
ussuritest002.cloud.kxdigit.com : ok=37 changed=0 unreachable=0 failed=0 skipped=96 rescued=0 ignored=0
ussuritest003.cloud.kxdigit.com : ok=37 changed=0 unreachable=0 failed=0 skipped=96 rescued=0 ignored=0
ussuritest004.cloud.kxdigit.com : ok=15 changed=5 unreachable=0 failed=0 skipped=9 rescued=0 ignored=0
[root@ussuritest001 software]#
[root@ussuritest004 kollapip]# docker images
REPOSITORY TAG IMAGE ID CREATED SIZE
registry.kxdigit.com/kolla/centos-source-nova-compute ussuri b41fd82a0eed 3 weeks ago 2.02GB
registry.kxdigit.com/kolla/centos-source-nova-ssh ussuri a7107a92c35c 3 weeks ago 1.15GB
registry.kxdigit.com/kolla/centos-source-neutron-openvswitch-agent ussuri 43d1aa96a99e 3 weeks ago 1.06GB
registry.kxdigit.com/kolla/centos-source-openvswitch-vswitchd ussuri feeeaf740a8d 3 weeks ago 437MB
registry.kxdigit.com/kolla/centos-source-openvswitch-db-server ussuri 5d30076c87dc 3 weeks ago 437MB
registry.kxdigit.com/kolla/centos-source-kolla-toolbox ussuri 926bd6a806bc 3 weeks ago 811MB
registry.kxdigit.com/kolla/centos-source-nova-libvirt ussuri 7798eb2a9fa6 3 weeks ago 1.31GB
registry.kxdigit.com/kolla/centos-source-fluentd ussuri 6b07e84bdc3b 3 weeks ago 637MB
registry.kxdigit.com/kolla/centos-source-cron ussuri ff783b02c9ab 3 weeks ago 313MB
registry.kxdigit.com/kolla/centos-source-chrony ussuri 8e433346635d 3 weeks ago 313MB
[root@ussuritest004 kollapip]#
4.2.4 部署新增的计算节点
[root@ussuritest001 software]# kolla-ansible -v -i /root/ansible/multinode deploy
经观察这个命令是增量命令，如果该节点上容器已经运行，则不会重启。
4.2.5 验证
经验证该计算节点，已成功部署计算服务。
[root@ussuritest004 ~]# docker ps -a
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
8f08ca343b8d registry.kxdigit.com/kolla/centos-source-neutron-openvswitch-agent:ussuri "dumb-init --single-…" 15 hours ago Up 15 hours neutron_openvswitch_agent
9a054b4e9fc7 registry.kxdigit.com/kolla/centos-source-openvswitch-vswitchd:ussuri "dumb-init --single-…" 15 hours ago Up 15 hours openvswitch_vswitchd
85c86a4b0435 registry.kxdigit.com/kolla/centos-source-openvswitch-db-server:ussuri "dumb-init --single-…" 15 hours ago Up 15 hours openvswitch_db
387f692c5b05 registry.kxdigit.com/kolla/centos-source-nova-compute:ussuri "dumb-init --single-…" 15 hours ago Up 15 hours nova_compute
8531322a2b10 registry.kxdigit.com/kolla/centos-source-nova-libvirt:ussuri "dumb-init --single-…" 15 hours ago Up 15 hours nova_libvirt
38994a403da7 registry.kxdigit.com/kolla/centos-source-nova-ssh:ussuri "dumb-init --single-…" 15 hours ago Up 15 hours nova_ssh
a81e2e337eb5 registry.kxdigit.com/kolla/centos-source-chrony:ussuri "dumb-init --single-…" 15 hours ago Up 15 hours chrony
c0d69ed11378 registry.kxdigit.com/kolla/centos-source-cron:ussuri "dumb-init --single-…" 16 hours ago Up 16 hours cron
f6cb11bf7104 registry.kxdigit.com/kolla/centos-source-kolla-toolbox:ussuri "dumb-init --single-…" 16 hours ago Up 16 hours kolla_toolbox
45f27038c773 registry.kxdigit.com/kolla/centos-source-fluentd:ussuri "dumb-init --single-…" 16 hours ago Up 16 hours fluentd
[root@ussuritest004 ~]#
posted on 2021-01-19 14:46 weiwei2021 阅读(645) 评论(0) 编辑 收藏 举报
