ceph-ansible 安装 ceph octopus
一 摘要
本文主要是基于ceph-ansible 在centos8.1 上部署ceph octopus
二 环境信息
（一）操作系统版本
[root@ceph001 ~]# cat /etc/centos-release
CentOS Linux release 8.1.1911 (Core)
[root@ceph001 ~]#
（二） ansible 版本
三 安装说明
(一) 服务器说明
| 主机名 | IP |磁盘 |角色|
| ---- | ---- | ---- | ---- | ---- |
| ceph001 | 172.31.185.105 | 系统盘：/dev/vda 数据盘：/dev/vdb |ceph-ansible,monitor,mgr,mds,osd|
| ceph002 | 172.31.185.200 | 系统盘：/dev/vda 数据盘：/dev/vdb |monitor,mgr,mds,osd|
| ceph003 | 172.31.185.60 | 系统盘：/dev/vda 数据盘：/dev/vdb |monitor,mgr,mds,osd|
四 部署实施
（一）基础环境安装
以下步骤，除了特殊说明外，默认三台节点都需要实施。主要是安装docker、ansible(2.9)、关闭防火墙、selinux 等
4.1.1 设置节点名称
hostnamectl -set-hostname ceph001
4.1.2 配置hosts解析或者dns解析
如果本地有dns 系统，可以不设置hsots解析
4.1.2.1 配置hosts 解析
[root@ceph001 ~]# cp /etc/hosts /etc/hosts.bak.orig
[root@ceph001 ~]# vi /etc/hosts
新增
172.31.185.105 ceph001
172.31.185.200 ceph002
172.31.185.60 ceph003
4.1.2.2 dns 系统新增域名
若本地有dns 系统，可以将各主机hostname 和域名保持一致，然后只用在dns 系统里新增配置就可以了。
4.1.3 关闭防火墙
[root@ceph001 ~]# systemctl stop firewalld | systemctl disable firewalld | systemctl status firewalld
● firewalld.service - firewalld - dynamic firewall daemon
Loaded: loaded (/usr/lib/systemd/system/firewalld.service; disabled; vendor preset: enabled)
Active: inactive (dead)
Docs: man:firewalld(1)
[root@ceph001 ~]#
4.1.4 关闭selinux
[root@ceph001 ~]# cp /etc/selinux/config /etc/selinux/config.bak.orig
[root@ceph001 ~]# sed -i 's/^SELINUX=.*/SELINUX=disabled/' /etc/selinux/config
[root@ceph001 ~]# cat /etc/selinux/config
# This file controls the state of SELinux on the system.
# SELINUX= can take one of these three values:
# enforcing - SELinux security policy is enforced.
# permissive - SELinux prints warnings instead of enforcing.
# disabled - No SELinux policy is loaded.
SELINUX=disabled
# SELINUXTYPE= can take one of these three values:
# targeted - Targeted processes are protected,
# minimum - Modification of targeted policy. Only selected processes are protected.
# mls - Multi Level Security protection.
SELINUXTYPE=targeted
[root@ceph001 ~]# ^C
[root@ceph001 ~]# reboot
4.1.6 配置免密登录
在ceph-ansible 机器上配置免密登录，可以免密登录所有节点服务器（含自己），我这里是ceph001.
[root@ceph001 ~]# ssh-keygen
Generating public/private rsa key pair.
Enter file in which to save the key (/root/.ssh/id_rsa):
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in /root/.ssh/id_rsa.
Your public key has been saved in /root/.ssh/id_rsa.pub.
The key fingerprint is:
SHA256:Vj9sSdq8/mALseKiUsEVBt7n0TSIRXv95TdguVqhsww root@ceph001
The key's randomart image is:
+---[RSA 3072]----+
| ..o=o.o |
| . oo .+ o . |
| .... + + o= .|
| o o + Boo+o |
| . SEooOo..o|
| . . o+=o o|
| . . o++ |
| . .. . + o |
| ... .. o.. |
+----[SHA256]-----+
[root@ceph001 ~]# ssh-copy-id root@172.31.185.105
/usr/bin/ssh-copy-id: INFO: Source of key(s) to be installed: "/root/.ssh/id_rsa.pub"
The authenticity of host '172.31.185.105 (172.31.185.105)' can't be established.
ECDSA key fingerprint is SHA256:ES6ytBX1siYV4WMG2CF3/21VKaDd5y27lbWQggeqRWM.
Are you sure you want to continue connecting (yes/no/[fingerprint])? yes
/usr/bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter out any that are already installed
/usr/bin/ssh-copy-id: INFO: 1 key(s) remain to be installed -- if you are prompted now it is to install the new keys
root@172.31.185.105's password:
Permission denied, please try again.
root@172.31.185.105's password:
Number of key(s) added: 1
Now try logging into the machine, with: "ssh 'root@172.31.185.105'"
and check to make sure that only the key(s) you wanted were added.
[root@ceph001 ~]# ssh-copy-id root@172.31.185.60
/usr/bin/ssh-copy-id: INFO: Source of key(s) to be installed: "/root/.ssh/id_rsa.pub"
The authenticity of host '172.31.185.60 (172.31.185.60)' can't be established.
ECDSA key fingerprint is SHA256:ES6ytBX1siYV4WMG2CF3/21VKaDd5y27lbWQggeqRWM.
Are you sure you want to continue connecting (yes/no/[fingerprint])? yes
/usr/bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter out any that are already installed
/usr/bin/ssh-copy-id: INFO: 1 key(s) remain to be installed -- if you are prompted now it is to install the new keys
root@172.31.185.60's password:
Number of key(s) added: 1
Now try logging into the machine, with: "ssh 'root@172.31.185.60'"
and check to make sure that only the key(s) you wanted were added.
[root@ceph001 ~]# ssh-copy-id root@172.31.185.200
[root@ceph001 ~]#
4.1.7 安装docker
配置docker yum 源，安装docker
4.1.7.1 配置docker yum源
[root@ceph002 ~]# yum install yum-utils
[root@ceph002 ~]# yum-config-manager --add-repo http://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo
centos8 自带的containerd.io 与docker-ce-19.03 不兼容，需要升级
dnf download https://download.docker.com/linux/centos/7/x86_64/stable/Packages/containerd.io-1.2.6-3.3.el7.x86_64.rpm
[root@ceph001 packages]# dnf download https://download.docker.com/linux/centos/7/x86_64/stable/Packages/containerd.io-1.2.6-3.3.el7.x86_64.rpm
Last metadata expiration check: 0:02:06 ago on Tue 17 Nov 2020 03:23:34 PM CST.
containerd.io-1.2.6-3.3.el7.x86_64.rpm 17 kB/s | 26 MB 25:20
[root@ceph001 packages]# ll
[root@ceph001 packages]# cp containerd.io-1.2.6-3.3.el7.x86_64.rpm /root/software/
[root@ceph001 packages]# ll
total 26484
-rw-r--r-- 1 root root 27119348 Nov 17 15:51 containerd.io-1.2.6-3.3.el7.x86_64.rpm
[root@ceph001 packages]# cd ~
[root@ceph001 ~]# cd software/
[root@ceph001 software]# ll
total 27216
-rw-r--r-- 1 root root 746011 Nov 17 10:35 ceph-ansible-5.0.2.zip
-rw-r--r-- 1 root root 27119348 Nov 17 15:52 containerd.io-1.2.6-3.3.el7.x86_64.rpm
[root@ceph001 software]# yum install /root/software/containerd.io-1.2.6-3.3.el7.x86_64.rpm
安装docker
[root@ceph001 software]# yum install docker-ce
设置开机启动，并启动docke
[root@ceph001 software]# systemctl enable --now docker
Created symlink /etc/systemd/system/multi-user.target.wants/docker.service → /usr/lib/systemd/system/docker.service.
[root@ceph001 software]# docker info
Client:
Debug Mode: false
Server:
Containers: 0
Running: 0
Paused: 0
Stopped: 0
Images: 0
Server Version: 19.03.13
4.1.8 安装ansible(仅ceph-ansible 节点安装)
此处需要安装ansible2.9
stable-3.0 Supports Ceph versions jewel and luminous. This branch requires Ansible version 2.4.
stable-3.1 Supports Ceph versions luminous and mimic. This branch requires Ansible version 2.4.
stable-3.2 Supports Ceph versions luminous and mimic. This branch requires Ansible version 2.6.
stable-4.0 Supports Ceph version nautilus. This branch requires Ansible version 2.8.
stable-5.0 Supports Ceph version octopus. This branch requires Ansible version 2.9.
master Supports the master branch of Ceph. This branch requires Ansible version 2.9.
4.1.8.1 yum 安装ansible 2.9
[root@ceph001 ~]# yum list | grep ansible
ansible.noarch 2.9.15-1.el8
ansible-doc.noarch 2.9.15-1.el8
ansible-freeipa.noarch 0.1.8-3.el8
ansible-freeipa.noarch 0.1.8-3.el8
centos-release-ansible-29.noarch 1-2.el8
vim-ansible.noarch 3.0-1.el8
[root@ceph001 ~]# yum install ansible
4.1.8.2 配置ansible
备份/etc/ansible/hosts 文件，然后新添如下配置
[root@ceph001 ansible]# cp hosts hosts.bak.orig
[root@ceph001 ansible]# vi hosts
[root@ceph001 ansible]# ping ceph003
PING ceph003 (172.31.185.60) 56(84) bytes of data.
64 bytes from ceph003 (172.31.185.60): icmp_seq=1 ttl=64 time=6.17 ms
64 bytes from ceph003 (172.31.185.60): icmp_seq=2 ttl=64 time=2.69 ms
^C
--- ceph003 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 4ms
rtt min/avg/max/mdev = 2.688/4.429/6.171/1.742 ms
[root@ceph001 group_vars]# cat /etc/ansible/hosts
[mons]
ceph001
ceph002
ceph003
[mgrs]
ceph001
ceph002
ceph003
[osds]
ceph001
ceph002
ceph003
[rgws]
ceph001
ceph002
ceph003
[clients]
ceph001
[grafana-server]
ceph001
[root@ceph001 group_vars]#
4.1.9 ceph-ansible 安装 （仅ceph-ansible 节点安装）
在gitlab 上下载stable 版本ceph-ansible代码，此处使用的是ceph-ansible-5.0.2 版本
4.1.9.1 下载代码
下载地址：https://github.com/ceph/ceph-ansible/releases
下载好后，上传到服务器
4.1.9.2 修改配置文件
4.1.9.2.1
[root@ceph001 ceph-ansible-5.0.2]# cp /root/software/ceph-ansible-5.0.2/site.yml.sample /root/software/ceph-ansible-5.0.2/site.yml
4.1.9.2.2
[root@ceph001 group_vars]# cat all.yml
cluster: ceph
configure_firewall: False
ceph_origin: repository
ceph_repository: community
ceph_stable_release: octopus
ceph_mirror: http://mirrors.aliyun.com/ceph
ceph_stable_key: http://mirrors.aliyun.com/ceph/keys/release.asc
ceph_stable_repo: "{{ ceph_mirror }}/rpm-{{ ceph_stable_release }}"
public_network: "172.31.185.0/24"
cluster_network: "172.31.185.0/24"
monitor_interface: ens3
cephx: true
devices:
- '/dev/sdb'
ceph_conf_overrides:
mon:
mon_allow_pool_delete: true
[root@ceph001 group_vars]#
（二）安装部署
4.2.1
安装ceph 集群
4.2.1.1 执行安装命令
[root@ceph001 ceph-ansible-5.0.2]# ansible-playbook site.yml -i /etc/ansible/hosts
[WARNING]: Could not match supplied host pattern, ignoring: mdss
[WARNING]: Could not match supplied host pattern, ignoring: nfss
[WARNING]: Could not match supplied host pattern, ignoring: rbdmirrors
[WARNING]: Could not match supplied host pattern, ignoring: iscsigws
[WARNING]: Could not match supplied host pattern, ignoring: iscsi-gws
[WARNING]: Could not match supplied host pattern, ignoring: grafana-server
[WARNING]: Could not match supplied host pattern, ignoring: rgwloadbalancers
PLAY [mons,osds,mdss,rgws,nfss,rbdmirrors,clients,mgrs,iscsigws,iscsi-gws,grafana-server,rgwloadbalancers] ************************************
TASK [check for python] ***********************************************************************************************************************
Wednesday 18 November 2020 14:18:01 +0800 (0:00:00.040) 0:00:00.040 ****
fatal: [ceph001]: FAILED! =>
msg: The ips_in_ranges filter requires python's netaddr be installed on the ansible controller.
fatal: [ceph002]: FAILED! =>
msg: The ips_in_ranges filter requires python's netaddr be installed on the ansible controller.
fatal: [ceph003]: FAILED! =>
msg: The ips_in_ranges filter requires python's netaddr be installed on the ansible controller.
安装netaddr
[root@ceph001 software]# mkdir pipnetaddr
[root@ceph001 software]# cd pipnetaddr/
[root@ceph001 pipnetaddr]# pwd
/root/software/pipnetaddr
[root@ceph001 pipnetaddr]# pip download -i https://pypi.tuna.tsinghua.edu.cn/simple -d /root/software/pipnetaddr/ netaddr
Collecting netaddr
[root@ceph001 pipnetaddr]# pwd
/root/software/pipnetaddr
[root@ceph001 pipnetaddr]# pip install --no-index --find-links=/root/software/pipnetaddr netaddr-0.8.0-py2.py3-none-any.whl
WARNING: Running pip install with root privileges is generally not a good idea. Try `pip install --user` instead.
Processing ./netaddr-0.8.0-py2.py3-none-any.whl
Collecting importlib-resources; python_version < "3.7" (from netaddr==0.8.0)
Collecting zipp>=0.4; python_version < "3.8" (from importlib-resources; python_version < "3.7"->netaddr==0.8.0)
Installing collected packages: zipp, importlib-resources, netaddr
Successfully installed importlib-resources-3.3.0 netaddr-0.8.0 zipp-3.4.0
[root@ceph001 pipnetaddr]#
ansible-playbook site.yml -i /etc/ansible/hosts
############下载包相关的源配置###############
ceph_origin: repository
ceph_repository: community
ceph_mirror: https://mirrors.aliyun.com/ceph/
ceph_stable_key: "{{ ceph_mirror }}/keys/release.asc"
ceph_stable_release: octopus
ceph_stable_repo: "{{ ceph_mirror }}/debian-{{ ceph_stable_release }}"
ceph_stable_distro_source: bionic
cephx: "true"
############非常重要参数###################
public_network: 172.31.185.0/24
cluster_network: 172.31.185.0/24
#mon_host: 192.168.26.120
#mon_initial_members: ceph001,ceph002,ceph003
monitor_interface: ens3
#########################################
rbd_cache: "true"
rbd_cache_writethrough_until_flush: "true"
rbd_concurrent_management_ops: 21
rbd_client_directories: true
############创建osd必要的参数###############
osd_objectstore: bluestore
devices:
- '/dev/vdb'
osd_scenario: lvm
#########################################
mds_max_mds: 1
radosgw_frontend_type: beast
radosgw_thread_pool_size: 512
radosgw_interface: "{{ monitor_interface }}"
email_address: xxx@xxx.com
dashboard_enabled: False
dashboard_protocol: http
dashboard_port: 8443
dashboard_admin_user: admin
dashboard_admin_password: admin@123!
grafana_admin_user: admin
grafana_admin_password: admin
grafana_uid: 472
grafana_datasource: Dashboard
grafana_dashboard_version: octopus
grafana_port: 3000
grafana_allow_embedding: True
posted on 2020-11-30 08:31 weiwei2021 阅读(2336) 评论(0) 编辑 收藏 举报
