ceph-ansible 安装 ceph octopus

一 摘要

本文主要是基于ceph-ansible 在centos8.1 上部署ceph octopus

二 环境信息

（一）操作系统版本

[root@ceph001 ~]# cat /etc/centos-release
CentOS Linux release 8.1.1911 (Core)
[root@ceph001 ~]#

（二） ansible 版本

三 安装说明

(一) 服务器说明

| 主机名 | IP |磁盘 |角色|
| ---- | ---- | ---- | ---- | ---- |
| ceph001 | 172.31.185.105 | 系统盘：/dev/vda 数据盘：/dev/vdb |ceph-ansible,monitor,mgr,mds,osd|
| ceph002 | 172.31.185.200 | 系统盘：/dev/vda 数据盘：/dev/vdb |monitor,mgr,mds,osd|
| ceph003 | 172.31.185.60 | 系统盘：/dev/vda 数据盘：/dev/vdb |monitor,mgr,mds,osd|

四 部署实施

（一）基础环境安装

以下步骤，除了特殊说明外，默认三台节点都需要实施。主要是安装docker、ansible(2.9)、关闭防火墙、selinux 等

4.1.1 设置节点名称

hostnamectl -set-hostname ceph001

4.1.2 配置hosts解析或者dns解析

如果本地有dns 系统，可以不设置hsots解析

4.1.2.1 配置hosts 解析

[root@ceph001 ~]# cp /etc/hosts /etc/hosts.bak.orig
[root@ceph001 ~]# vi /etc/hosts

新增

172.31.185.105 ceph001
172.31.185.200 ceph002
172.31.185.60 ceph003

4.1.2.2 dns 系统新增域名

若本地有dns 系统，可以将各主机hostname 和域名保持一致，然后只用在dns 系统里新增配置就可以了。

4.1.3 关闭防火墙

[root@ceph001 ~]# systemctl stop firewalld | systemctl disable firewalld | systemctl status firewalld
● firewalld.service - firewalld - dynamic firewall daemon
   Loaded: loaded (/usr/lib/systemd/system/firewalld.service; disabled; vendor preset: enabled)
   Active: inactive (dead)
     Docs: man:firewalld(1)
[root@ceph001 ~]#

4.1.4 关闭selinux

[root@ceph001 ~]# cp /etc/selinux/config /etc/selinux/config.bak.orig
[root@ceph001 ~]# sed -i 's/^SELINUX=.*/SELINUX=disabled/' /etc/selinux/config
[root@ceph001 ~]# cat /etc/selinux/config

# This file controls the state of SELinux on the system.
# SELINUX= can take one of these three values:
#     enforcing - SELinux security policy is enforced.
#     permissive - SELinux prints warnings instead of enforcing.
#     disabled - No SELinux policy is loaded.
SELINUX=disabled
# SELINUXTYPE= can take one of these three values:
#     targeted - Targeted processes are protected,
#     minimum - Modification of targeted policy. Only selected processes are protected.
#     mls - Multi Level Security protection.
SELINUXTYPE=targeted


[root@ceph001 ~]# ^C
[root@ceph001 ~]# reboot

4.1.6 配置免密登录

在ceph-ansible 机器上配置免密登录，可以免密登录所有节点服务器（含自己），我这里是ceph001.

[root@ceph001 ~]# ssh-keygen
Generating public/private rsa key pair.
Enter file in which to save the key (/root/.ssh/id_rsa):
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in /root/.ssh/id_rsa.
Your public key has been saved in /root/.ssh/id_rsa.pub.
The key fingerprint is:
SHA256:Vj9sSdq8/mALseKiUsEVBt7n0TSIRXv95TdguVqhsww root@ceph001
The key's randomart image is:
+---[RSA 3072]----+
|    ..o=o.o      |
|   . oo .+ o  .  |
|   .... + + o=  .|
|    o  o + Boo+o |
|     .  SEooOo..o|
|    .  .  o+=o  o|
|   .    . o++    |
|  .   .. . + o   |
|   ... ..   o..  |
+----[SHA256]-----+
[root@ceph001 ~]# ssh-copy-id root@172.31.185.105
/usr/bin/ssh-copy-id: INFO: Source of key(s) to be installed: "/root/.ssh/id_rsa.pub"
The authenticity of host '172.31.185.105 (172.31.185.105)' can't be established.
ECDSA key fingerprint is SHA256:ES6ytBX1siYV4WMG2CF3/21VKaDd5y27lbWQggeqRWM.
Are you sure you want to continue connecting (yes/no/[fingerprint])? yes
/usr/bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter out any that are already installed
/usr/bin/ssh-copy-id: INFO: 1 key(s) remain to be installed -- if you are prompted now it is to install the new keys
root@172.31.185.105's password:
Permission denied, please try again.
root@172.31.185.105's password:

Number of key(s) added: 1

Now try logging into the machine, with:   "ssh 'root@172.31.185.105'"
and check to make sure that only the key(s) you wanted were added.

[root@ceph001 ~]# ssh-copy-id root@172.31.185.60
/usr/bin/ssh-copy-id: INFO: Source of key(s) to be installed: "/root/.ssh/id_rsa.pub"
The authenticity of host '172.31.185.60 (172.31.185.60)' can't be established.
ECDSA key fingerprint is SHA256:ES6ytBX1siYV4WMG2CF3/21VKaDd5y27lbWQggeqRWM.
Are you sure you want to continue connecting (yes/no/[fingerprint])? yes
/usr/bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter out any that are already installed
/usr/bin/ssh-copy-id: INFO: 1 key(s) remain to be installed -- if you are prompted now it is to install the new keys
root@172.31.185.60's password:

Number of key(s) added: 1

Now try logging into the machine, with:   "ssh 'root@172.31.185.60'"
and check to make sure that only the key(s) you wanted were added.

[root@ceph001 ~]# ssh-copy-id root@172.31.185.200



[root@ceph001 ~]#

4.1.7 安装docker

配置docker yum 源，安装docker

4.1.7.1 配置docker yum源

[root@ceph002 ~]# yum install yum-utils
[root@ceph002 ~]# yum-config-manager --add-repo http://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo

centos8 自带的containerd.io 与docker-ce-19.03 不兼容，需要升级
dnf download https://download.docker.com/linux/centos/7/x86_64/stable/Packages/containerd.io-1.2.6-3.3.el7.x86_64.rpm

[root@ceph001 packages]# dnf download  https://download.docker.com/linux/centos/7/x86_64/stable/Packages/containerd.io-1.2.6-3.3.el7.x86_64.rpm
Last metadata expiration check: 0:02:06 ago on Tue 17 Nov 2020 03:23:34 PM CST.
containerd.io-1.2.6-3.3.el7.x86_64.rpm                                                                          17 kB/s |  26 MB     25:20
[root@ceph001 packages]# ll

[root@ceph001 packages]# cp containerd.io-1.2.6-3.3.el7.x86_64.rpm  /root/software/
[root@ceph001 packages]# ll
total 26484
-rw-r--r-- 1 root root 27119348 Nov 17 15:51 containerd.io-1.2.6-3.3.el7.x86_64.rpm
[root@ceph001 packages]# cd ~
[root@ceph001 ~]# cd software/
[root@ceph001 software]# ll
total 27216
-rw-r--r-- 1 root root   746011 Nov 17 10:35 ceph-ansible-5.0.2.zip
-rw-r--r-- 1 root root 27119348 Nov 17 15:52 containerd.io-1.2.6-3.3.el7.x86_64.rpm
[root@ceph001 software]# yum install /root/software/containerd.io-1.2.6-3.3.el7.x86_64.rpm

安装docker

[root@ceph001 software]# yum install docker-ce

设置开机启动，并启动docke

[root@ceph001 software]# systemctl enable --now docker
Created symlink /etc/systemd/system/multi-user.target.wants/docker.service → /usr/lib/systemd/system/docker.service.
[root@ceph001 software]# docker info
Client:
 Debug Mode: false

Server:
 Containers: 0
  Running: 0
  Paused: 0
  Stopped: 0
 Images: 0
 Server Version: 19.03.13

4.1.8 安装ansible(仅ceph-ansible 节点安装)

此处需要安装ansible2.9

参考ceph-ansible 官网 版本说明

stable-3.0 Supports Ceph versions jewel and luminous. This branch requires Ansible version 2.4.
stable-3.1 Supports Ceph versions luminous and mimic. This branch requires Ansible version 2.4.
stable-3.2 Supports Ceph versions luminous and mimic. This branch requires Ansible version 2.6.
stable-4.0 Supports Ceph version nautilus. This branch requires Ansible version 2.8.
stable-5.0 Supports Ceph version octopus. This branch requires Ansible version 2.9.
master Supports the master branch of Ceph. This branch requires Ansible version 2.9.

4.1.8.1 yum 安装ansible 2.9

[root@ceph001 ~]# yum list | grep ansible
ansible.noarch                                       2.9.15-1.el8                                     
ansible-doc.noarch                                   2.9.15-1.el8                                      
ansible-freeipa.noarch                               0.1.8-3.el8                                      
ansible-freeipa.noarch                               0.1.8-3.el8                                      
centos-release-ansible-29.noarch                     1-2.el8                                           
vim-ansible.noarch                                   3.0-1.el8                                        
[root@ceph001 ~]# yum install ansible

4.1.8.2 配置ansible

备份/etc/ansible/hosts 文件，然后新添如下配置

[root@ceph001 ansible]# cp hosts hosts.bak.orig
[root@ceph001 ansible]# vi hosts
[root@ceph001 ansible]# ping ceph003
PING ceph003 (172.31.185.60) 56(84) bytes of data.
64 bytes from ceph003 (172.31.185.60): icmp_seq=1 ttl=64 time=6.17 ms
64 bytes from ceph003 (172.31.185.60): icmp_seq=2 ttl=64 time=2.69 ms
^C
--- ceph003 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 4ms
rtt min/avg/max/mdev = 2.688/4.429/6.171/1.742 ms
[root@ceph001 group_vars]# cat /etc/ansible/hosts
[mons]
ceph001
ceph002
ceph003

[mgrs]
ceph001
ceph002
ceph003

[osds]
ceph001
ceph002
ceph003

[rgws]
ceph001
ceph002
ceph003


[clients]
ceph001

[grafana-server]
ceph001
[root@ceph001 group_vars]#

4.1.9 ceph-ansible 安装 （仅ceph-ansible 节点安装）

在gitlab 上下载stable 版本ceph-ansible代码，此处使用的是ceph-ansible-5.0.2 版本

4.1.9.1 下载代码

下载地址：https://github.com/ceph/ceph-ansible/releases
下载好后，上传到服务器

4.1.9.2 修改配置文件

4.1.9.2.1

[root@ceph001 ceph-ansible-5.0.2]# cp /root/software/ceph-ansible-5.0.2/site.yml.sample /root/software/ceph-ansible-5.0.2/site.yml

4.1.9.2.2

[root@ceph001 group_vars]# cat all.yml
cluster: ceph
configure_firewall: False
ceph_origin: repository
ceph_repository: community
ceph_stable_release: octopus
ceph_mirror: http://mirrors.aliyun.com/ceph
ceph_stable_key: http://mirrors.aliyun.com/ceph/keys/release.asc
ceph_stable_repo: "{{ ceph_mirror }}/rpm-{{ ceph_stable_release }}"
public_network: "172.31.185.0/24"
cluster_network: "172.31.185.0/24"
monitor_interface: ens3
cephx: true

devices:
  - '/dev/sdb'
ceph_conf_overrides:
   mon:
     mon_allow_pool_delete: true
[root@ceph001 group_vars]#

（二）安装部署

4.2.1

安装ceph 集群

4.2.1.1 执行安装命令

[root@ceph001 ceph-ansible-5.0.2]# ansible-playbook site.yml -i /etc/ansible/hosts
[WARNING]: Could not match supplied host pattern, ignoring: mdss
[WARNING]: Could not match supplied host pattern, ignoring: nfss
[WARNING]: Could not match supplied host pattern, ignoring: rbdmirrors
[WARNING]: Could not match supplied host pattern, ignoring: iscsigws
[WARNING]: Could not match supplied host pattern, ignoring: iscsi-gws
[WARNING]: Could not match supplied host pattern, ignoring: grafana-server
[WARNING]: Could not match supplied host pattern, ignoring: rgwloadbalancers

PLAY [mons,osds,mdss,rgws,nfss,rbdmirrors,clients,mgrs,iscsigws,iscsi-gws,grafana-server,rgwloadbalancers] ************************************

TASK [check for python] ***********************************************************************************************************************
Wednesday 18 November 2020  14:18:01 +0800 (0:00:00.040)       0:00:00.040 ****
fatal: [ceph001]: FAILED! =>
  msg: The ips_in_ranges filter requires python's netaddr be installed on the ansible controller.
fatal: [ceph002]: FAILED! =>
  msg: The ips_in_ranges filter requires python's netaddr be installed on the ansible controller.
fatal: [ceph003]: FAILED! =>
  msg: The ips_in_ranges filter requires python's netaddr be installed on the ansible controller.

安装netaddr

[root@ceph001 software]# mkdir pipnetaddr
[root@ceph001 software]# cd pipnetaddr/
[root@ceph001 pipnetaddr]# pwd
/root/software/pipnetaddr
[root@ceph001 pipnetaddr]# pip download -i https://pypi.tuna.tsinghua.edu.cn/simple -d /root/software/pipnetaddr/ netaddr
Collecting netaddr


[root@ceph001 pipnetaddr]# pwd
/root/software/pipnetaddr
[root@ceph001 pipnetaddr]# pip install --no-index --find-links=/root/software/pipnetaddr netaddr-0.8.0-py2.py3-none-any.whl
WARNING: Running pip install with root privileges is generally not a good idea. Try `pip install --user` instead.
Processing ./netaddr-0.8.0-py2.py3-none-any.whl
Collecting importlib-resources; python_version < "3.7" (from netaddr==0.8.0)
Collecting zipp>=0.4; python_version < "3.8" (from importlib-resources; python_version < "3.7"->netaddr==0.8.0)
Installing collected packages: zipp, importlib-resources, netaddr
Successfully installed importlib-resources-3.3.0 netaddr-0.8.0 zipp-3.4.0
[root@ceph001 pipnetaddr]#

ansible-playbook site.yml -i /etc/ansible/hosts

############下载包相关的源配置###############
ceph_origin: repository
ceph_repository: community
ceph_mirror: https://mirrors.aliyun.com/ceph/
ceph_stable_key: "{{ ceph_mirror }}/keys/release.asc"
ceph_stable_release: octopus
ceph_stable_repo: "{{ ceph_mirror }}/debian-{{ ceph_stable_release }}"
ceph_stable_distro_source: bionic

cephx: "true"

############非常重要参数###################
public_network: 172.31.185.0/24
cluster_network: 172.31.185.0/24

#mon_host: 192.168.26.120
#mon_initial_members: ceph001,ceph002,ceph003
monitor_interface: ens3
#########################################

rbd_cache: "true"
rbd_cache_writethrough_until_flush: "true"
rbd_concurrent_management_ops: 21
rbd_client_directories: true

############创建osd必要的参数###############
osd_objectstore: bluestore
devices:
  - '/dev/vdb'
osd_scenario: lvm
#########################################

mds_max_mds: 1
radosgw_frontend_type: beast
radosgw_thread_pool_size: 512
radosgw_interface: "{{ monitor_interface }}"
email_address: xxx@xxx.com
dashboard_enabled: False
dashboard_protocol: http
dashboard_port: 8443
dashboard_admin_user: admin
dashboard_admin_password: admin@123!
grafana_admin_user: admin
grafana_admin_password: admin
grafana_uid: 472
grafana_datasource: Dashboard
grafana_dashboard_version: octopus
grafana_port: 3000
grafana_allow_embedding: True