ceph 纵向扩容 nautilus版本
零 修订记录
| 序号 | 修订内容 | 修订时间 |
|---|---|---|
| 1 | 新增 | 2021/2/6 |
一 摘要
前文linux 基于三台物理机安装ceph nautilus 介绍了 ceph 在centos 7.6 上的安装。
ceph nautilus 横向扩容 介绍了ceph 横向扩容，本文在此两篇基础上 在介绍纵向扩容
二 环境信息
| 主机名 | 状态|IP |磁盘 |角色|
| ---- | ---- | ---- | ---- | ---- | ---- |---|
| cephtest001.ceph.kxdigit.com |已完成|10.3.176.10 | 系统盘：/dev/sda 数据盘：/dev/sdb /dev/sdc /dev/sdd |ceph-deploy,monitor,mgr,mds,osd|
| cephtest002.ceph.kxdigit.com |已完成 |10.3.176.16 | 系统盘：/dev/sda 数据盘：/dev/sdb /dev/sdc /dev/sdd /dev/sde /dev/sdf |monitor,mgr,mds,osd|
| cephtest003.ceph.kxdigit.com |已完成|10.3.176.44 | 系统盘：/dev/sda 数据盘：/dev/sdb /dev/sdc /dev/sdd /dev/sde /dev/sdf /dev/sdg |monitor,mgr,mds,osd|
| cephtest004.ceph.kxdigit.com |本次纵向扩容一张硬盘|10.3.176.36 | 系统盘：/dev/sda 数据盘：/dev/sdb /dev/sdc /dev/sdd /dev/sde /dev/sdf(待纵向扩容) |monitor,mgr,mds,osd|
三 实施
(一)扩容前ceph 信息
3.1.1 ceph 健康信息
[root@cephtest001 ~]# ceph -s
cluster:
id: 6cd05235-66dd-4929-b697-1562d308d5c3
health: HEALTH_WARN
1 pools have many more objects per pg than average
services:
mon: 4 daemons, quorum cephtest001,cephtest002,cephtest003,cephtest004 (age 47h)
mgr: cephtest001(active, since 6w), standbys: cephtest002, cephtest003, cephtest004
osd: 18 osds: 18 up (since 47h), 18 in (since 47h)
rgw: 1 daemon active (cephtest004)
task status:
data:
pools: 8 pools, 400 pgs
objects: 24.07k objects, 167 GiB
usage: 520 GiB used, 66 TiB / 67 TiB avail
pgs: 400 active+clean
io:
client: 147 KiB/s rd, 21 KiB/s wr, 166 op/s rd, 3 op/s wr
[root@cephtest001 ~]#
3.1.2 ceph osd 信息
可见 cephtest004 目前有4个osd,目前计划在扩容一张
[root@cephtest001 ~]# ceph osd tree
ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF
-1 66.93628 root default
-3 3.26669 host cephtest001
0 hdd 1.08890 osd.0 up 1.00000 1.00000
1 hdd 1.08890 osd.1 up 1.00000 1.00000
2 hdd 1.08890 osd.2 up 1.00000 1.00000
-5 5.45547 host cephtest002
3 hdd 1.09109 osd.3 up 1.00000 1.00000
4 hdd 1.09109 osd.4 up 1.00000 1.00000
5 hdd 1.09109 osd.5 up 1.00000 1.00000
6 hdd 1.09109 osd.6 up 1.00000 1.00000
7 hdd 1.09109 osd.7 up 1.00000 1.00000
-7 43.66132 host cephtest003
8 hdd 7.27689 osd.8 up 1.00000 1.00000
9 hdd 7.27689 osd.9 up 1.00000 1.00000
10 hdd 7.27689 osd.10 up 1.00000 1.00000
11 hdd 7.27689 osd.11 up 1.00000 1.00000
12 hdd 7.27689 osd.12 up 1.00000 1.00000
13 hdd 7.27689 osd.13 up 1.00000 1.00000
-9 14.55280 host cephtest004
14 hdd 3.63820 osd.14 up 1.00000 1.00000
15 hdd 3.63820 osd.15 up 1.00000 1.00000
16 hdd 3.63820 osd.16 up 1.00000 1.00000
17 hdd 3.63820 osd.17 up 1.00000 1.00000
[root@cephtest001 ~]#
(二)扩容前物理机准备
3.2.1 待扩容物理机准备工作
关闭该物理机，插入硬盘，并对该硬盘做raid0
cephtest004.ceph.kxdigit.com
可见该机器，已新添了一张硬盘，待扩容卷/dev/sdf
[root@cephtest004 ~]# lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
sda 8:0 0 1.8T 0 disk
├─sda1 8:1 0 1G 0 part /boot
└─sda2 8:2 0 1.8T 0 part
├─centos-root 253:0 0 1.7T 0 lvm /
├─centos-swap 253:1 0 4G 0 lvm [SWAP]
└─centos-home 253:5 0 100G 0 lvm /home
sdb 8:16 0 3.7T 0 disk
└─ceph--b1011de5--e580--4eb0--b477--70a7a56e6c76-osd--block--f43f5d64--bb23--4a4b--aaa2--1138bca784ea 253:4 0 3.7T 0 lvm
sdc 8:32 0 3.7T 0 disk
└─ceph--42d1e684--fadc--46e8--8816--2aedaeed1725-osd--block--5e1cd70d--470b--4821--9f7f--195a65e95c86 253:6 0 3.7T 0 lvm
sdd 8:48 0 3.7T 0 disk
└─ceph--74ce24e0--92a0--4913--8127--03c6565574ee-osd--block--f1e446eb--557a--43c0--b82d--f4503dd93e18 253:3 0 3.7T 0 lvm
sde 8:64 0 3.7T 0 disk
└─ceph--34f94761--91a8--4f5a--8d29--8829fe5561de-osd--block--0d229ba1--f7fb--4660--827a--571cc41e20cc 253:2 0 3.7T 0 lvm
sdf 8:80 0 3.7T 0 disk
[root@cephtest004 ~]#
需在部署节点执行
[cephadmin@cephtest001 cephcluster]$ ceph-deploy disk list cephtest004
[ceph_deploy.conf][DEBUG ] found configuration file at: /home/cephadmin/.cephdeploy.conf
[ceph_deploy.cli][INFO ] Invoked (2.0.1): /bin/ceph-deploy disk list cephtest004
[ceph_deploy.cli][INFO ] ceph-deploy options:
[ceph_deploy.cli][INFO ] username : None
[ceph_deploy.cli][INFO ] verbose : False
[ceph_deploy.cli][INFO ] debug : False
[ceph_deploy.cli][INFO ] overwrite_conf : False
[ceph_deploy.cli][INFO ] subcommand : list
[ceph_deploy.cli][INFO ] quiet : False
[ceph_deploy.cli][INFO ] cd_conf : <ceph_deploy.conf.cephdeploy.Conf instance at 0x7f82ecc8fc20>
[ceph_deploy.cli][INFO ] cluster : ceph
[ceph_deploy.cli][INFO ] host : ['cephtest004']
[ceph_deploy.cli][INFO ] func : <function disk at 0x7f82ed0e2cf8>
[ceph_deploy.cli][INFO ] ceph_conf : None
[ceph_deploy.cli][INFO ] default_release : False
[cephtest004][DEBUG ] connection detected need for sudo
[cephtest004][DEBUG ] connected to host: cephtest004
[cephtest004][DEBUG ] detect platform information from remote host
[cephtest004][DEBUG ] detect machine type
[cephtest004][DEBUG ] find the location of an executable
[cephtest004][INFO ] Running command: sudo fdisk -l
[cephtest004][INFO ] Disk /dev/sda: 1999.8 GB, 1999844147200 bytes, 3905945600 sectors
[cephtest004][INFO ] Disk /dev/sdb: 4000.2 GB, 4000225165312 bytes, 7812939776 sectors
[cephtest004][INFO ] Disk /dev/sdc: 4000.2 GB, 4000225165312 bytes, 7812939776 sectors
[cephtest004][INFO ] Disk /dev/sdd: 4000.2 GB, 4000225165312 bytes, 7812939776 sectors
[cephtest004][INFO ] Disk /dev/sde: 4000.2 GB, 4000225165312 bytes, 7812939776 sectors
[cephtest004][INFO ] Disk /dev/sdf: 4000.2 GB, 4000225165312 bytes, 7812939776 sectors
[cephtest004][INFO ] Disk /dev/mapper/centos-root: 1879.0 GB, 1879048192000 bytes, 3670016000 sectors
[cephtest004][INFO ] Disk /dev/mapper/centos-swap: 4294 MB, 4294967296 bytes, 8388608 sectors
[cephtest004][INFO ] Disk /dev/mapper/ceph--34f94761--91a8--4f5a--8d29--8829fe5561de-osd--block--0d229ba1--f7fb--4660--827a--571cc41e20cc: 4000.2 GB, 4000220971008 bytes, 7812931584 sectors
[cephtest004][INFO ] Disk /dev/mapper/ceph--74ce24e0--92a0--4913--8127--03c6565574ee-osd--block--f1e446eb--557a--43c0--b82d--f4503dd93e18: 4000.2 GB, 4000220971008 bytes, 7812931584 sectors
[cephtest004][INFO ] Disk /dev/mapper/ceph--b1011de5--e580--4eb0--b477--70a7a56e6c76-osd--block--f43f5d64--bb23--4a4b--aaa2--1138bca784ea: 4000.2 GB, 4000220971008 bytes, 7812931584 sectors
[cephtest004][INFO ] Disk /dev/mapper/centos-home: 107.4 GB, 107374182400 bytes, 209715200 sectors
[cephtest004][INFO ] Disk /dev/mapper/ceph--42d1e684--fadc--46e8--8816--2aedaeed1725-osd--block--5e1cd70d--470b--4821--9f7f--195a65e95c86: 4000.2 GB, 4000220971008 bytes, 7812931584 sectors
[cephadmin@cephtest001 cephcluster]$
（三）纵向扩容
3.3.1 清理待扩容磁盘（部署节点执行）
cephadmin 用户登录部署节点执行清理命令
ceph-deploy disk zap cephtest004 /dev/sdf
[cephadmin@cephtest001 cephcluster]$ pwd
/home/cephadmin/cephcluster
[cephadmin@cephtest001 cephcluster]$ ceph-deploy disk zap cephtest004 /dev/sdf
[ceph_deploy.conf][DEBUG ] found configuration file at: /home/cephadmin/.cephdeploy.conf
[ceph_deploy.cli][INFO ] Invoked (2.0.1): /bin/ceph-deploy disk zap cephtest004 /dev/sdf
[ceph_deploy.cli][INFO ] ceph-deploy options:
[ceph_deploy.cli][INFO ] username : None
[ceph_deploy.cli][INFO ] verbose : False
[ceph_deploy.cli][INFO ] debug : False
[ceph_deploy.cli][INFO ] overwrite_conf : False
[ceph_deploy.cli][INFO ] subcommand : zap
[ceph_deploy.cli][INFO ] quiet : False
[ceph_deploy.cli][INFO ] cd_conf : <ceph_deploy.conf.cephdeploy.Conf instance at 0x7fc618f5dc20>
[ceph_deploy.cli][INFO ] cluster : ceph
[ceph_deploy.cli][INFO ] host : cephtest004
[ceph_deploy.cli][INFO ] func : <function disk at 0x7fc6193b0cf8>
[ceph_deploy.cli][INFO ] ceph_conf : None
[ceph_deploy.cli][INFO ] default_release : False
[ceph_deploy.cli][INFO ] disk : ['/dev/sdf']
[ceph_deploy.osd][DEBUG ] zapping /dev/sdf on cephtest004
[cephtest004][DEBUG ] connection detected need for sudo
[cephtest004][DEBUG ] connected to host: cephtest004
[cephtest004][DEBUG ] detect platform information from remote host
[cephtest004][DEBUG ] detect machine type
[cephtest004][DEBUG ] find the location of an executable
[ceph_deploy.osd][INFO ] Distro info: CentOS Linux 7.6.1810 Core
[cephtest004][DEBUG ] zeroing last few blocks of device
[cephtest004][DEBUG ] find the location of an executable
[cephtest004][INFO ] Running command: sudo /usr/sbin/ceph-volume lvm zap /dev/sdf
[cephtest004][WARNIN] --> Zapping: /dev/sdf
[cephtest004][WARNIN] --> --destroy was not specified, but zapping a whole device will remove the partition table
[cephtest004][WARNIN] Running command: /bin/dd if=/dev/zero of=/dev/sdf bs=1M count=10 conv=fsync
[cephtest004][WARNIN] stderr: 10+0 records in
[cephtest004][WARNIN] 10+0 records out
[cephtest004][WARNIN] 10485760 bytes (10 MB) copied
[cephtest004][WARNIN] stderr: , 0.0190883 s, 549 MB/s
[cephtest004][WARNIN] --> Zapping successful for: <Raw Device: /dev/sdf>
[cephadmin@cephtest001 cephcluster]$
3.3.2 创建osd
[cephadmin@cephtest001 cephcluster]$ ceph-deploy osd create cephtest004 --data /dev/sdf
[ceph_deploy.conf][DEBUG ] found configuration file at: /home/cephadmin/.cephdeploy.conf
[ceph_deploy.cli][INFO ] Invoked (2.0.1): /bin/ceph-deploy osd create cephtest004 --data /dev/sdf
四 检查
可见新增的硬盘已经加入到集群osd 里。
[cephadmin@cephtest001 cephcluster]$ ceph -s
cluster:
id: 6cd05235-66dd-4929-b697-1562d308d5c3
health: HEALTH_WARN
1 pools have many more objects per pg than average
Degraded data redundancy: 1725/72222 objects degraded (2.388%), 7 pgs degraded
services:
mon: 4 daemons, quorum cephtest001,cephtest002,cephtest003,cephtest004 (age 30h)
mgr: cephtest001(active, since 6w), standbys: cephtest002, cephtest003, cephtest004
osd: 19 osds: 19 up (since 35s), 19 in (since 35s); 43 remapped pgs
rgw: 1 daemon active (cephtest004)
task status:
data:
pools: 8 pools, 400 pgs
objects: 24.07k objects, 167 GiB
usage: 522 GiB used, 70 TiB / 71 TiB avail
pgs: 1725/72222 objects degraded (2.388%)
10462/72222 objects misplaced (14.486%)
355 active+clean
31 active+remapped+backfill_wait
7 active+recovery_wait+undersized+degraded+remapped
4 active+remapped+backfilling
2 active+recovering+undersized+remapped
1 active+remapped
io:
client: 741 KiB/s rd, 22 KiB/s wr, 795 op/s rd, 2 op/s wr
recovery: 101 MiB/s, 18 objects/s
[cephadmin@cephtest001 cephcluster]$ ceph osd tree
ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF
-1 70.57448 root default
-3 3.26669 host cephtest001
0 hdd 1.08890 osd.0 up 1.00000 1.00000
1 hdd 1.08890 osd.1 up 1.00000 1.00000
2 hdd 1.08890 osd.2 up 1.00000 1.00000
-5 5.45547 host cephtest002
3 hdd 1.09109 osd.3 up 1.00000 1.00000
4 hdd 1.09109 osd.4 up 1.00000 1.00000
5 hdd 1.09109 osd.5 up 1.00000 1.00000
6 hdd 1.09109 osd.6 up 1.00000 1.00000
7 hdd 1.09109 osd.7 up 1.00000 1.00000
-7 43.66132 host cephtest003
8 hdd 7.27689 osd.8 up 1.00000 1.00000
9 hdd 7.27689 osd.9 up 1.00000 1.00000
10 hdd 7.27689 osd.10 up 1.00000 1.00000
11 hdd 7.27689 osd.11 up 1.00000 1.00000
12 hdd 7.27689 osd.12 up 1.00000 1.00000
13 hdd 7.27689 osd.13 up 1.00000 1.00000
-9 18.19099 host cephtest004
14 hdd 3.63820 osd.14 up 1.00000 1.00000
15 hdd 3.63820 osd.15 up 1.00000 1.00000
16 hdd 3.63820 osd.16 up 1.00000 1.00000
17 hdd 3.63820 osd.17 up 1.00000 1.00000
18 hdd 3.63820 osd.18 up 1.00000 1.00000
[cephadmin@cephtest001 cephcluster]$
五 参考
https://docs.ceph.com/en/nautilus/rados/deployment/ceph-deploy-osd/#list-osds
posted on 2021-02-06 14:26 weiwei2021 阅读(246) 评论(0) 编辑 收藏 举报
