Ceph运维记录2022

RBD块存储使用：

rbd list

sudo rbd device map {pool-name}/{image-name} --id {user-name}
sudo rbd device map rbd/myimage --id admin

sudo rbd device map rbd/myimage --id admin --keyring /path/to/keyring
sudo rbd device map rbd/myimage --id admin --keyfile /path/to/file

rbd device list

sudo rbd device unmap /dev/rbd/{poolname}/{imagename}
sudo rbd device unmap /dev/rbd/rbd/foo


因为 RBD 不知道映像（卷）中的任何文件系统，所以快照不是崩溃一致的，除非它们在挂载（附加）操作系统中进行协调。
因此，我们建议您在拍摄快照之前暂停或停止 I/O。如果卷包含文件系统，则在拍摄快照之前它必须处于内部一致状态。在不一致的点拍摄的快照可能需要在后续安装之前通过fsck 。
要停止I/O，您可以使用fsfreeze命令。对于虚拟机，可以使用qemu-guest-agent在创建快照时自动冻结文件系统。

备份：

# rbd info vm-100-disk-0
rbd image 'vm-100-disk-0':
        size 20 GiB in 5120 objects
        order 22 (4 MiB objects)
        id: 6487c225fb36
        block_name_prefix: rbd_data.6487c225fb36
        format: 2
        features: layering, exclusive-lock, object-map, fast-diff, deep-flatten
        op_features:
        flags:
        create_timestamp: Mon Jan 10 20:00:11 2022
# rbd snap create vm-100-disk-0@v1  # 第一次快照
# rbd export-diff vm-100-disk-0@v1 testimage_v1  # 第一次全量备份

# rbd snap create vm-100-disk-0@v2  # 第二次快照
# rbd export-diff vm-100-disk-0@v2 testimage_v2  # 第二次全量备份

# rbd export-diff vm-100-disk-0@v2 --from-snap v1 testimage_v1_v2  # 增量备份（第二次和第一次的差异数据）

恢复：

# rbd create --size 2048 testbacknew2  # 随便创建一个image，名称大小都不限制，因为后面恢复的时候会覆盖掉大小的信息
# rbd import-diff testimage_v2 testbacknew_v2  # 直接基于v2的时间点的快照做恢复

# rbd create --size 2048 testbacknew
# rbd info testbacknew
rbd image 'testbacknew':
        size 2 GiB in 512 objects
        order 22 (4 MiB objects)
        id: bbc5a46b8b4567
        block_name_prefix: rbd_data.bbc5a46b8b4567
        format: 2
        features: layering, exclusive-lock, object-map, fast-diff, deep-flatten
        op_features:
        flags:
        create_timestamp: Sat Apr  2 13:44:13 2022
# rbd import-diff testimage_v1 testbacknew  # 直接基于v1的时间点的数据，和后面的增量的v1_v2数据（要按顺序导入）
# rbd import-diff testimage_v1_v2 testbacknew
# rbd info testbacknew
rbd image 'testbacknew':
        size 20 GiB in 5120 objects
        order 22 (4 MiB objects)
        id: bbc5a46b8b4567
        block_name_prefix: rbd_data.bbc5a46b8b4567
        format: 2
        features: layering, exclusive-lock, object-map, fast-diff, deep-flatten
        op_features:
        flags:
        create_timestamp: Sat Apr  2 13:44:13 2022
# rbd snap purge <imagename>  # 删除镜像所有未保护的快照
例如：
# rbd snap purge vm-100-disk-0
Removing all snapshots: 100% complete...done.

重建OSD：

ceph osd destroy {id} --yes-i-really-mean-it

ceph-volume lvm list
ceph-volume lvm zap /dev/ceph-ssd-1/ssd-1
ceph-volume lvm create --osd-id 29 --data /dev/ceph-ssd-1/ssd-1

移除writeback缓存层：

ceph osd tier cache-mode {cachepool} forward
rados -p {cachepool} ls
rados -p {cachepool} cache-flush-evict-all
ceph osd tier remove-overlay {storagetier}
ceph osd tier remove {storagepool} {cachepool}

修改BMC IP：

yum install ipmitool -y                        #安装ipmitool工具
ipmitool lan print 2                           #查看BMC信息
ipmitool lan set 2 ipaddr 192.168.1.1          #修改BMC IP
ipmitool lan set 2 netmask 255.255.255.0       #修改BMC 掩码
ipmitool lan set 2 defgw ipaddr 192.168.1.254  #修改BMC网关

cephfs客户端管理：

ceph tell mds.* config set debug_mds 1/5
ceph tell mds.* client ls 2>/dev/null | grep -w id | awk '{print $2}' | awk -F, '{print $1}'
ceph tell mds.* client evict id=1364566

$ ceph osd blacklist ls
listed 1 entries
127.0.0.1:0/3710147553 2018-03-19 11:32:24.716146

$ ceph osd blacklist rm 127.0.0.1:0/3710147553
un-blacklisting 127.0.0.1:0/3710147553

修改ceph运行时配置：

ceph tell osd.* config get osd_recovery_op_priority
ceph tell osd.* config get osd_client_op_priority

ceph tell osd.* config set osd_recovery_op_priority 44
ceph tell osd.* config set osd_client_op_priority 22

ceph tell osd.* config get osd_recovery_max_active
3
ceph tell osd.* config get osd_recovery_sleep
0
ceph tell osd.* config get osd_max_backfills
1
ceph tell osd.* config get osd_recovery_max_single_start
1
ceph tell osd.* config get osd_max_backfills
1

ceph tell osd.* config set osd_max_backfills 5
ceph tell osd.* config set osd_recovery_max_active 5
ceph tell osd.* config set osd_recovery_max_single_start 3

ceph tell mon.* injectargs "--mon-osd-full-ratio 0.95"
ceph tell mon.* injectargs "--mon-allow-pool-delete false"

ceph部署记录：

ceph crash info 2022-05-22_21:53:10.154311Z_3ed873c8-da2b-4188-b12e-8198c9548036

/var/log/dmesg
dmesg -T

# rpm --import https://www.elrepo.org/RPM-GPG-KEY-elrepo.org
# yum install -y https://www.elrepo.org/elrepo-release-7.el7.elrepo.noarch.rpm
# yum list available --disablerepo=* --enablerepo=elrepo-kernel
# yum install kernel-lt.x86_64 --enablerepo=elrepo-kernel
# cat /boot/grub2/grub.cfg | grep menuentry
# cat /etc/grub2-efi.cfg | grep ^menuentry  // 如果是UEFI启动
# grub2-set-default "CentOS Linux (5.4.190-1.el7.elrepo.x86_64) 7 (Core)"
# grub2-editenv list
# reboot
# uname -r

rpm --import https://www.elrepo.org/RPM-GPG-KEY-elrepo.org
yum install -y https://www.elrepo.org/elrepo-release-7.el7.elrepo.noarch.rpm
yum install -y kernel-lt.x86_64 --enablerepo=elrepo-kernel
grub2-set-default "CentOS Linux (5.4.190-1.el7.elrepo.x86_64) 7 (Core)"
grub2-editenv list
reboot

# systemctl stop firewalld.service
# systemctl disable firewalld.service
# setenforce 0
# vim /etc/sysconfig/selinux
SELINUX=disabled

curl --silent --remote-name --location https://github.com/ceph/ceph/raw/octopus/src/cephadm/cephadm
chmod +x cephadm
./cephadm add-repo --release octopus
./cephadm install
which cephadm
cephadm pull

kernel.pid_max=4194304
fs.file-max=26345836
vm.swappiness=0

/etc/security/limits.d/ceph.conf
* soft nofile 655360
* hard nofile 655360
* soft nproc 655350
* hard nproc 655350
* soft memlock unlimited
* hard memlock unlimited

posted @ 2022-06-29 13:34 Varden 阅读(307) 评论(0) 编辑收藏举报

会员力量，点亮园子希望

刷新页面返回顶部

Varden

Ceph运维记录2022

公告