分布式存储系统Ceph
https://docs.ceph.com/en/latest/start/intro/
一、Ceph简介
1. Ceph的几种存储类型
Ceph是一种开源的分布式的存储系统，包含以下几种存储类型：
块存储（rbd），对象存储(RADOS Fateway)，文件系统（Cephfs）
1）. 块存储（rbd）
块是一个字节序列（例如，512字节的数据块）。 基于块的存储接口是使用旋转介质（如硬盘，CD，软盘甚至传统的9轨磁带）是存储数据的最常用方法。
Ceph块设备是精简配置，可调整大小并存储在Ceph集群中多个OSD条带化的数据。 Ceph块设备利用RADOS功能，如快照，复制和一致性。 Ceph的RADOS块设备（RBD）使用内核模块或librbd库与OSD进行交互。可以使用同一个集群同时运行Ceph RADOS Gateway，CephFS文件系统和Ceph块设备。
Linux系统中，ls /dev/下有很多块设备文件，这些文件就是添加硬盘时识别出来的；rbd就是由Ceph集群提供出来的块设备。即：sda是通过数据线连接到了真实的硬盘，而rbd是通过网络连接到了Ceph集群中的一块存储区域，往rbd设备文件写入数据，最终会被存储到Ceph集群的这块区域中。
总结：块设备可理解成一块硬盘，用户可以直接使用不含文件系统的块设备，也可以将其格式化成特定的文件系统，由文件系统来组织管理存储空间，从而为用户提供丰富而友好的数据操作支持。
2）. 文件系统CephFS
Ceph文件系统（CephFS）是一个符合POSIX标准的文件系统，它使用Ceph存储集群来存储其数据。 Ceph文件系统使用与Ceph块设备相同的Ceph存储集群系统。用户可以在块设备上创建xfs文件系统，也可以创建ext4等其他文件系统，Ceph集群实现了自己的文件系统来组织管理集群的存储空间，用户可以直接将Ceph集群的文件系统挂载到用户机上使用。
Ceph有了块设备接口，在块设备上完全可以构建一个文件系统，那么Ceph为什么还需要文件系统接口呢？主要是因为应用场景的不同，Ceph的块设备具有优异的读写性能，但不能多处挂载同时读写，目前主要用在OpenStack上作为虚拟磁盘，而Ceph的文件系统接口读写性能较块设备接口差，但具有优异的共享性。
3）. 对象存储
Ceph对象存储使用Ceph对象网关守护进程（radosgw），它是一个用于与Ceph存储集群交互的HTTP服务器。由于它提供与OpenStack Swift和Amazon S3兼容的接口，因此Ceph对象网关具有自己的用户管理。 Ceph对象网关可以将数据存储在用于存储来自Ceph文件系统客户端或Ceph块设备客户端的数据的相同Ceph存储集群中，使用方式就是通过http协议上传下载删除对象（文件即对象）。
那么，有了块设备接口存储和文件系统接口存储，为什么还整个对象存储呢? 是因为Ceph的块设备存储具有优异的存储性能但不具有共享性，而Ceph的文件系统具有共享性然而性能较块设备存储差，而对象存储既具有共享性且存储性能又好于文件系统存储。
2. 分布式存储的优点
高可靠：既满足存储读取不丢失，还要保证数据长期存储。 在保证部分硬件损坏后依然可以保证数据安全
高性能：读写速度快
可扩展：分布式存储的优势就是“分布式”，所谓的“分布式”就是能够将多个物理节点整合在一起形成共享的存储池，节点可以线性扩充，这样可以源源不断的通过扩充节点提升性能和扩大容量。
二、Ceph核心组件
Ceph存储集群至少需要一个Ceph Monitor，Ceph Manager和Ceph OSD（对象存储守护进程）。 运行Ceph Filesystem客户端时也需要Ceph元数据服务器。
1. Ceph Monitor
Ceph Monitor 维护整个ceph的状态，是Ceph的监视器（ceph-mon）维护集群状态的映射，包括监视器映射，管理器映射，OSD映射和CRUSH映射。这些映射是Ceph守护进程相互协调所需的关键集群状态。监视器还负责管理守护进程和客户端之间的身份验证。冗余和高可用性通常至少需要三个监视器。
2. Ceph Manager
Ceph Manager守护程序（ceph-mgr）负责跟踪运行时指标和Ceph集群的当前状态，包括存储利用率，当前性能指标和系统负载。 Ceph Manager守护进程还托管基于python的模块来管理和公开Ceph集群信息，包括基于Web的Ceph Dashboard和REST API。ceph集群的运行指标统计，可以通过web显示ceph集群的运行状态，包括存储池信息、集群IO和负载。高可用性通常至少需要两名Managers。
3. Ceph OSD
Ceph OSD（对象存储守护进程，ceph-osd）存储数据，可以理解为一块磁盘创建一个OSDs，每个磁盘都会启动一个OSD进程。处理数据复制，恢复，重新平衡，并通过检查其他Ceph OSD守护进程来获取心跳，为Ceph监视器和管理器提供一些监视信息。冗余和高可用性通常至少需要3个Ceph OSD。
4. MDS
Ceph元数据服务器（MDS，ceph-mds）代表Ceph文件系统存储元数据（即，Ceph块设备和Ceph对象存储不使用MDS，它们的元数据是直接写到磁盘上的）。 Ceph元数据服务器允许POSIX文件系统用户执行基本命令（如ls，find等），而不会给Ceph存储集群带来巨大负担。
三、安装Ceph集群
1. 环境准备
机器配置：CentOS 7.9；网络模式：NAT；准备三台机器，每台机器需要两块硬盘
2. 初始化实验环境
1）配置hosts文件
在上述三台机器上分别执行如下命令，修改/etc/hosts文件内容：
cat >>/etc/hosts <<EOF > 10.0.0.130 ceph-admin > 10.0.0.129 ceph-monitor > 10.0.0.128 ceph-osd > > EOF cat /etc/hosts 127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4 ::1 localhost localhost.localdomain localhost6 localhost6.localdomain6 10.0.0.130 ceph-admin 10.0.0.129 ceph-monitor 10.0.0.128 ceph-osd
2）配置互信
生成ssh 密钥对，并将本地的ssh公钥文件上传至其他主机
[root@ceph-admin ~]# ssh-keygen -t rsa Generating public/private rsa key pair. Enter file in which to save the key (/root/.ssh/id_rsa): Created directory '/root/.ssh'. Enter passphrase (empty for no passphrase): Enter same passphrase again: Your identification has been saved in /root/.ssh/id_rsa. Your public key has been saved in /root/.ssh/id_rsa.pub. The key fingerprint is: SHA256:Pwig0NCfayeXYJDPt3SSco9txTs2KpeHk3U5I1CCggY root@ceph-admin The key's randomart image is: +---[RSA 2048]----+ |Eo o . | | o* . . . . | |...=.o . + | | . .O.* o o | | .. B.OSo . . | | + *.+oB = | | . + ..Bo= o | | . B .. | | o o | +----[SHA256]-----+ #上传ssh公钥文件 [root@ceph-admin ~]# ssh-copy-id ceph-admin /usr/bin/ssh-copy-id: INFO: Source of key(s) to be installed: "/root/.ssh/id_rsa.pub" The authenticity of host 'ceph-admin (10.0.0.130)' can't be established. ECDSA key fingerprint is SHA256:J9UnR8HG9Iws8xvmhv4HMjfjJUgOGgEV/3yQ/kFT87c. ECDSA key fingerprint is MD5:af:38:29:b9:6b:1c:eb:03:bd:93:ad:0d:5a:68:4d:06. Are you sure you want to continue connecting (yes/no)? yes /usr/bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter out any that are already installed /usr/bin/ssh-copy-id: INFO: 1 key(s) remain to be installed -- if you are prompted now it is to install the new keys root@ceph-admin's password: Number of key(s) added: 1 Now try logging into the machine, with: "ssh 'ceph-admin'" and check to make sure that only the key(s) you wanted were added. [root@ceph-admin ~]# ssh-copy-id ceph-monitor /usr/bin/ssh-copy-id: INFO: Source of key(s) to be installed: "/root/.ssh/id_rsa.pub" The authenticity of host 'ceph-monitor (10.0.0.129)' can't be established. ECDSA key fingerprint is SHA256:J9UnR8HG9Iws8xvmhv4HMjfjJUgOGgEV/3yQ/kFT87c. ECDSA key fingerprint is MD5:af:38:29:b9:6b:1c:eb:03:bd:93:ad:0d:5a:68:4d:06. Are you sure you want to continue connecting (yes/no)? yes /usr/bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter out any that are already installed /usr/bin/ssh-copy-id: INFO: 1 key(s) remain to be installed -- if you are prompted now it is to install the new keys root@ceph-monitor's password: Number of key(s) added: 1 Now try logging into the machine, with: "ssh 'ceph-monitor'" and check to make sure that only the key(s) you wanted were added. [root@ceph-admin ~]# ssh-copy-id ceph-osd /usr/bin/ssh-copy-id: INFO: Source of key(s) to be installed: "/root/.ssh/id_rsa.pub" The authenticity of host 'ceph-osd (10.0.0.128)' can't be established. ECDSA key fingerprint is SHA256:J9UnR8HG9Iws8xvmhv4HMjfjJUgOGgEV/3yQ/kFT87c. ECDSA key fingerprint is MD5:af:38:29:b9:6b:1c:eb:03:bd:93:ad:0d:5a:68:4d:06. Are you sure you want to continue connecting (yes/no)? yes /usr/bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter out any that are already installed /usr/bin/ssh-copy-id: INFO: 1 key(s) remain to be installed -- if you are prompted now it is to install the new keys root@ceph-osd's password: Number of key(s) added: 1 Now try logging into the machine, with: "ssh 'ceph-osd'" and check to make sure that only the key(s) you wanted were added. [root@ceph-admin ~]#
其他节点同理操作。
3）配置Ceph安装源
[root@ceph-admin ~]# cat /etc/yum.repos.d/ceph.repo [Ceph] name=Ceph packages for $basearch baseurl=http://mirrors.tuna.tsinghua.edu.cn/ceph/rpm-jewel/el7/x86_64/ enabled=1 gpgcheck=0 type=rpm-md gpgkey=https://mirrors.tuna.tsinghua.edu.cn/ceph/keys/release.asc priority=1 [Ceph-noarch] name=Ceph noarch packages baseurl=http://mirrors.tuna.tsinghua.edu.cn/ceph/rpm-jewel/el7/noarch/ enabled=1 gpgcheck=0 type=rpm-md gpgkey=https://mirrors.tuna.tsinghua.edu.cn/ceph/keys/release.asc priority=1 [ceph-source] name=Ceph source packages baseurl=http://mirrors.tuna.tsinghua.edu.cn/ceph/rpm-jewel/el7/SRPMS/ enabled=1 gpgcheck=0 type=rpm-md gpgkey=https://mirrors.tuna.tsinghua.edu.cn/ceph/keys/release.asc priority=1
检测源的可用性
[root@ceph-admin ~]# yum makecache Loaded plugins: fastestmirror Loading mirror speeds from cached hostfile epel/x86_64/metalink | 9.4 kB 00:00:00 * epel: mirror.lzu.edu.cn Ceph | 2.9 kB 00:00:00 Ceph-noarch | 2.9 kB 00:00:00 base | 3.6 kB 00:00:00 ceph-source | 2.9 kB 00:00:00 docker-ce-stable | 3.5 kB 00:00:00 extras | 2.9 kB 00:00:00 updates | 2.9 kB 00:00:00 (1/9): Ceph/filelists_db | 269 kB 00:00:00 (2/9): Ceph/primary_db | 329 kB 00:00:00 (3/9): Ceph/other_db | 28 kB 00:00:00 (4/9): Ceph-noarch/filelists_db | 13 kB 00:00:00 (5/9): Ceph-noarch/other_db | 1.9 kB 00:00:00 (6/9): ceph-source/primary_db | 20 kB 00:00:00 (7/9): ceph-source/other_db | 2.8 kB 00:00:00 (8/9): ceph-source/filelists_db | 3.3 kB 00:00:00 (9/9): Ceph-noarch/primary_db | 6.4 kB 00:00:01 Metadata Cache Created #查看源列表 [root@ceph-admin ~]# yum repolist -v Loading "fastestmirror" plugin Config time: 0.009 Yum version: 3.4.3 Loading mirror speeds from cached hostfile * epel: mirror.lzu.edu.cn Setting up Package Sacks pkgsack time: 0.101 Repo-id : Ceph Repo-name : Ceph packages for x86_64 Repo-revision: 1531250891 Repo-updated : Wed Jul 11 03:32:26 2018 Repo-pkgs : 499 Repo-size : 20 G Repo-baseurl : http://mirrors.tuna.tsinghua.edu.cn/ceph/rpm-jewel/el7/x86_64/ Repo-expire : 21,600 second(s) (last: Tue Sep 27 22:36:53 2022) Filter : read-only:present Repo-filename: /etc/yum.repos.d/ceph.repo Repo-id : Ceph-noarch Repo-name : Ceph noarch packages Repo-revision: 1531250880 Repo-updated : Wed Jul 11 03:28:04 2018 Repo-pkgs : 16 Repo-size : 3.2 M Repo-baseurl : http://mirrors.tuna.tsinghua.edu.cn/ceph/rpm-jewel/el7/noarch/ Repo-expire : 21,600 second(s) (last: Tue Sep 27 22:36:53 2022) Filter : read-only:present Repo-filename: /etc/yum.repos.d/ceph.repo Repo-id : base/7/x86_64 Repo-name : CentOS-7 - Base Repo-revision: 1604001756 Repo-updated : Fri Oct 30 04:03:00 2020 Repo-pkgs : 10,072 Repo-size : 8.9 G Repo-baseurl : https://mirrors.tuna.tsinghua.edu.cn/centos/7/os/x86_64/ Repo-expire : 21,600 second(s) (last: Tue Sep 27 22:36:54 2022) Filter : read-only:present Repo-filename: /etc/yum.repos.d/CentOS-Base.repo Repo-id : ceph-source Repo-name : Ceph source packages Repo-revision: 1531250885 Repo-updated : Wed Jul 11 03:28:11 2018 Repo-pkgs : 0 Repo-size : 0 Repo-baseurl : http://mirrors.tuna.tsinghua.edu.cn/ceph/rpm-jewel/el7/SRPMS/ Repo-expire : 21,600 second(s) (last: Tue Sep 27 22:36:54 2022) Filter : read-only:present Repo-filename: /etc/yum.repos.d/ceph.repo Repo-id : docker-ce-stable/7/x86_64 Repo-name : Docker CE Stable - x86_64 Repo-revision: 1662713508 Repo-updated : Fri Sep 9 16:51:48 2022 Repo-pkgs : 169 Repo-size : 3.8 G Repo-baseurl : https://mirrors.tuna.tsinghua.edu.cn/docker-ce/linux/centos/7/x86_64/stable/ Repo-expire : 21,600 second(s) (last: Tue Sep 27 22:36:54 2022) Filter : read-only:present Repo-filename: /etc/yum.repos.d/docker-ce.repo Repo-id : epel/x86_64 Repo-name : Extra Packages for Enterprise Linux 7 - x86_64 Repo-revision: 1664157257 Repo-updated : Mon Sep 26 10:14:48 2022 Repo-pkgs : 13,752 Repo-size : 16 G Repo-metalink: https://mirrors.fedoraproject.org/metalink?repo=epel-7&arch=x86_64&infra=stock&content=centos Updated : Mon Sep 26 10:14:48 2022 Repo-baseurl : https://mirror.lzu.edu.cn/epel/7/x86_64/ (22 more) Repo-expire : 21,600 second(s) (last: Tue Sep 27 22:36:54 2022) Filter : read-only:present Repo-filename: /etc/yum.repos.d/epel.repo Repo-id : extras/7/x86_64 Repo-name : CentOS-7 - Extras Repo-revision: 1662061316 Repo-updated : Fri Sep 2 03:41:58 2022 Repo-pkgs : 516 Repo-size : 1.0 G Repo-baseurl : https://mirrors.tuna.tsinghua.edu.cn/centos/7/extras/x86_64/ Repo-expire : 21,600 second(s) (last: Tue Sep 27 22:36:54 2022) Filter : read-only:present Repo-filename: /etc/yum.repos.d/CentOS-Base.repo Repo-id : updates/7/x86_64 Repo-name : CentOS-7 - Updates Repo-revision: 1663935737 Repo-updated : Fri Sep 23 20:23:22 2022 Repo-pkgs : 4,244 Repo-size : 28 G Repo-baseurl : https://mirrors.tuna.tsinghua.edu.cn/centos/7/updates/x86_64/ Repo-expire : 21,600 second(s) (last: Tue Sep 27 22:36:54 2022) Filter : read-only:present Repo-filename: /etc/yum.repos.d/CentOS-Base.repo repolist: 29,268 [root@ceph-admin ~]#
将配置好的ceph.repo源文件上传至ceph-monitor和ceph-osd节点上
[root@ceph-admin ~]# scp /etc/yum.repos.d/ceph.repo 10.0.0.129:/etc/yum.repos.d/ceph.repo ceph.repo 100% 681 665.8KB/s 00:00 [root@ceph-admin ~]# scp /etc/yum.repos.d/ceph.repo 10.0.0.128:/etc/yum.repos.d/ceph.repo ceph.repo 100% 681 256.6KB/s 00:00 [root@ceph-admin ~]#
3. 安装ceph集群
1）安装插件ceph-deploy,ceph,ceph-radosgw
在ceph-admin节点安装ceph-deploy
[root@ceph-admin ~]# yum install python-setuptools ceph-deploy -y
在ceph-admin、ceph-monitor和ceph-osd节点安装ceph
[root@ceph-admin ~]# yum install ceph ceph-radosgw -y [root@ceph-osd ~]# yum install ceph ceph-radosgw -y [root@ceph-monitor ~]# yum install ceph ceph-radosgw -y
查看ceph版本
[root@ceph-admin ~]# ceph --version ceph version 10.2.11 (e4b061b47f07f583c92a050d9e84b1813a35671e) [root@ceph-admin ~]#
2）创建monitor节点
a. 创建一个目录，用于保存 ceph-deploy 生成的配置文件信息的
[root@ceph-admin ~]# cd /etc/ceph [root@ceph-admin ceph]# ll total 4 -rw-r--r-- 1 root root 92 Jul 10 2018 rbdmap [root@ceph-admin ceph]# ceph-deploy new ceph-admin ceph-monitor ceph-osd [ceph_deploy.conf][DEBUG ] found configuration file at: /root/.cephdeploy.conf [ceph_deploy.cli][INFO ] Invoked (1.5.39): /usr/bin/ceph-deploy new ceph-admin ceph-monitor ceph-osd [ceph_deploy.cli][INFO ] ceph-deploy options: [ceph_deploy.cli][INFO ] username : None [ceph_deploy.cli][INFO ] func : <function new at 0x7f91d7f70758> [ceph_deploy.cli][INFO ] verbose : False [ceph_deploy.cli][INFO ] overwrite_conf : False [ceph_deploy.cli][INFO ] quiet : False [ceph_deploy.cli][INFO ] cd_conf : <ceph_deploy.conf.cephdeploy.Conf instance at 0x7f91d76dcf38> [ceph_deploy.cli][INFO ] cluster : ceph [ceph_deploy.cli][INFO ] ssh_copykey : True [ceph_deploy.cli][INFO ] mon : ['ceph-admin', 'ceph-monitor', 'ceph-osd'] [ceph_deploy.cli][INFO ] public_network : None [ceph_deploy.cli][INFO ] ceph_conf : None [ceph_deploy.cli][INFO ] cluster_network : None [ceph_deploy.cli][INFO ] default_release : False [ceph_deploy.cli][INFO ] fsid : None [ceph_deploy.new][DEBUG ] Creating new cluster named ceph [ceph_deploy.new][INFO ] making sure passwordless SSH succeeds [ceph-admin][DEBUG ] connected to host: ceph-admin [ceph-admin][DEBUG ] detect platform information from remote host [ceph-admin][DEBUG ] detect machine type [ceph-admin][DEBUG ] find the location of an executable [ceph-admin][INFO ] Running command: /usr/sbin/ip link show [ceph-admin][INFO ] Running command: /usr/sbin/ip addr show [ceph-admin][DEBUG ] IP addresses found: [u'172.17.0.1', u'10.0.0.130'] [ceph_deploy.new][DEBUG ] Resolving host ceph-admin [ceph_deploy.new][DEBUG ] Monitor ceph-admin at 10.0.0.130 [ceph_deploy.new][INFO ] making sure passwordless SSH succeeds [ceph-monitor][DEBUG ] connected to host: ceph-admin [ceph-monitor][INFO ] Running command: ssh -CT -o BatchMode=yes ceph-monitor [ceph-monitor][DEBUG ] connected to host: ceph-monitor [ceph-monitor][DEBUG ] detect platform information from remote host [ceph-monitor][DEBUG ] detect machine type [ceph-monitor][DEBUG ] find the location of an executable [ceph-monitor][INFO ] Running command: /usr/sbin/ip link show [ceph-monitor][INFO ] Running command: /usr/sbin/ip addr show [ceph-monitor][DEBUG ] IP addresses found: [u'10.0.0.129', u'172.17.0.1'] [ceph_deploy.new][DEBUG ] Resolving host ceph-monitor [ceph_deploy.new][DEBUG ] Monitor ceph-monitor at 10.0.0.129 [ceph_deploy.new][INFO ] making sure passwordless SSH succeeds [ceph-osd][DEBUG ] connected to host: ceph-admin [ceph-osd][INFO ] Running command: ssh -CT -o BatchMode=yes ceph-osd [ceph-osd][DEBUG ] connected to host: ceph-osd [ceph-osd][DEBUG ] detect platform information from remote host [ceph-osd][DEBUG ] detect machine type [ceph-osd][DEBUG ] find the location of an executable [ceph-osd][INFO ] Running command: /usr/sbin/ip link show [ceph-osd][INFO ] Running command: /usr/sbin/ip addr show [ceph-osd][DEBUG ] IP addresses found: [u'10.0.0.128', u'172.17.0.1'] [ceph_deploy.new][DEBUG ] Resolving host ceph-osd [ceph_deploy.new][DEBUG ] Monitor ceph-osd at 10.0.0.128 [ceph_deploy.new][DEBUG ] Monitor initial members are ['ceph-admin', 'ceph-monitor', 'ceph-osd'] [ceph_deploy.new][DEBUG ] Monitor addrs are ['10.0.0.130', '10.0.0.129', '10.0.0.128'] [ceph_deploy.new][DEBUG ] Creating a random mon key... [ceph_deploy.new][DEBUG ] Writing monitor keyring to ceph.mon.keyring... [ceph_deploy.new][DEBUG ] Writing initial config to ceph.conf...
查看共生成Ceph配置文件、一个monitor密钥环和一个日志文件
[root@ceph-admin ceph]# ls -lrt total 20 -rw-r--r-- 1 root root 92 Jul 10 2018 rbdmap -rw------- 1 root root 73 Sep 27 22:59 ceph.mon.keyring -rw-r--r-- 1 root root 5115 Sep 27 22:59 ceph-deploy-ceph.log -rw-r--r-- 1 root root 244 Sep 27 22:59 ceph.conf [root@ceph-admin ceph]#
b. 修改ceph配置文件，修改以下内容
#将ceph.conf配置文件里的默认副本数从3改成1
#将osd_pool_default_size = 2 加入[global]段，这样只有2个osd也能达到active+clean状态
#mon clock drift allowed #监视器间允许的时钟漂移量默认值0.05
#mon clock drift warn backoff #时钟偏移警告的退避指数。默认值5
[root@ceph-admin ceph]# vim ceph.conf You have new mail in /var/spool/mail/root [root@ceph-admin ceph]# cat ceph.conf [global] fsid = 75bf70df-8562-4f70-bb28-db9376a1ce98 mon_initial_members = ceph-monitor mon_host = 10.0.0.129 auth_cluster_required = cephx auth_service_required = cephx auth_client_required = cephx osd_pool_default_size = 2 mon clock drift allowed = 0.500 mon clock drift warn backoff = 10 [root@ceph-admin ceph]#
ceph对每个monitor之间的时间同步延时默认要求在0.05s之间，这个时间有的时候太短了。所以如果ceph集群如果出现clock问题就检查ntp时间同步或者适当放宽这个误差时间。
cephx是认证机制是整个 Ceph 系统的用户名/密码
c. 配置初始monitor、收集所有的密钥
[root@ceph-admin ceph]# ceph-deploy mon create-initial
[ceph_deploy.conf][DEBUG ] found configuration file at: /root/.cephdeploy.conf
[ceph_deploy.cli][INFO ] Invoked (1.5.39): /usr/bin/ceph-deploy mon create-initial
[ceph_deploy.cli][INFO ] ceph-deploy options:
[ceph_deploy.cli][INFO ] username : None
[ceph_deploy.cli][INFO ] verbose : False
[ceph_deploy.cli][INFO ] overwrite_conf : False
[ceph_deploy.cli][INFO ] subcommand : create-initial
[ceph_deploy.cli][INFO ] quiet : False
[ceph_deploy.cli][INFO ] cd_conf : <ceph_deploy.conf.cephdeploy.Conf instance at 0x7f69f5c49488>
[ceph_deploy.cli][INFO ] cluster : ceph
[ceph_deploy.cli][INFO ] func : <function mon at 0x7f69f5c2e848>
[ceph_deploy.cli][INFO ] ceph_conf : None
[ceph_deploy.cli][INFO ] default_release : False
[ceph_deploy.cli][INFO ] keyrings : None
[ceph_deploy.mon][DEBUG ] Deploying mon, cluster ceph hosts ceph-monitor
[ceph_deploy.mon][DEBUG ] detecting platform for host ceph-monitor ...
[ceph-monitor][DEBUG ] connected to host: ceph-monitor
[ceph-monitor][DEBUG ] detect platform information from remote host
[ceph-monitor][DEBUG ] detect machine type
[ceph-monitor][DEBUG ] find the location of an executable
[ceph_deploy.mon][INFO ] distro info: CentOS Linux 7.9.2009 Core
[ceph-monitor][DEBUG ] determining if provided host has same hostname in remote
[ceph-monitor][DEBUG ] get remote short hostname
[ceph-monitor][DEBUG ] deploying mon to ceph-monitor
[ceph-monitor][DEBUG ] get remote short hostname
[ceph-monitor][DEBUG ] remote hostname: ceph-monitor
[ceph-monitor][DEBUG ] write cluster configuration to /etc/ceph/{cluster}.conf
[ceph-monitor][DEBUG ] create the mon path if it does not exist
[ceph-monitor][DEBUG ] checking for done path: /var/lib/ceph/mon/ceph-ceph-monitor/done
[ceph-monitor][DEBUG ] done path does not exist: /var/lib/ceph/mon/ceph-ceph-monitor/done
[ceph-monitor][INFO ] creating keyring file: /var/lib/ceph/tmp/ceph-ceph-monitor.mon.keyring
[ceph-monitor][DEBUG ] create the monitor keyring file
[ceph-monitor][INFO ] Running command: ceph-mon --cluster ceph --mkfs -i ceph-monitor --keyring /var/lib/ceph/tmp/ceph-ceph-monitor.mon.keyring --setuser 167 --setgroup 167
[ceph-monitor][DEBUG ] ceph-mon: mon.noname-a 10.0.0.129:6789/0 is local, renaming to mon.ceph-monitor
[ceph-monitor][DEBUG ] ceph-mon: set fsid to 75bf70df-8562-4f70-bb28-db9376a1ce98
[ceph-monitor][DEBUG ] ceph-mon: created monfs at /var/lib/ceph/mon/ceph-ceph-monitor for mon.ceph-monitor
[ceph-monitor][INFO ] unlinking keyring file /var/lib/ceph/tmp/ceph-ceph-monitor.mon.keyring
[ceph-monitor][DEBUG ] create a done file to avoid re-doing the mon deployment
[ceph-monitor][DEBUG ] create the init path if it does not exist
[ceph-monitor][INFO ] Running command: systemctl enable ceph.target
[ceph-monitor][INFO ] Running command: systemctl enable ceph-mon@ceph-monitor
[ceph-monitor][WARNIN] Created symlink from /etc/systemd/system/ceph-mon.target.wants/ceph-mon@ceph-monitor.service to /usr/lib/systemd/system/ceph-mon@.service.
[ceph-monitor][INFO ] Running command: systemctl start ceph-mon@ceph-monitor
[ceph-monitor][INFO ] Running command: ceph --cluster=ceph --admin-daemon /var/run/ceph/ceph-mon.ceph-monitor.asok mon_status
[ceph-monitor][DEBUG ] ********************************************************************************
[ceph-monitor][DEBUG ] status for monitor: mon.ceph-monitor
[ceph-monitor][DEBUG ] {
[ceph-monitor][DEBUG ] "election_epoch": 3,
[ceph-monitor][DEBUG ] "extra_probe_peers": [],
[ceph-monitor][DEBUG ] "monmap": {
[ceph-monitor][DEBUG ] "created": "2022-09-27 23:06:45.440351",
[ceph-monitor][DEBUG ] "epoch": 1,
[ceph-monitor][DEBUG ] "fsid": "75bf70df-8562-4f70-bb28-db9376a1ce98",
[ceph-monitor][DEBUG ] "modified": "2022-09-27 23:06:45.440351",
[ceph-monitor][DEBUG ] "mons": [
[ceph-monitor][DEBUG ] {
[ceph-monitor][DEBUG ] "addr": "10.0.0.129:6789/0",
[ceph-monitor][DEBUG ] "name": "ceph-monitor",
[ceph-monitor][DEBUG ] "rank": 0
[ceph-monitor][DEBUG ] }
[ceph-monitor][DEBUG ] ]
[ceph-monitor][DEBUG ] },
[ceph-monitor][DEBUG ] "name": "ceph-monitor",
[ceph-monitor][DEBUG ] "outside_quorum": [],
[ceph-monitor][DEBUG ] "quorum": [
[ceph-monitor][DEBUG ] 0
[ceph-monitor][DEBUG ] ],
[ceph-monitor][DEBUG ] "rank": 0,
[ceph-monitor][DEBUG ] "state": "leader",
[ceph-monitor][DEBUG ] "sync_provider": []
[ceph-monitor][DEBUG ] }
[ceph-monitor][DEBUG ] ********************************************************************************
[ceph-monitor][INFO ] monitor: mon.ceph-monitor is running
[ceph-monitor][INFO ] Running command: ceph --cluster=ceph --admin-daemon /var/run/ceph/ceph-mon.ceph-monitor.asok mon_status
[ceph_deploy.mon][INFO ] processing monitor mon.ceph-monitor
[ceph-monitor][DEBUG ] connected to host: ceph-monitor
[ceph-monitor][DEBUG ] detect platform information from remote host
[ceph-monitor][DEBUG ] detect machine type
[ceph-monitor][DEBUG ] find the location of an executable
[ceph-monitor][INFO ] Running command: ceph --cluster=ceph --admin-daemon /var/run/ceph/ceph-mon.ceph-monitor.asok mon_status
[ceph_deploy.mon][INFO ] mon.ceph-monitor monitor has reached quorum!
[ceph_deploy.mon][INFO ] all initial monitors are running and have formed quorum
[ceph_deploy.mon][INFO ] Running gatherkeys...
[ceph_deploy.gatherkeys][INFO ] Storing keys in temp directory /tmp/tmphUcJ_k
[ceph-monitor][DEBUG ] connected to host: ceph-monitor
[ceph-monitor][DEBUG ] detect platform information from remote host
[ceph-monitor][DEBUG ] detect machine type
[ceph-monitor][DEBUG ] get remote short hostname
[ceph-monitor][DEBUG ] fetch remote file
[ceph-monitor][INFO ] Running command: /usr/bin/ceph --connect-timeout=25 --cluster=ceph --admin-daemon=/var/run/ceph/ceph-mon.ceph-monitor.asok mon_status
[ceph-monitor][INFO ] Running command: /usr/bin/ceph --connect-timeout=25 --cluster=ceph --name mon. --keyring=/var/lib/ceph/mon/ceph-ceph-monitor/keyring auth get client.admin
[ceph-monitor][INFO ] Running command: /usr/bin/ceph --connect-timeout=25 --cluster=ceph --name mon. --keyring=/var/lib/ceph/mon/ceph-ceph-monitor/keyring auth get client.bootstrap-mds
[ceph-monitor][INFO ] Running command: /usr/bin/ceph --connect-timeout=25 --cluster=ceph --name mon. --keyring=/var/lib/ceph/mon/ceph-ceph-monitor/keyring auth get client.bootstrap-mgr
[ceph-monitor][INFO ] Running command: /usr/bin/ceph --connect-timeout=25 --cluster=ceph --name mon. --keyring=/var/lib/ceph/mon/ceph-ceph-monitor/keyring auth get-or-create client.bootstrap-mgr mon allow profile bootstrap-mgr
[ceph-monitor][INFO ] Running command: /usr/bin/ceph --connect-timeout=25 --cluster=ceph --name mon. --keyring=/var/lib/ceph/mon/ceph-ceph-monitor/keyring auth get client.bootstrap-osd
[ceph-monitor][INFO ] Running command: /usr/bin/ceph --connect-timeout=25 --cluster=ceph --name mon. --keyring=/var/lib/ceph/mon/ceph-ceph-monitor/keyring auth get client.bootstrap-rgw
[ceph_deploy.gatherkeys][INFO ] Storing ceph.client.admin.keyring
[ceph_deploy.gatherkeys][INFO ] Storing ceph.bootstrap-mds.keyring
[ceph_deploy.gatherkeys][INFO ] Storing ceph.bootstrap-mgr.keyring
[ceph_deploy.gatherkeys][INFO ] keyring 'ceph.mon.keyring' already exists
[ceph_deploy.gatherkeys][INFO ] Storing ceph.bootstrap-osd.keyring
[ceph_deploy.gatherkeys][INFO ] Storing ceph.bootstrap-rgw.keyring
[ceph_deploy.gatherkeys][INFO ] Destroy temp directory /tmp/tmphUcJ_k
执行上述命令后，生成以下密钥文件
[root@ceph-admin ceph]# ls -lrt total 48 -rw-r--r-- 1 root root 92 Jul 10 2018 rbdmap -rw------- 1 root root 73 Sep 27 22:59 ceph.mon.keyring -rw-r--r-- 1 root root 292 Sep 27 23:04 ceph.conf -rw------- 1 root root 129 Sep 27 23:06 ceph.client.admin.keyring -rw------- 1 root root 113 Sep 27 23:06 ceph.bootstrap-mds.keyring -rw------- 1 root root 71 Sep 27 23:06 ceph.bootstrap-mgr.keyring -rw------- 1 root root 113 Sep 27 23:06 ceph.bootstrap-osd.keyring -rw------- 1 root root 113 Sep 27 23:06 ceph.bootstrap-rgw.keyring -rw-r--r-- 1 root root 15385 Sep 27 23:06 ceph-deploy-ceph.log [root@ceph-admin ceph]#
3）部署osd服务
#准备osd
[root@ceph-admin ceph]# ceph-deploy osd prepare ceph-admin:/dev/sdb [root@ceph-admin ceph]# ceph-deploy osd prepare ceph-monitor:/dev/sdb [root@ceph-admin ceph]# ceph-deploy osd prepare ceph-osd:/dev/sdb
#激活osd
[root@ceph-admin ceph]# ceph-deploy osd activate ceph-admin:/dev/sdb [root@ceph-admin ceph]# ceph-deploy osd activate ceph-monitor:/dev/sdb [root@ceph-admin ceph]# ceph-deploy osd activate ceph-osd:/dev/sdb
#查看状态
[root@ceph-admin ceph]# ceph-deploy osd list ceph-admin ceph-monitor ceph-osd
[ceph_deploy.conf][DEBUG ] found configuration file at: /root/.cephdeploy.conf
[ceph_deploy.cli][INFO ] Invoked (1.5.39): /usr/bin/ceph-deploy osd list ceph-admin ceph-monitor ceph-osd
[ceph_deploy.cli][INFO ] ceph-deploy options:
[ceph_deploy.cli][INFO ] username : None
[ceph_deploy.cli][INFO ] verbose : False
[ceph_deploy.cli][INFO ] overwrite_conf : False
[ceph_deploy.cli][INFO ] subcommand : list
[ceph_deploy.cli][INFO ] quiet : False
[ceph_deploy.cli][INFO ] cd_conf : <ceph_deploy.conf.cephdeploy.Conf instance at 0x7f6818351680>
[ceph_deploy.cli][INFO ] cluster : ceph
[ceph_deploy.cli][INFO ] func : <function osd at 0x7f68183a3140>
[ceph_deploy.cli][INFO ] ceph_conf : None
[ceph_deploy.cli][INFO ] default_release : False
[ceph_deploy.cli][INFO ] disk : [('ceph-admin', None, None), ('ceph-monitor', None, None), ('ceph-osd', None, None)]
[ceph-monitor][DEBUG ] connected to host: ceph-monitor
[ceph-monitor][DEBUG ] detect platform information from remote host
[ceph-monitor][DEBUG ] detect machine type
[ceph-monitor][DEBUG ] find the location of an executable
[ceph-monitor][DEBUG ] find the location of an executable
[ceph-monitor][INFO ] Running command: /bin/ceph --cluster=ceph osd tree --format=json
[ceph-admin][DEBUG ] connected to host: ceph-admin
[ceph-admin][DEBUG ] detect platform information from remote host
[ceph-admin][DEBUG ] detect machine type
[ceph-admin][DEBUG ] find the location of an executable
[ceph-admin][INFO ] Running command: /usr/sbin/ceph-disk list
[ceph-admin][INFO ] ----------------------------------------
[ceph-admin][INFO ] ceph-0
[ceph-admin][INFO ] ----------------------------------------
[ceph-admin][INFO ] Path /var/lib/ceph/osd/ceph-0
[ceph-admin][INFO ] ID 0
[ceph-admin][INFO ] Name osd.0
[ceph-admin][INFO ] Status up
[ceph-admin][INFO ] Reweight 1.0
[ceph-admin][INFO ] Magic ceph osd volume v026
[ceph-admin][INFO ] Journal_uuid 9b5756c6-2d0d-4ae6-8b55-cb965ed3d69e
[ceph-admin][INFO ] Active ok
[ceph-admin][INFO ] Device /dev/sdb1
[ceph-admin][INFO ] Whoami 0
[ceph-admin][INFO ] Journal path /dev/sdb2
[ceph-admin][INFO ] ----------------------------------------
[ceph-monitor][DEBUG ] connected to host: ceph-monitor
[ceph-monitor][DEBUG ] detect platform information from remote host
[ceph-monitor][DEBUG ] detect machine type
[ceph-monitor][DEBUG ] find the location of an executable
[ceph-monitor][INFO ] Running command: /usr/sbin/ceph-disk list
[ceph-monitor][INFO ] ----------------------------------------
[ceph-monitor][INFO ] ceph-1
[ceph-monitor][INFO ] ----------------------------------------
[ceph-monitor][INFO ] Path /var/lib/ceph/osd/ceph-1
[ceph-monitor][INFO ] ID 1
[ceph-monitor][INFO ] Name osd.1
[ceph-monitor][INFO ] Status up
[ceph-monitor][INFO ] Reweight 1.0
[ceph-monitor][INFO ] Magic ceph osd volume v026
[ceph-monitor][INFO ] Journal_uuid 655bc91d-f7d1-4145-b2c3-5d5f63c1a798
[ceph-monitor][INFO ] Active ok
[ceph-monitor][INFO ] Device /dev/sdb1
[ceph-monitor][INFO ] Whoami 1
[ceph-monitor][INFO ] Journal path /dev/sdb2
[ceph-monitor][INFO ] ----------------------------------------
[ceph-osd][DEBUG ] connected to host: ceph-osd
[ceph-osd][DEBUG ] detect platform information from remote host
[ceph-osd][DEBUG ] detect machine type
[ceph-osd][DEBUG ] find the location of an executable
[ceph-osd][INFO ] Running command: /usr/sbin/ceph-disk list
[ceph-osd][INFO ] ----------------------------------------
[ceph-osd][INFO ] ceph-2
[ceph-osd][INFO ] ----------------------------------------
[ceph-osd][INFO ] Path /var/lib/ceph/osd/ceph-2
[ceph-osd][INFO ] ID 2
[ceph-osd][INFO ] Name osd.2
[ceph-osd][INFO ] Status up
[ceph-osd][INFO ] Reweight 1.0
[ceph-osd][INFO ] Magic ceph osd volume v026
[ceph-osd][INFO ] Journal_uuid 7c5e1478-92ab-44d2-9445-620b0e535c71
[ceph-osd][INFO ] Active ok
[ceph-osd][INFO ] Device /dev/sdb1
[ceph-osd][INFO ] Whoami 2
[ceph-osd][INFO ] Journal path /dev/sdb2
[ceph-osd][INFO ] ----------------------------------------
[root@ceph-admin ceph]#
4）创建ceph文件系统
要使用Ceph文件系统，Ceph的存储集群里至少需要存在一个Ceph的元数据服务器(mds)
#创建mds
[root@ceph-admin ceph]# ceph-deploy mds create ceph-admin ceph-monitor ceph-osd
[ceph_deploy.conf][DEBUG ] found configuration file at: /root/.cephdeploy.conf
[ceph_deploy.cli][INFO ] Invoked (1.5.39): /usr/bin/ceph-deploy mds create ceph-admin ceph-monitor ceph-osd
[ceph_deploy.cli][INFO ] ceph-deploy options:
[ceph_deploy.cli][INFO ] username : None
[ceph_deploy.cli][INFO ] verbose : False
[ceph_deploy.cli][INFO ] overwrite_conf : False
[ceph_deploy.cli][INFO ] subcommand : create
[ceph_deploy.cli][INFO ] quiet : False
[ceph_deploy.cli][INFO ] cd_conf : <ceph_deploy.conf.cephdeploy.Conf instance at 0x7f88afbd68c0>
[ceph_deploy.cli][INFO ] cluster : ceph
[ceph_deploy.cli][INFO ] func : <function mds at 0x7f88afbad848>
[ceph_deploy.cli][INFO ] ceph_conf : None
[ceph_deploy.cli][INFO ] mds : [('ceph-admin', 'ceph-admin'), ('ceph-monitor', 'ceph-monitor'), ('ceph-osd', 'ceph-osd')]
[ceph_deploy.cli][INFO ] default_release : False
[ceph_deploy.mds][DEBUG ] Deploying mds, cluster ceph hosts ceph-admin:ceph-admin ceph-monitor:ceph-monitor ceph-osd:ceph-osd
[ceph-admin][DEBUG ] connected to host: ceph-admin
[ceph-admin][DEBUG ] detect platform information from remote host
[ceph-admin][DEBUG ] detect machine type
[ceph_deploy.mds][INFO ] Distro info: CentOS Linux 7.9.2009 Core
[ceph_deploy.mds][DEBUG ] remote host will use systemd
[ceph_deploy.mds][DEBUG ] deploying mds bootstrap to ceph-admin
[ceph-admin][DEBUG ] write cluster configuration to /etc/ceph/{cluster}.conf
[ceph-admin][WARNIN] mds keyring does not exist yet, creating one
[ceph-admin][DEBUG ] create a keyring file
[ceph-admin][DEBUG ] create path if it doesn't exist
[ceph-admin][INFO ] Running command: ceph --cluster ceph --name client.bootstrap-mds --keyring /var/lib/ceph/bootstrap-mds/ceph.keyring auth get-or-create mds.ceph-admin osd allow rwx mds allow mon allow profile mds -o /var/lib/ceph/mds/ceph-ceph-admin/keyring
[ceph-admin][INFO ] Running command: systemctl enable ceph-mds@ceph-admin
[ceph-admin][WARNIN] Created symlink from /etc/systemd/system/ceph-mds.target.wants/ceph-mds@ceph-admin.service to /usr/lib/systemd/system/ceph-mds@.service.
[ceph-admin][INFO ] Running command: systemctl start ceph-mds@ceph-admin
[ceph-admin][INFO ] Running command: systemctl enable ceph.target
[ceph-monitor][DEBUG ] connected to host: ceph-monitor
[ceph-monitor][DEBUG ] detect platform information from remote host
[ceph-monitor][DEBUG ] detect machine type
[ceph_deploy.mds][INFO ] Distro info: CentOS Linux 7.9.2009 Core
[ceph_deploy.mds][DEBUG ] remote host will use systemd
[ceph_deploy.mds][DEBUG ] deploying mds bootstrap to ceph-monitor
[ceph-monitor][DEBUG ] write cluster configuration to /etc/ceph/{cluster}.conf
[ceph-monitor][DEBUG ] create path if it doesn't exist
[ceph-monitor][INFO ] Running command: ceph --cluster ceph --name client.bootstrap-mds --keyring /var/lib/ceph/bootstrap-mds/ceph.keyring auth get-or-create mds.ceph-monitor osd allow rwx mds allow mon allow profile mds -o /var/lib/ceph/mds/ceph-ceph-monitor/keyring
[ceph-monitor][INFO ] Running command: systemctl enable ceph-mds@ceph-monitor
[ceph-monitor][WARNIN] Created symlink from /etc/systemd/system/ceph-mds.target.wants/ceph-mds@ceph-monitor.service to /usr/lib/systemd/system/ceph-mds@.service.
[ceph-monitor][INFO ] Running command: systemctl start ceph-mds@ceph-monitor
[ceph-monitor][INFO ] Running command: systemctl enable ceph.target
[ceph-osd][DEBUG ] connected to host: ceph-osd
[ceph-osd][DEBUG ] detect platform information from remote host
[ceph-osd][DEBUG ] detect machine type
[ceph_deploy.mds][INFO ] Distro info: CentOS Linux 7.9.2009 Core
[ceph_deploy.mds][DEBUG ] remote host will use systemd
[ceph_deploy.mds][DEBUG ] deploying mds bootstrap to ceph-osd
[ceph-osd][DEBUG ] write cluster configuration to /etc/ceph/{cluster}.conf
[ceph-osd][WARNIN] mds keyring does not exist yet, creating one
[ceph-osd][DEBUG ] create a keyring file
[ceph-osd][DEBUG ] create path if it doesn't exist
[ceph-osd][INFO ] Running command: ceph --cluster ceph --name client.bootstrap-mds --keyring /var/lib/ceph/bootstrap-mds/ceph.keyring auth get-or-create mds.ceph-osd osd allow rwx mds allow mon allow profile mds -o /var/lib/ceph/mds/ceph-ceph-osd/keyring
[ceph-osd][INFO ] Running command: systemctl enable ceph-mds@ceph-osd
[ceph-osd][WARNIN] Created symlink from /etc/systemd/system/ceph-mds.target.wants/ceph-mds@ceph-osd.service to /usr/lib/systemd/system/ceph-mds@.service.
[ceph-osd][INFO ] Running command: systemctl start ceph-mds@ceph-osd
[ceph-osd][INFO ] Running command: systemctl enable ceph.target
#查看ceph当前文件系统
[root@ceph-admin ceph]# ceph fs ls No filesystems enabled [root@ceph-admin ceph]#
一个cephfs至少要求两个librados存储池，一个为data，一个为metadata。当配置这两个存储池时，注意：
（1）为metadata pool设置较高级别的副本级别，因为metadata的损坏可能导致整个文件系统不用
（2）建议，metadata pool使用低延时存储，比如SSD，因为metadata会直接影响客户端的响应速度
#创建存储池
[root@ceph-admin ceph]# ceph osd pool create cephfs_data 128 pool 'cephfs_data' created [root@ceph-admin ceph]# ceph osd pool create cephfs_metadata 128 pool 'cephfs_metadata' created [root@ceph-admin ceph]#
关于创建存储池
确定 pg_num 取值是强制性的，因为不能自动计算。下面是几个常用的值：
*少于 5 个 OSD 时可把 pg_num 设置为 128
*OSD 数量在 5 到 10 个时，可把 pg_num 设置为 512
*OSD 数量在 10 到 50 个时，可把 pg_num 设置为 4096
*OSD 数量大于 50 时，得理解权衡方法、以及如何计算 pg_num 取值,自己计算 pg_num 取值时可借助 pgcalc 工具
随着 OSD 数量的增加，正确的 pg_num 取值变得更加重要，因为它显著地影响着集群的行为、以及出错时的数据持久性（即灾难性事件导致数据丢失的概率）。
#创建文件系统
[root@ceph-admin ceph]# ceph fs new test-cephfs cephfs_metadata cephfs_data new fs with metadata pool 2 and data pool 1
其中：new后的fsname 可自定义
#查看创建后的cephfs
[root@ceph-admin ceph]# ceph fs ls name: test-cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ]
#查看mds节点状态
[root@ceph-admin ceph]# ceph mds stat
e7: 1/1/1 up {0=ceph-osd=up:active}, 2 up:standby
其中，active是活跃的，另1个是处于热备份的状态
#查看集群状态
[root@ceph-admin ceph]# ceph -s
cluster 75bf70df-8562-4f70-bb28-db9376a1ce98
health HEALTH_OK
monmap e1: 1 mons at {ceph-monitor=10.0.0.129:6789/0}
election epoch 3, quorum 0 ceph-monitor
fsmap e7: 1/1/1 up {0=ceph-osd=up:active}, 2 up:standby
osdmap e21: 3 osds: 3 up, 3 in
flags sortbitwise,require_jewel_osds
pgmap v58: 320 pgs, 3 pools, 2068 bytes data, 20 objects
326 MB used, 45720 MB / 46046 MB avail
320 active+clean
[root@ceph-admin ceph]#
其中，HEALTH_OK表示ceph集群正常
四、Kubernetes挂载ceph rbd存储卷
kubernetes是支持Ceph存储系统创建的块存储、存储文件系统等。kubernetes要想使用ceph，需要在k8s的每个节点安装ceph-common。
1. 安装ceph-common
#k8s环境节点
[root@k8s-master1 ~]# kubectl get nodes NAME STATUS ROLES AGE VERSION k8s-master1 Ready control-plane,master 58d v1.20.6 k8s-node1 Ready worker 58d v1.20.6 k8s-node2 Ready worker 58d v1.20.6
#每个节点配置ceph.repo源
[root@ceph-admin ~]# scp /etc/yum.repos.d/ceph.repo 10.0.0.131:/etc/yum.repos.d/ceph.repo The authenticity of host '10.0.0.131 (10.0.0.131)' can't be established. ECDSA key fingerprint is SHA256:O2leSOvudbcqIRBokjf4cUtbvjzdf/Yl49VkIQGfLxE. ECDSA key fingerprint is MD5:de:41:d0:68:53:e3:08:09:b0:7a:55:2e:b6:1d:af:d3. Are you sure you want to continue connecting (yes/no)? yes Warning: Permanently added '10.0.0.131' (ECDSA) to the list of known hosts. root@10.0.0.131's password: ceph.repo 100% 681 42.5KB/s 00:00 #k8s的控制节点上查看ceph.repo [root@k8s-master1 ~]# ll /etc/yum.repos.d/ceph.repo -rw-r--r-- 1 root root 681 Sep 28 21:16 /etc/yum.repos.d/ceph.repo You have new mail in /var/spool/mail/root [root@k8s-master1 ~]# scp /etc/yum.repos.d/ceph.repo 10.0.0.132:/etc/yum.repos.d/ceph.repo ceph.repo 100% 681 511.5KB/s 00:00 [root@k8s-master1 ~]# scp /etc/yum.repos.d/ceph.repo 10.0.0.133:/etc/yum.repos.d/ceph.repo ceph.repo 100% 681 338.9KB/s 00:00 [root@k8s-master1 ~]#
#每个节点上安装ceph-common
[root@k8s-master1 ~]# yum install ceph-common -y [root@k8s-node1 ~]# yum install ceph-common -y [root@k8s-node2 ~]# yum install ceph-common -y
2. 拷贝ceph的配置文件
[root@ceph-admin ~]# scp /etc/ceph/* 10.0.0.131:/etc/ceph/ root@10.0.0.131's password: ceph.bootstrap-mds.keyring 100% 113 0.2KB/s 00:00 ceph.bootstrap-mgr.keyring 100% 71 3.5KB/s 00:00 ceph.bootstrap-osd.keyring 100% 113 31.5KB/s 00:00 ceph.bootstrap-rgw.keyring 100% 113 1.0KB/s 00:00 ceph.client.admin.keyring 100% 129 84.2KB/s 00:00 ceph.conf 100% 292 136.0KB/s 00:00 ceph-deploy-ceph.log 100% 93KB 144.0KB/s 00:00 ceph.mon.keyring 100% 73 0.1KB/s 00:00 rbdmap 100% 92 1.5KB/s 00:00 [root@k8s-master1 ~]# scp /etc/ceph/* 10.0.0.132:/etc/ceph/ ceph.bootstrap-mds.keyring 100% 113 30.2KB/s 00:00 ceph.bootstrap-mgr.keyring 100% 71 31.0KB/s 00:00 ceph.bootstrap-osd.keyring 100% 113 80.5KB/s 00:00 ceph.bootstrap-rgw.keyring 100% 113 44.5KB/s 00:00 ceph.client.admin.keyring 100% 129 92.3KB/s 00:00 ceph.conf 100% 292 215.9KB/s 00:00 ceph-deploy-ceph.log 100% 93KB 29.5MB/s 00:00 ceph.mon.keyring 100% 73 49.9KB/s 00:00 rbdmap 100% 92 65.6KB/s 00:00 [root@k8s-master1 ~]# scp /etc/ceph/* 10.0.0.133:/etc/ceph/ ceph.bootstrap-mds.keyring 100% 113 82.1KB/s 00:00 ceph.bootstrap-mgr.keyring 100% 71 47.8KB/s 00:00 ceph.bootstrap-osd.keyring 100% 113 105.3KB/s 00:00 ceph.bootstrap-rgw.keyring 100% 113 75.9KB/s 00:00 ceph.client.admin.keyring 100% 129 83.4KB/s 00:00 ceph.conf 100% 292 246.0KB/s 00:00 ceph-deploy-ceph.log 100% 93KB 5.9MB/s 00:00 ceph.mon.keyring 100% 73 27.7KB/s 00:00 rbdmap 100% 92 84.8KB/s 00:00 [root@k8s-master1 ~]#
3. 创建ceph rbd
[root@ceph-admin ~]# ceph osd pool create k8srbd1 6 pool 'k8srbd1' created You have new mail in /var/spool/mail/root [root@ceph-admin ~]# rbd create rbda -s 1024 -p k8srbd1 [root@ceph-admin ~]# rbd feature disable k8srbd1/rbda object-map fast-diff deep-flatten
4. 创建pod，挂载ceph rbd
[root@k8s-master1 ~]# mkdir ceph
[root@k8s-master1 ~]# cd ceph/
[root@k8s-master1 ceph]# vim rbd-pod.yaml
[root@k8s-master1 ceph]# cat rbd-pod.yaml
apiVersion: v1
kind: Pod
metadata:
name: rbd-pod
spec:
containers:
- image: nginx:latest
name: nginx
imagePullPolicy: IfNotPresent
volumeMounts:
- name: testrbd
mountPath: /usr/share/nginx/html
volumes:
- name: testrbd
rbd:
monitors:
- '10.0.0.130:6789'
- '10.0.0.129:6789'
- '10.0.0.128:6789'
pool: k8srbd1
image: rbda
fsType: xfs
readOnly: false
user: admin
keyring: /etc/ceph/ceph.client.admin.keyring
[root@k8s-master1 ceph]# kubectl apply -f rbd-pod.yaml
pod/rbd-pod created
[root@k8s-master1 ceph]# kubectl get pods
NAME READY STATUS RESTARTS AGE
nfs-client-provisioner-5d65b75f7-2tv4q 1/1 Running 10 2d23h
rbd-pod 0/1 ContainerCreating 0 4s
[root@k8s-master1 ceph]# kubectl get pods
NAME READY STATUS RESTARTS AGE
nfs-client-provisioner-5d65b75f7-2tv4q 1/1 Running 10 2d23h
rbd-pod 1/1 Running 0 47s
[root@k8s-master1 ceph]#
rbd-pod已经创建成功，可以看到k8srbd1下的rbda被rbd-pod挂载了。通过以下命令，可以可能到rbd-pod的详情
[root@k8s-master1 ceph]# kubectl describe pods rbd-pod
Name: rbd-pod
Namespace: default
Priority: 0
Node: k8s-node1/10.0.0.132
Start Time: Wed, 28 Sep 2022 21:30:26 +0800
Labels: <none>
Annotations: cni.projectcalico.org/podIP: 10.244.36.99/32
cni.projectcalico.org/podIPs: 10.244.36.99/32
Status: Running
IP: 10.244.36.99
IPs:
IP: 10.244.36.99
Containers:
nginx:
Container ID: docker://e62cd5f3e715d1ad2f47cfbaf8b61583c700dd11cca03a884c637b11ad3047c3
Image: nginx:latest
Image ID: docker-pullable://nginx@sha256:b95a99feebf7797479e0c5eb5ec0bdfa5d9f504bc94da550c2f58e839ea6914f
Port: <none>
Host Port: <none>
State: Running
Started: Wed, 28 Sep 2022 21:30:33 +0800
Ready: True
Restart Count: 0
Environment: <none>
Mounts:
/usr/share/nginx/html from testrbd (rw)
/var/run/secrets/kubernetes.io/serviceaccount from default-token-5n29f (ro)
Conditions:
Type Status
Initialized True
Ready True
ContainersReady True
PodScheduled True
Volumes:
testrbd:
Type: RBD (a Rados Block Device mount on the host that shares a pod's lifetime)
CephMonitors: [10.0.0.130:6789 10.0.0.129:6789 10.0.0.128:6789]
RBDImage: rbda
FSType: xfs
RBDPool: k8srbd1
RadosUser: admin
Keyring: /etc/ceph/ceph.client.admin.keyring
SecretRef: nil
ReadOnly: false
default-token-5n29f:
Type: Secret (a volume populated by a Secret)
SecretName: default-token-5n29f
Optional: false
QoS Class: BestEffort
Node-Selectors: <none>
Tolerations: node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 3m8s default-scheduler Successfully assigned default/rbd-pod to k8s-node1
Normal SuccessfulAttachVolume 3m9s attachdetach-controller AttachVolume.Attach succeeded for volume "testrbd"
Normal Pulled 3m3s kubelet Container image "nginx:latest" already present on machine
Normal Created 3m3s kubelet Created container nginx
Normal Started 3m2s kubelet Started container nginx
验证其他pod是否能还能挂载这个k8srbd1下的rbda，再创建一个rbd-pod1.yaml清单文件
[root@k8s-master1 ceph]# vim rbd-pod1.yaml
[root@k8s-master1 ceph]# cat rbd-pod1.yaml
apiVersion: v1
kind: Pod
metadata:
name: rbd-pod1
spec:
containers:
- image: nginx:latest
name: nginx
imagePullPolicy: IfNotPresent
volumeMounts:
- name: testrbd
mountPath: /usr/share/nginx/html
volumes:
- name: testrbd
rbd:
monitors:
- '10.0.0.130:6789'
- '10.0.0.129:6789'
- '10.0.0.128:6789'
pool: k8srbd1
image: rbda
fsType: xfs
readOnly: false
user: admin
keyring: /etc/ceph/ceph.client.admin.keyring
[root@k8s-master1 ceph]# kubectl apply -f rbd-pod1.yaml
pod/rbd-pod1 created
[root@k8s-master1 ceph]# kubectl get pods |grep rbd-pod
rbd-pod 1/1 Running 0 8m39s
rbd-pod1 0/1 ContainerCreating 0 11s
[root@k8s-master1 ceph]# kubectl get pods |grep rbd-pod
rbd-pod 1/1 Running 0 13m
rbd-pod1 0/1 ContainerCreating 0 5m14s
可以看到rbd-pod1一直在创建中，使用kubectl describe pods rbd-pod1查看详细信息：
[root@k8s-master1 ~]# kubectl describe pods rbd-pod1
Name: rbd-pod1
Namespace: default
Priority: 0
Node: k8s-node2/10.0.0.133
Start Time: Wed, 28 Sep 2022 21:38:54 +0800
Labels: <none>
Annotations: <none>
Status: Pending
IP:
IPs: <none>
Containers:
nginx:
Container ID:
Image: nginx:latest
Image ID:
Port: <none>
Host Port: <none>
State: Waiting
Reason: ContainerCreating
Ready: False
Restart Count: 0
Environment: <none>
Mounts:
/usr/share/nginx/html from testrbd (rw)
/var/run/secrets/kubernetes.io/serviceaccount from default-token-5n29f (ro)
Conditions:
Type Status
Initialized True
Ready False
ContainersReady False
PodScheduled True
Volumes:
testrbd:
Type: RBD (a Rados Block Device mount on the host that shares a pod's lifetime)
CephMonitors: [10.0.0.130:6789 10.0.0.129:6789 10.0.0.128:6789]
RBDImage: rbda
FSType: xfs
RBDPool: k8srbd1
RadosUser: admin
Keyring: /etc/ceph/ceph.client.admin.keyring
SecretRef: nil
ReadOnly: false
default-token-5n29f:
Type: Secret (a volume populated by a Secret)
SecretName: default-token-5n29f
Optional: false
QoS Class: BestEffort
Node-Selectors: <none>
Tolerations: node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 99s default-scheduler Successfully assigned default/rbd-pod1 to k8s-node2
Normal SuccessfulAttachVolume 99s attachdetach-controller AttachVolume.Attach succeeded for volume "testrbd"
Warning FailedMount 21s kubelet MountVolume.WaitForAttach failed for volume "testrbd" : rbd image k8srbd1/rbda is still being used
通过warnning可以发现是因为pool: k8srbd1的rbda已经被其他pod占用了
五、基于ceph rbd生成pv
1. 创建ceph-secret
创建k8s secret对象ceph-secret，这个secret对象用于k8s volume插件访问ceph集群，获取client.admin的keyring值，并用base64编码，在ceph-admin（ceph管理节点）操作
[root@ceph-admin ~]# ceph auth get-key client.admin | base64 QVFDR0VUTmpFcE0rTGhBQWRCbWkzbEw3YVNzMmlRZDQ5Z2NEWGc9PQ==
创建ceph的secret
[root@k8s-master1 ceph]# vim ceph-secret.yaml [root@k8s-master1 ceph]# cat ceph-secret.yaml apiVersion: v1 kind: Secret metadata: name: ceph-secret data: key: QVFDR0VUTmpFcE0rTGhBQWRCbWkzbEw3YVNzMmlRZDQ5Z2NEWGc9PQ== [root@k8s-master1 ceph]# kubectl apply -f ceph-secret.yaml secret/ceph-secret created [root@k8s-master1 ceph]# kubectl get secret NAME TYPE DATA AGE ceph-secret Opaque 1 7s
2. 创建pool池
[root@ceph-admin ~]# ceph osd pool create k8stest 6 pool 'k8stest' created You have new mail in /var/spool/mail/root [root@ceph-admin ~]# rbd create rbda -s 1024 -p k8stest [root@ceph-admin ~]# rbd feature disable k8stest/rbda object-map fast-diff deep-flatten [root@ceph-admin ~]# ceph osd lspools 0 rbd,1 cephfs_data,2 cephfs_metadata,3 k8srbd1,4 k8stest,
3. 创建PV
[root@k8s-master1 ceph]# vim ceph-pv.yaml
[root@k8s-master1 ceph]# cat ceph-pv.yaml
apiVersion: v1
kind: PersistentVolume
metadata:
name: ceph-pv
spec:
capacity:
storage: 1Gi
accessModes:
- ReadWriteOnce
rbd:
monitors:
- '10.0.0.130:6789'
- '10.0.0.129:6789'
- '10.0.0.128:6789'
pool: k8stest
image: rbda
user: admin
secretRef:
name: ceph-secret
fsType: xfs
readOnly: false
persistentVolumeReclaimPolicy: Recycle
[root@k8s-master1 ceph]# kubectl apply -f ceph-pv.yaml
persistentvolume/ceph-pv created
[root@k8s-master1 ceph]# kubectl get pv
NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGE
ceph-pv 1Gi RWO Recycle Available 4s
4. 创建PVC
[root@k8s-master1 ceph]# vim ceph-pvc.yaml
[root@k8s-master1 ceph]# cat ceph-pvc.yaml
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
name: ceph-pvc
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 1Gi
[root@k8s-master1 ceph]# kubectl apply -f ceph-pvc.yaml
persistentvolumeclaim/ceph-pvc created
[root@k8s-master1 ceph]# kubectl get pvc
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
ceph-pvc Bound ceph-pv 1Gi RWO 5s
创建的PVC自动与之前创建的ceph-pv绑定
5. 挂载PVC
[root@k8s-master1 ceph]# vim ceph-deploy.yaml
[root@k8s-master1 ceph]# cat ceph-deploy.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: ceph-deploy
spec:
selector:
matchLabels:
app: nginx
replicas: 2
template:
metadata:
labels:
app: nginx
spec:
containers:
- name: nginx
image: nginx:latest
imagePullPolicy: IfNotPresent
ports:
- containerPort: 80
volumeMounts:
- mountPath: "/ceph-data"
name: ceph-data
volumes:
- name: ceph-data
persistentVolumeClaim:
claimName: ceph-pvc
[root@k8s-master1 ceph]# kubectl apply -f ceph-deploy.yaml
deployment.apps/ceph-deploy created
[root@k8s-master1 ceph]# kubectl get pods
NAME READY STATUS RESTARTS AGE
ceph-deploy-5bb47d859f-chtpr 0/1 ContainerCreating 0 5s
ceph-deploy-5bb47d859f-fdp7w 0/1 ContainerCreating 0 5s
[root@k8s-master1 ceph]# kubectl get pods
NAME READY STATUS RESTARTS AGE
ceph-deploy-5bb47d859f-chtpr 1/1 Running 0 46s
ceph-deploy-5bb47d859f-fdp7w 1/1 Running 0 46s
可以看到有deployment创建的两个pod正常运行，而且它们均调度到k8s-node1节点上，正常挂载ceph-pvc。所以可以发现ceph rbd块存储能在同一个node上跨pod以ReadWriteOnce共享挂载。
[root@k8s-master1 ceph]# kubectl get pods -o wide NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES ceph-deploy-5bb47d859f-chtpr 1/1 Running 0 4m38s 10.244.36.103 k8s-node1 <none> <none> ceph-deploy-5bb47d859f-fdp7w 1/1 Running 0 4m38s 10.244.36.91 k8s-node1 <none> <none>
再创建一个调度到不同节点上的pod，挂载ceph-pvc，测试是否能挂载成功
[root@k8s-master1 ceph]# vim ceph-deploy-test.yaml
You have new mail in /var/spool/mail/root
[root@k8s-master1 ceph]# cat ceph-deploy-test.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: ceph-deploy-test
spec:
selector:
matchLabels:
app: nginx
replicas: 1
template:
metadata:
labels:
app: nginx
spec:
affinity:
podAntiAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
- labelSelector:
matchLabels:
app: nginx
topologyKey: kubernetes.io/hostname
containers:
- name: nginx
image: nginx:latest
imagePullPolicy: IfNotPresent
ports:
- containerPort: 80
volumeMounts:
- mountPath: "/ceph-data"
name: ceph-data
volumes:
- name: ceph-data
persistentVolumeClaim:
claimName: ceph-pvc
[root@k8s-master1 ceph]# kubectl apply -f ceph-deploy-test.yaml
deployment.apps/ceph-deploy-test created
[root@k8s-master1 ceph]# kubectl get pods -o wide |grep ceph-deploy-test
ceph-deploy-test-6c88bffc9d-rsgzk 0/1 ContainerCreating 0 113s <none> k8s-node2 <none> <none>
可以看到调度到k8s-node2节点上的ceph-deploy-test-6c88bffc9d-rsgzk一直处于异常状态，通过以下命令查看详情发现，pod是挂载失败
[root@k8s-master1 ceph]# kubectl describe pods ceph-deploy-test-6c88bffc9d-rsgzk
Name: ceph-deploy-test-6c88bffc9d-rsgzk
Namespace: default
Priority: 0
Node: k8s-node2/10.0.0.133
Start Time: Wed, 28 Sep 2022 22:50:33 +0800
Labels: app=nginx
pod-template-hash=6c88bffc9d
Annotations: <none>
Status: Pending
IP:
IPs: <none>
Controlled By: ReplicaSet/ceph-deploy-test-6c88bffc9d
Containers:
nginx:
Container ID:
Image: nginx:latest
Image ID:
Port: 80/TCP
Host Port: 0/TCP
State: Waiting
Reason: ContainerCreating
Ready: False
Restart Count: 0
Environment: <none>
Mounts:
/ceph-data from ceph-data (rw)
/var/run/secrets/kubernetes.io/serviceaccount from default-token-5n29f (ro)
Conditions:
Type Status
Initialized True
Ready False
ContainersReady False
PodScheduled True
Volumes:
ceph-data:
Type: PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
ClaimName: ceph-pvc
ReadOnly: false
default-token-5n29f:
Type: Secret (a volume populated by a Secret)
SecretName: default-token-5n29f
Optional: false
QoS Class: BestEffort
Node-Selectors: <none>
Tolerations: node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 4m27s default-scheduler Successfully assigned default/ceph-deploy-test-6c88bffc9d-rsgzk to k8s-node2
Warning FailedAttachVolume 4m27s attachdetach-controller Multi-Attach error for volume "ceph-pv" Volume is already used by pod(s) ceph-deploy-5bb47d859f-chtpr, ceph-deploy-5bb47d859f-fdp7w
Warning FailedMount 2m24s kubelet Unable to attach or mount volumes: unmounted volumes=[ceph-data], unattached volumes=[ceph-data default-token-5n29f]: timed out waiting for the condition
Warning FailedMount 8s kubelet Unable to attach or mount volumes: unmounted volumes=[ceph-data], unattached volumes=[default-token-5n29f ceph-data]: timed out waiting for the condition
6. 总结
通过以上实验，可以得出ceph rbd块存储的特点：
1）ceph rbd块存储能在同一个node上同一个pod多个容器中以ReadWriteOnce共享挂载
2）ceph rbd块存储能在同一个node上跨pod以ReadWriteOnce共享挂载
3）ceph rbd块存储不能跨node以ReadWriteOnce共享挂载
如果一个使用ceph rdb的pod所在的node挂掉，这个pod虽然会被调度到其它node，但是由于rbd不能跨node多次挂载和挂掉的pod不能自动解绑pv的问题，这个新pod不会正常运行
解决办法：
1）使用能支持跨node和pod之间挂载的共享存储，例如cephfs，GlusterFS等
2）给node添加label，只允许deployment所管理的pod调度到一个固定的node上。（不建议，这个node挂掉的话，服务就故障了）
六、基于Cephfs创建pv
cephfs方式支持k8s的pv的3种访问模式ReadWriteOnce，ReadOnlyMany ，ReadWriteMany
1. 查看ceph文件系统
[root@ceph-admin ~]# ceph fs ls name: test-cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ]
2. 创建ceph子目录
为了能挂载cephfs，先创建一个secretfile
[root@ceph-admin ~]# cat /etc/ceph/ceph.client.admin.keyring |grep key|awk -F" " '{print $3}' > /etc/ceph/admin.secret
[root@ceph-admin ~]# cat /etc/ceph/admin.secret
AQCGETNjEpM+LhAAdBmi3lL7aSs2iQd49gcDXg==
[root@ceph-admin ~]#
挂载cephfs的根目录到集群的mon节点下的一个目录，比如test_data，因为挂载后，就可以直接在test_data下面用Linux命令创建子目录
[root@ceph-admin ~]# mkdir test_data You have new mail in /var/spool/mail/root [root@ceph-admin ~]# mount -t ceph 10.0.0.129:6789:/ /root/test_data -o name=admin,secretfile=/etc/ceph/admin.secret [root@ceph-admin ~]# df -hT Filesystem Type Size Used Avail Use% Mounted on devtmpfs devtmpfs 2.0G 0 2.0G 0% /dev tmpfs tmpfs 2.0G 0 2.0G 0% /dev/shm tmpfs tmpfs 2.0G 12M 2.0G 1% /run tmpfs tmpfs 2.0G 0 2.0G 0% /sys/fs/cgroup /dev/sda2 xfs 40G 3.5G 37G 9% / /dev/sda1 xfs 473M 140M 333M 30% /boot /dev/sdb1 xfs 15G 123M 15G 1% /var/lib/ceph/osd/ceph-0 tmpfs tmpfs 394M 0 394M 0% /run/user/0 10.0.0.129:6789:/ ceph 45G 372M 45G 1% /root/test_data [root@ceph-admin ~]#
在cephfs的根目录里面创建了一个子目录lucky，k8s以后就可以挂载这个目录
[root@ceph-admin ~]# cd test_data/ You have new mail in /var/spool/mail/root [root@ceph-admin test_data]# mkdir lucky [root@ceph-admin test_data]# chmod 0777 lucky/ [root@ceph-admin test_data]# ll -a total 0 drwxr-xr-x 1 root root 1 Sep 29 22:32 . dr-xr-x---. 5 root root 216 Sep 29 22:29 .. drwxrwxrwx 1 root root 0 Sep 29 22:32 lucky
3. 创建Secret对象
获取client.admin的keyring值，并用base64编码
[root@ceph-admin test_data]# ceph auth get-key client.admin | base64 QVFDR0VUTmpFcE0rTGhBQWRCbWkzbEw3YVNzMmlRZDQ5Z2NEWGc9PQ==
创建cephfs-secret这个secret对象，这个secret对象用于k8s volume插件访问cephfs文件系统
[root@k8s-master1 ceph]# vim cephfs-secret.yaml [root@k8s-master1 ceph]# cat cephfs-secret.yaml apiVersion: v1 kind: Secret metadata: name: cephfs-secret data: key: QVFDR0VUTmpFcE0rTGhBQWRCbWkzbEw3YVNzMmlRZDQ5Z2NEWGc9PQ== You have new mail in /var/spool/mail/root [root@k8s-master1 ceph]# kubectl apply -f cephfs-secret.yaml secret/cephfs-secret created [root@k8s-master1 ceph]# kubectl get secret |grep cephfs-secret cephfs-secret Opaque 1 21s
4. 创建PV
[root@k8s-master1 ceph]# vim cephfs-pv.yaml
You have new mail in /var/spool/mail/root
[root@k8s-master1 ceph]# cat cephfs-pv.yaml
apiVersion: v1
kind: PersistentVolume
metadata:
name: cephfs-pv
spec:
capacity:
storage: 1Gi
accessModes:
- ReadWriteMany
cephfs:
monitors:
- 10.0.0.129:6789
path: /lucky
user: admin
readOnly: false
secretRef:
name: cephfs-secret
persistentVolumeReclaimPolicy: Recycle
[root@k8s-master1 ceph]# kubectl apply -f cephfs-pv.yaml
persistentvolume/cephfs-pv created
[root@k8s-master1 ceph]# kubectl get pv
NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGE
ceph-pv 1Gi RWO Recycle Bound default/ceph-pvc 24h
cephfs-pv 1Gi RWX Recycle Available 23s
5. 创建PVC
[root@k8s-master1 ceph]# vim cephfs-pvc.yaml
[root@k8s-master1 ceph]# cat cephfs-pvc.yaml
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
name: cephfs-pvc
spec:
accessModes:
- ReadWriteMany
volumeName: cephfs-pv
resources:
requests:
storage: 1Gi
[root@k8s-master1 ceph]# kubectl apply -f cephfs-pvc.yaml
persistentvolumeclaim/cephfs-pvc created
[root@k8s-master1 ceph]# kubectl get pvc
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
ceph-pvc Bound ceph-pv 1Gi RWO 24h
cephfs-pvc Pending cephfs-pv 0 4s
[root@k8s-master1 ceph]# kubectl get pvc
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
ceph-pvc Bound ceph-pv 1Gi RWO 24h
cephfs-pvc Bound cephfs-pv 1Gi RWX 56s
发现创建的cephfs-pvc已经自动与cephfs-pv绑定
6. 创建pod挂载cephfs-pvc
[root@k8s-master1 ceph]# vim cephfs-pod-1.yaml
You have new mail in /var/spool/mail/root
[root@k8s-master1 ceph]# cat cephfs-pod-1.yaml
apiVersion: v1
kind: Pod
metadata:
name: cephfs-pod-1
spec:
containers:
- image: nginx:latest
name: nginx
imagePullPolicy: IfNotPresent
volumeMounts:
- name: test-v1
mountPath: /usr/share/nginx/html
volumes:
- name: test-v1
persistentVolumeClaim:
claimName: cephfs-pvc
[root@k8s-master1 ceph]# kubectl apply -f cephfs-pod-1.yaml
pod/cephfs-pod-1 created
[root@k8s-master1 ceph]# kubectl get pods
NAME READY STATUS RESTARTS AGE
cephfs-pod-1 1/1 Running 0 79s
登录到该pod中，执行以下命令：
[root@k8s-master1 ceph]# kubectl exec -it cephfs-pod-1 -- /bin/sh # cd /usr/share/nginx/html # ls # echo "hello world" >> index.html # echo "access to cephfs-pod-1" >>index.html # cat index.html hello world access to cephfs-pod-1 # exit
回到ceph-admin节点上，查看cephfs文件目录下是否有内容
[root@ceph-admin test_data]# ll lucky/ total 1 -rw-r--r-- 1 root root 35 Sep 29 22:52 index.html [root@ceph-admin test_data]# cat lucky/index.html hello world access to cephfs-pod-1 [root@ceph-admin test_data]# pwd /root/test_data
再创建一个pod，挂载该cephfs-pvc卷
[root@k8s-master1 ceph]# vim cephfs-pod-2.yaml
You have new mail in /var/spool/mail/root
[root@k8s-master1 ceph]# cat cephfs-pod-2.yaml
apiVersion: v1
kind: Pod
metadata:
name: cephfs-pod-2
spec:
containers:
- image: nginx
name: nginx
imagePullPolicy: IfNotPresent
volumeMounts:
- name: test-v2
mountPath: /usr/share/nginx/html
volumes:
- name: test-v2
persistentVolumeClaim:
claimName: cephfs-pvc
[root@k8s-master1 ceph]# kubectl apply -f cephfs-pod-2.yaml
pod/cephfs-pod-2 created
[root@k8s-master1 ceph]# kubectl get pods
NAME READY STATUS RESTARTS AGE
cephfs-pod-1 1/1 Running 0 7m53s
cephfs-pod-2 1/1 Running 0 4s
nfs-client-provisioner-5d65b75f7-2tv4q 1/1 Running 12 4d1h
rbd-pod 1/1 Running 1 25h
[root@k8s-master1 ceph]# kubectl get pods -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
cephfs-pod-1 1/1 Running 0 8m11s 10.244.169.164 k8s-node2 <none> <none>
cephfs-pod-2 1/1 Running 0 22s 10.244.36.109 k8s-node1 <none> <none>
访问cephfs-pod-2,并登录到该pod容器中，查看/usr/share/nginx/html该目录下是否有文件
[root@k8s-master1 ceph]# curl 10.244.36.109 hello world access to cephfs-pod-1 You have new mail in /var/spool/mail/root [root@k8s-master1 ceph]# kubectl exec -it cephfs-pod-2 -- /bin/sh # ls /usr/share/nginx/html index.html # cat /usr/share/nginx/html/index.html hello world access to cephfs-pod-1 # exit You have new mail in /var/spool/mail/root [root@k8s-master1 ceph]#
从上述实验中可以看出：调度到不同节点上的cephfs-pod-1和cephfs-pod-2可以共享cephfs-pvc卷
七、使用Ceph实现动态持久卷存储
动态供给能够自动帮忙创建pv，需要多大的空间就创建多大的pv。k8s帮助创建pv，创建pvc就直接api调用存储类来寻找pv。
如果是存储静态供给的话，需要手动去创建pv，如果没有足够的资源，找不到合适的pv，那么pod就会处于pending等待的状态。而动态供给主要的一个实现就是StorageClass存储对象，其实它就是声明使用哪个存储，然后帮忙去连接，再自动创建pv。
1. 使用Ceph RBD作为持久数据卷
1） 为Ceph配置文件赋权
在所有节点，ceph集群节点和k8s集群节点执行赋权操作：
[root@k8s-master1 ~]# chmod 777 -R /etc/ceph/* [root@k8s-master1 ~]# mkdir /root/.ceph/ [root@k8s-master1 ~]# cp -ar /etc/ceph/ /root/.ceph/
其他节点与k8s-master1节点操作相同
2）配置 rbd-provisioner
[root@k8s-master1 ceph]# vim external-storage-rbd-provisioner.yaml
You have new mail in /var/spool/mail/root
[root@k8s-master1 ceph]# cat external-storage-rbd-provisioner.yaml
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: rbd-provisioner
rules:
- apiGroups: [""]
resources: ["persistentvolumes"]
verbs: ["get", "list", "watch", "create", "delete"]
- apiGroups: [""]
resources: ["persistentvolumeclaims"]
verbs: ["get", "list", "watch", "update"]
- apiGroups: ["storage.k8s.io"]
resources: ["storageclasses"]
verbs: ["get", "list", "watch"]
- apiGroups: [""]
resources: ["events"]
verbs: ["create", "update", "patch"]
- apiGroups: [""]
resources: ["services"]
resourceNames: ["kube-dns","coredns"]
verbs: ["list", "get"]
---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: rbd-provisioner
subjects:
- kind: ServiceAccount
name: rbd-provisioner
namespace: default
roleRef:
kind: ClusterRole
name: rbd-provisioner
apiGroup: rbac.authorization.k8s.io
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
name: rbd-provisioner
rules:
- apiGroups: [""]
resources: ["secrets"]
verbs: ["get"]
- apiGroups: [""]
resources: ["endpoints"]
verbs: ["get", "list", "watch", "create", "update", "patch"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
name: rbd-provisioner
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: Role
name: rbd-provisioner
subjects:
- kind: ServiceAccount
name: rbd-provisioner
namespace: default
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: rbd-provisioner
spec:
selector:
matchLabels:
app: rbd-provisioner
replicas: 1
strategy:
type: Recreate
template:
metadata:
labels:
app: rbd-provisioner
spec:
containers:
- name: rbd-provisioner
image: quay.io/external_storage/rbd-provisioner:latest
imagePullPolicy: IfNotPresent
env:
- name: PROVISIONER_NAME
value: ceph.com/rbd
serviceAccount: rbd-provisioner
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: rbd-provisioner
[root@k8s-master1 ceph]# kubectl apply -f external-storage-rbd-provisioner.yaml
clusterrole.rbac.authorization.k8s.io/rbd-provisioner created
clusterrolebinding.rbac.authorization.k8s.io/rbd-provisioner created
role.rbac.authorization.k8s.io/rbd-provisioner created
rolebinding.rbac.authorization.k8s.io/rbd-provisioner created
deployment.apps/rbd-provisioner created
serviceaccount/rbd-provisioner created
[root@k8s-master1 ceph]# kubectl get pods -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
rbd-provisioner-65bf77fd59-9jnm8 1/1 Running 0 11s 10.244.36.114 k8s-node1 <none> <none>
3）创建ceph-secret
#创建pool池
[root@ceph-admin ~]# ceph osd pool create k8stest1 6 pool 'k8stest1' created [root@ceph-admin ~]# ceph osd lspools 0 rbd,1 cephfs_data,2 cephfs_metadata,3 k8srbd1,4 k8stest,5 k8stest1,
创建secret资源
[root@k8s-master1 ceph]# vim ceph-secret-1.yaml You have new mail in /var/spool/mail/root [root@k8s-master1 ceph]# cat ceph-secret-1.yaml apiVersion: v1 kind: Secret metadata: name: ceph-secret-1 type: "ceph.com/rbd" data: key: QVFDR0VUTmpFcE0rTGhBQWRCbWkzbEw3YVNzMmlRZDQ5Z2NEWGc9PQ== [root@k8s-master1 ceph]# kubectl apply -f ceph-secret-1.yaml secret/ceph-secret-1 created [root@k8s-master1 ceph]# kubectl get secret NAME TYPE DATA AGE ceph-secret-1 ceph.com/rbd 1 12s default-token-5n29f kubernetes.io/service-account-token 3 60d rbd-provisioner-token-qbtmt kubernetes.io/service-account-token 3 11m
4）配置 storageclass
[root@k8s-master1 ceph]# vim rbd-storageclass.yaml You have new mail in /var/spool/mail/root [root@k8s-master1 ceph]# cat rbd-storageclass.yaml apiVersion: storage.k8s.io/v1 kind: StorageClass metadata: name: rbd-storageclass provisioner: ceph.com/rbd parameters: monitors: 10.0.0.130:6789,10.0.0.129:6789,10.0.0.128:6789 adminId: admin adminSecretName: ceph-secret-1 pool: k8stest1 userId: admin userSecretName: ceph-secret-1 fsType: xfs imageFormat: "2" imageFeatures: "layering" [root@k8s-master1 ceph]# kubectl apply -f rbd-storageclass.yaml storageclass.storage.k8s.io/rbd-storageclass created [root@k8s-master1 ceph]# kubectl get sc NAME PROVISIONER RECLAIMPOLICY VOLUMEBINDINGMODE ALLOWVOLUMEEXPANSION AGE rbd-storageclass ceph.com/rbd Delete Immediate false 6s
5）创建PVC
[root@k8s-master1 ceph]# vim rbd-pvc.yaml
[root@k8s-master1 ceph]# cat rbd-pvc.yaml
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
name: rbd-pvc
spec:
accessModes:
- ReadWriteOnce
volumeMode: Filesystem
resources:
requests:
storage: 1Gi
storageClassName: rbd-storageclass
[root@k8s-master1 ceph]# kubectl apply -f rbd-pvc.yaml
persistentvolumeclaim/rbd-pvc created
[root@k8s-master1 ceph]# kubectl get pvc
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
rbd-pvc Pending rbd-storageclass 6s
发现创建的rbd-pvc一直处于pending状态，用以下命令查看pvc详情
[root@k8s-master1 ceph]# kubectl describe pvc rbd-pvc Name: rbd-pvc Namespace: default StorageClass: rbd-storageclass Status: Pending Volume: Labels: <none> Annotations: volume.beta.kubernetes.io/storage-provisioner: ceph.com/rbd Finalizers: [kubernetes.io/pvc-protection] Capacity: Access Modes: VolumeMode: Filesystem Used By: <none> Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal ExternalProvisioning 11s (x3 over 30s) persistentvolume-controller waiting for a volume to be created, either by external provisioner "ceph.com/rbd" or manually created by system administrator
发现找不到external provisioner "ceph.com/rbd"，查看rbd-provisioner-65bf77fd59-9jnm8 日志
[root@k8s-master1 ceph]# kubectl logs rbd-provisioner-65bf77fd59-9jnm8
I0930 13:26:50.323553 1 main.go:85] Creating RBD provisioner ceph.com/rbd with identity: ceph.com/rbd
I0930 13:26:50.325012 1 leaderelection.go:185] attempting to acquire leader lease default/ceph.com-rbd...
E0930 13:26:50.336351 1 event.go:259] Could not construct reference to: '&v1.Endpoints{TypeMeta:v1.TypeMeta{Kind:"", APIVersion:""}, ObjectMeta:v1.ObjectMeta{Name:"ceph.com-rbd", GenerateName:"", Namespace:"default", SelfLink:"", UID:"e2fc1dc4-aa04-4667-b179-051d1b2721cb", ResourceVersion:"544566", Generation:0, CreationTimestamp:v1.Time{Time:time.Time{wall:0x0, ext:63800141210, loc:(*time.Location)(0x1bc94e0)}}, DeletionTimestamp:(*v1.Time)(nil), DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string(nil), Annotations:map[string]string{"control-plane.alpha.kubernetes.io/leader":"{\"holderIdentity\":\"rbd-provisioner-65bf77fd59-9jnm8_8a68381c-40c3-11ed-b978-befb50ef5a7c\",\"leaseDurationSeconds\":15,\"acquireTime\":\"2022-09-30T13:26:50Z\",\"renewTime\":\"2022-09-30T13:26:50Z\",\"leaderTransitions\":0}"}, OwnerReferences:[]v1.OwnerReference(nil), Initializers:(*v1.Initializers)(nil), Finalizers:[]string(nil), ClusterName:""}, Subsets:[]v1.EndpointSubset(nil)}' due to: 'selfLink was empty, can't make reference'. Will not report event: 'Normal' 'LeaderElection' 'rbd-provisioner-65bf77fd59-9jnm8_8a68381c-40c3-11ed-b978-befb50ef5a7c became leader'
I0930 13:26:50.336551 1 leaderelection.go:194] successfully acquired lease default/ceph.com-rbd
I0930 13:26:50.336868 1 controller.go:631] Starting provisioner controller ceph.com/rbd_rbd-provisioner-65bf77fd59-9jnm8_8a68381c-40c3-11ed-b978-befb50ef5a7c!
I0930 13:26:50.438154 1 controller.go:680] Started provisioner controller ceph.com/rbd_rbd-provisioner-65bf77fd59-9jnm8_8a68381c-40c3-11ed-b978-befb50ef5a7c!
I0930 13:49:52.814754 1 controller.go:987] provision "default/rbd-pvc" class "rbd-storageclass": started
E0930 13:49:52.822329 1 controller.go:1004] provision "default/rbd-pvc" class "rbd-storageclass": unexpected error getting claim reference: selfLink was empty, can't make reference
这是因为使用的是1.20版本的k8s集群，它通过rbd provisioner动态生成pv会报如上错误。解决方法：修改/etc/kubernetes/manifests/kube-apiserver.yaml配置文件，添加- --feature-gates=RemoveSelfLink=false 一行这样的参数，如下：
添加完成后，保存退出。重启kubelet服务，查看kube-system空间下的组件正常运行，即可。
[root@k8s-master1 ceph]# systemctl restart kubelet
You have new mail in /var/spool/mail/root
[root@k8s-master1 ceph]# systemctl status kubelet
● kubelet.service - kubelet: The Kubernetes Node Agent
Loaded: loaded (/usr/lib/systemd/system/kubelet.service; enabled; vendor preset: disabled)
Drop-In: /usr/lib/systemd/system/kubelet.service.d
└─10-kubeadm.conf
Active: active (running) since Fri 2022-09-30 22:00:44 CST; 8s ago
Docs: https://kubernetes.io/docs/
Main PID: 71245 (kubelet)
Memory: 43.3M
CGroup: /system.slice/kubelet.service
└─71245 /usr/bin/kubelet --bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf --kubeconfig=/etc/kubernetes/kubelet.conf --config=/var/lib/kub...
Sep 30 22:00:52 k8s-master1 kubelet[71245]: I0930 22:00:52.485710 71245 reconciler.go:224] operationExecutor.VerifyControllerAttachedVolume started fo...2156438")
Sep 30 22:00:52 k8s-master1 kubelet[71245]: I0930 22:00:52.485815 71245 reconciler.go:224] operationExecutor.VerifyControllerAttachedVolume started fo...2156438")
Sep 30 22:00:52 k8s-master1 kubelet[71245]: I0930 22:00:52.486011 71245 reconciler.go:224] operationExecutor.VerifyControllerAttachedVolume started fo...b621314")
Sep 30 22:00:52 k8s-master1 kubelet[71245]: I0930 22:00:52.486158 71245 reconciler.go:224] operationExecutor.VerifyControllerAttachedVolume started for volume ...
Sep 30 22:00:52 k8s-master1 kubelet[71245]: I0930 22:00:52.486238 71245 reconciler.go:224] operationExecutor.VerifyControllerAttachedVolume started for volume ...
Sep 30 22:00:52 k8s-master1 kubelet[71245]: I0930 22:00:52.486316 71245 reconciler.go:224] operationExecutor.VerifyControllerAttachedVolume started fo...5c91746")
Sep 30 22:00:52 k8s-master1 kubelet[71245]: I0930 22:00:52.486405 71245 reconciler.go:224] operationExecutor.VerifyControllerAttachedVolume started for volume ...
Sep 30 22:00:52 k8s-master1 kubelet[71245]: I0930 22:00:52.486485 71245 reconciler.go:224] operationExecutor.VerifyControllerAttachedVolume started fo...b621314")
Sep 30 22:00:52 k8s-master1 kubelet[71245]: I0930 22:00:52.486560 71245 reconciler.go:224] operationExecutor.VerifyControllerAttachedVolume started for volume ...
Sep 30 22:00:52 k8s-master1 kubelet[71245]: I0930 22:00:52.486634 71245 reconciler.go:157] Reconciler: start to sync state
Hint: Some lines were ellipsized, use -l to show in full.
[root@k8s-master1 ceph]# kubectl get pods -n kube-system
NAME READY STATUS RESTARTS AGE
calico-kube-controllers-6949477b58-82r4z 1/1 Running 24 60d
calico-node-kr8bt 1/1 Running 35 60d
calico-node-pzzlf 1/1 Running 38 60d
calico-node-wwrjq 1/1 Running 53 60d
coredns-7f89b7bc75-2cgxw 1/1 Running 25 60d
coredns-7f89b7bc75-gm6jp 1/1 Running 25 60d
etcd-k8s-master1 1/1 Running 24 60d
kube-apiserver-k8s-master1 1/1 Running 0 4m35s
kube-controller-manager-k8s-master1 1/1 Running 88 60d
kube-proxy-4tnzv 1/1 Running 24 60d
kube-proxy-mnnsg 1/1 Running 24 60d
kube-proxy-mxnhb 1/1 Running 26 60d
kube-scheduler-k8s-master1 1/1 Running 84 60d
[root@k8s-master1 ceph]#
再次查看之前创建的rbd-pvc，已显示正常绑定pv
[root@k8s-master1 ceph]# kubectl get pvc NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE rbd-pvc Bound pvc-459cb59a-8504-4800-9264-388f3a7dc397 1Gi RWO rbd-storageclass 13m You have new mail in /var/spool/mail/root [root@k8s-master1 ceph]# kubectl get pv NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGE pvc-459cb59a-8504-4800-9264-388f3a7dc397 1Gi RWO Delete Bound default/rbd-pvc rbd-storageclass 5m17s
6）创建pod挂载PVC
[root@k8s-master1 ceph]# vim rbd-sc-nginx-pod.yaml
[root@k8s-master1 ceph]# cat rbd-sc-nginx-pod.yaml
apiVersion: v1
kind: Pod
metadata:
name: rbd-sc-nginx-pod
labels:
name: rbd-sc-nginx-pod
spec:
containers:
- name: rbd-sc-nginx-pod
image: nginx:latest
imagePullPolicy: IfNotPresent
ports:
- name: web
containerPort: 80
volumeMounts:
- name: ceph-rdb
mountPath: /usr/share/nginx/html
volumes:
- name: ceph-rdb
persistentVolumeClaim:
claimName: rbd-pvc
[root@k8s-master1 ceph]# kubectl apply -f rbd-sc-nginx-pod.yaml
pod/rbd-sc-nginx-pod created
[root@k8s-master1 ceph]# kubectl get pods
NAME READY STATUS RESTARTS AGE
rbd-provisioner-65bf77fd59-9jnm8 1/1 Running 2 42m
rbd-sc-nginx-pod 0/1 ContainerCreating 0 5s
[root@k8s-master1 ceph]# kubectl get pods
NAME READY STATUS RESTARTS AGE
rbd-provisioner-65bf77fd59-9jnm8 1/1 Running 2 43m
rbd-sc-nginx-pod 1/1 Running 0 66s
[root@k8s-master1 ceph]# kubectl get pods -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
rbd-provisioner-65bf77fd59-9jnm8 1/1 Running 2 44m 10.244.36.114 k8s-node1 <none> <none>
rbd-sc-nginx-pod 1/1 Running 0 111s 10.244.36.116 k8s-node1 <none> <none>
7）测试访问
[root@k8s-master1 ceph]# kubectl exec -it rbd-sc-nginx-pod -- /bin/sh -c 'echo Hello World from Ceph RBD!!! > /usr/share/nginx/html/index.html' You have new mail in /var/spool/mail/root [root@k8s-master1 ceph]# curl 10.244.36.116 Hello World from Ceph RBD!!! [root@k8s-master1 ceph]#
8)清理
[root@k8s-master1 ceph]# kubectl delete pods rbd-sc-nginx-pod pod "rbd-sc-nginx-pod" deleted [root@k8s-master1 ceph]# kubectl delete pvc rbd-pvc persistentvolumeclaim "rbd-pvc" deleted [root@k8s-master1 ceph]# kubectl get pvc No resources found in default namespace. [root@k8s-master1 ceph]# kubectl get pv NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGE pvc-459cb59a-8504-4800-9264-388f3a7dc397 1Gi RWO Delete Released default/rbd-pvc rbd-storageclass 19m [root@k8s-master1 ceph]# kubectl get pv No resources found [root@k8s-master1 ceph]#
可以看到pvc删除后，pv也会自动删除
2. 使用CephFS作为持久数据卷
1）查看当前CephFS
[root@ceph-admin ~]# ceph fs ls name: test-cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ]
可以看到已经创建了ceph文件系统test_cephfs，存储数据池为cephfs_data，存储元数据池为cephfs_metadata
若显示No filesystems enabled，则使用以下命令创建，创建两个pool分别存储数据和元数据，然后在创建CephFS
ceph osd pool create fs_data 8 ceph osd pool create fs_metadata 8 ceph fs new cephfs_name fs_metadata fs_data
2）配置 cephfs-provisioner
[root@k8s-master1 ceph]# vim external-storage-cephfs-provisioner.yaml
[root@k8s-master1 ceph]# cat external-storage-cephfs-provisioner.yaml
apiVersion: v1
kind: ServiceAccount
metadata:
name: cephfs-provisioner
---
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: cephfs-provisioner
rules:
- apiGroups: [""]
resources: ["persistentvolumes"]
verbs: ["get", "list", "watch", "create", "delete"]
- apiGroups: [""]
resources: ["persistentvolumeclaims"]
verbs: ["get", "list", "watch", "update"]
- apiGroups: ["storage.k8s.io"]
resources: ["storageclasses"]
verbs: ["get", "list", "watch"]
- apiGroups: [""]
resources: ["events"]
verbs: ["create", "update", "patch"]
- apiGroups: [""]
resources: ["endpoints"]
verbs: ["get", "list", "watch", "create", "update", "patch"]
- apiGroups: [""]
resources: ["secrets"]
verbs: ["create", "get", "delete"]
---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: cephfs-provisioner
subjects:
- kind: ServiceAccount
name: cephfs-provisioner
namespace: default
roleRef:
kind: ClusterRole
name: cephfs-provisioner
apiGroup: rbac.authorization.k8s.io
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
name: cephfs-provisioner
rules:
- apiGroups: [""]
resources: ["secrets"]
verbs: ["create", "get", "delete"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
name: cephfs-provisioner
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: Role
name: cephfs-provisioner
subjects:
- kind: ServiceAccount
name: cephfs-provisioner
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: cephfs-provisioner
spec:
selector:
matchLabels:
app: cephfs-provisioner
replicas: 1
strategy:
type: Recreate
template:
metadata:
labels:
app: cephfs-provisioner
spec:
containers:
- name: cephfs-provisioner
image: "registry.cn-chengdu.aliyuncs.com/ives/cephfs-provisioner:latest"
imagePullPolicy: IfNotPresent
env:
- name: PROVISIONER_NAME
value: ceph.com/cephfs
command:
- "/usr/local/bin/cephfs-provisioner"
args:
- "-id=cephfs-provisioner-1"
serviceAccount: cephfs-provisioner
[root@k8s-master1 ceph]# kubectl apply -f external-storage-cephfs-provisioner.yaml
serviceaccount/cephfs-provisioner created
clusterrole.rbac.authorization.k8s.io/cephfs-provisioner created
clusterrolebinding.rbac.authorization.k8s.io/cephfs-provisioner created
role.rbac.authorization.k8s.io/cephfs-provisioner created
rolebinding.rbac.authorization.k8s.io/cephfs-provisioner created
deployment.apps/cephfs-provisioner created
[root@k8s-master1 ceph]# kubectl get pods
NAME READY STATUS RESTARTS AGE
cephfs-provisioner-78c5545448-mtnbz 1/1 Running 0 6s
rbd-provisioner-65bf77fd59-9jnm8 1/1 Running 2 57m
3）创建cephfs-secret
[root@k8s-master1 ceph]# vim cephfs-secret-1.yaml You have new mail in /var/spool/mail/root [root@k8s-master1 ceph]# cat cephfs-secret-1.yaml apiVersion: v1 kind: Secret metadata: name: cephfs-secret-1 type: "ceph.com/cephfs" data: key: QVFDR0VUTmpFcE0rTGhBQWRCbWkzbEw3YVNzMmlRZDQ5Z2NEWGc9PQ== [root@k8s-master1 ceph]# kubectl apply -f cephfs-secret-1.yaml secret/cephfs-secret-1 created [root@k8s-master1 ceph]# kubectl get secrets NAME TYPE DATA AGE ceph-secret-1 ceph.com/rbd 1 49m cephfs-provisioner-token-ltvpd kubernetes.io/service-account-token 3 3m41s cephfs-secret-1 ceph.com/cephfs 1 9s default-token-5n29f kubernetes.io/service-account-token 3 60d rbd-provisioner-token-qbtmt kubernetes.io/service-account-token 3 61m
4）配置 storageclass
[root@k8s-master1 ceph]# vim cephfs-storageclass.yaml
[root@k8s-master1 ceph]# cat cephfs-storageclass.yaml
kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
name: cephfs-storageclass
provisioner: ceph.com/cephfs
parameters:
monitors: 10.0.0.130:6789,10.0.0.129:6789,10.0.0.128:6789
adminId: admin
adminSecretName: cephfs-secret-1
claimRoot: /volumes/kubernetes
[root@k8s-master1 ceph]# kubectl apply -f cephfs-storageclass.yaml
storageclass.storage.k8s.io/cephfs-storageclass created
[root@k8s-master1 ceph]# kubectl get sc
NAME PROVISIONER RECLAIMPOLICY VOLUMEBINDINGMODE ALLOWVOLUMEEXPANSION AGE
cephfs-storageclass ceph.com/cephfs Delete Immediate false 4s
rbd-storageclass ceph.com/rbd Delete Immediate false 49m
5）创建PVC
[root@k8s-master1 ceph]# vim cephfs-sc-pvc.yaml
You have new mail in /var/spool/mail/root
[root@k8s-master1 ceph]# cat cephfs-sc-pvc.yaml
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
name: cephfs-sc-pvc
spec:
accessModes:
- ReadWriteMany
storageClassName: cephfs-storageclass
resources:
requests:
storage: 2Gi
[root@k8s-master1 ceph]# kubectl apply -f cephfs-sc-pvc.yaml
persistentvolumeclaim/cephfs-sc-pvc created
[root@k8s-master1 ceph]# kubectl get pvc
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
cephfs-sc-pvc Pending cephfs-storageclass 5s
[root@k8s-master1 ceph]# kubectl get pvc
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
cephfs-sc-pvc Bound pvc-54b09482-965a-445c-8232-c99127e8839b 2Gi RWX cephfs-storageclass 17s
[root@k8s-master1 ceph]# kubectl get pv
NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGE
pvc-54b09482-965a-445c-8232-c99127e8839b 2Gi RWX Delete Bound default/cephfs-sc-pvc cephfs-storageclass 7s
创建的cephfs-sc-pvc已经自动与pv绑定
6）创建pod挂载PVC
[root@k8s-master1 ceph]# vim cephfs-sc-nginx-pod.yaml
You have new mail in /var/spool/mail/root
[root@k8s-master1 ceph]# cat cephfs-sc-nginx-pod.yaml
apiVersion: v1
kind: Pod
metadata:
name: cephfs-sc-nginx-pod
labels:
name: cephfs-sc-nginx-pod
spec:
containers:
- name: cephfs-sc-nginx-pod
image: nginx:latest
imagePullPolicy: IfNotPresent
ports:
- name: web
containerPort: 80
volumeMounts:
- name: cephfs-data
mountPath: /usr/share/nginx/html
volumes:
- name: cephfs-data
persistentVolumeClaim:
claimName: cephfs-sc-pvc
[root@k8s-master1 ceph]# kubectl apply -f cephfs-sc-nginx-pod.yaml
pod/cephfs-sc-nginx-pod created
[root@k8s-master1 ceph]# kubectl get pods
NAME READY STATUS RESTARTS AGE
cephfs-provisioner-78c5545448-mtnbz 1/1 Running 0 17m
cephfs-sc-nginx-pod 0/1 ContainerCreating 0 6s
rbd-provisioner-65bf77fd59-9jnm8 1/1 Running 2 75m
[root@k8s-master1 ceph]# kubectl get pods -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
cephfs-provisioner-78c5545448-mtnbz 1/1 Running 0 17m 10.244.36.117 k8s-node1 <none> <none>
cephfs-sc-nginx-pod 1/1 Running 0 40s 10.244.36.121 k8s-node1 <none> <none>
rbd-provisioner-65bf77fd59-9jnm8 1/1 Running 2 75m 10.244.36.114 k8s-node1 <none> <none>
7）测试访问
[root@k8s-master1 ceph]# kubectl exec -it cephfs-sc-nginx-pod -- /bin/sh -c 'echo Hello World from CephFS!!! > /usr/share/nginx/html/index.html' [root@k8s-master1 ceph]# curl 10.244.36.121 Hello World from CephFS!!! [root@k8s-master1 ceph]#
