ceph安装（ nautilus 版本 ）

 

一、基础环境准备

（1）关闭firewalled

systemctl stop firewalld
systemctl disable firewalld

（2）关闭selinux：

sed -i 's/enforcing/disabled/' /etc/selinux/config
setenforce 0

（3）关闭NetworkManager

systemctl disable NetworkManager && systemctl stop NetworkManager

（4）添加主机名与IP对应关系：

vim /etc/hosts
172.30.112.78 deploy
172.30.112.179 node1
172.30.112.115 node2
172.30.112.82 node13

（5）设置主机名：

hostnamectl set-hostname deploy
hostnamectl set-hostname node1
hostnamectl set-hostname node2
hostnamectl set-hostname node3

（6）同步网络时间和修改时区

cat >/etc/yum.repos.d/ceph.repo <<EOF
[noarch]
name=ceph-noarch
baseurl=http://mirrors.aliyun.com/ceph/rpm-nautilus/el7/noarch/
enable=1
gpgcheck=0

[ceph]
name=ceph
baseurl=http://mirrors.aliyun.com/ceph/rpm-nautilus/el7/x86_64/
enable=1
gpgcheck=0
EOF

（7）同步网络时间和修改时区

yum install chrony -y 
systemctl restart chronyd.service && systemctl enable chronyd.service
cp /usr/share/zoneinfo/Asia/Shanghai /etc/localtime

（8）设置文件描述符

echo "ulimit -SHn 102400" >> /etc/rc.local
cat >> /etc/security/limits.conf << EOF
* soft nofile 65535
* hard nofile 65535
EOF

（9）内核参数优化

echo 'vm.swappiness = 0' >> /etc/sysctl.conf
echo 'kernel.pid_max = 4194303' >> /etc/sysctl.conf
sysctl -p

（10）在deploy上配置免密登录到node1 node2 node3

# 每台增加普通用户
[root@ST-K8S-01 ~]# useradd cephadmin
[root@ST-K8S-01 ~]# passwd cephadmin

# 在 deploy 配置其他机器的免密登录,切换到普通用户
[cephadmin@ST-K8S-01 ~]$ ssh-keygen
[cephadmin@ST-K8S-01 ~]$ ssh-copy-id cephadmin@10.2.57.3
[cephadmin@ST-K8S-01 ~]$ ssh-copy-id cephadmin@10.2.57.4
[cephadmin@ST-K8S-01 ~]$ ssh-copy-id cephadmin@10.2.57.5

# 配置 cephadmin 免密 sudo
echo "cephadmin ALL = (root) NOPASSWD:ALL" | sudo tee /etc/sudoers.d/cephadmin
sudo chmod 0440 /etc/sudoers.d/cephadmin

(11) read_ahead,通过数据预读并且记载到随机访问内存方式提高磁盘读操作

echo "8192" > /sys/block/sda/queue/read_ahead_kb

(12) I/O Scheduler，SSD要用noop，SATA/SAS使用deadline

echo "deadline" >/sys/block/sd[x]/queue/scheduler
echo "noop" >/sys/block/sd[x]/queue/scheduler

二、安装

####如果有就清理
ceph-deploy purge ST-K8S-01 ST-K8S-02 ST-K8S-03
ceph-deploy purgedata ST-K8S-01 ST-K8S-02 ST-K8S-03
ceph-deploy forgetkeys

所有节点安装

yum install -y ceph ceph-radosgw

在admin上安装

[cephadmin@ST-K8S-01 ~]$ sudo yum install -y ceph-deploy
[cephadmin@ST-K8S-01 ~]$ mkdir /data/ceph-cluster
[cephadmin@ST-K8S-01 ~]$ cd /data/ceph-cluster/
[cephadmin@ST-K8S-01 ceph-cluster]$ ceph-deploy new --public-network 10.2.57.0/24 --cluster-network 10.2.57.0/24 ST-K8S-01 ST-K8S-02 ST-K8S-03
[cephadmin@ST-K8S-01 ceph-cluster]$ cat ceph.conf
[global]
fsid = aca2b777-962a-4f7b-8663-20e0c1e30bc4
ms_bind_ipv6 = true
mon_initial_members = ST-K8S-01, ST-K8S-02, ST-K8S-03
mon_host = 10.2.57.3,10.2.57.4,10.2.57.5
auth_cluster_required = cephx
auth_service_required = cephx
auth_client_required = cephx
public_network = 10.2.57.0/24
cluster_network = 10.2.57.0/24
osd pool default size = 2
[cephadmin@ST-K8S-01 ceph-cluster]$ ceph-deploy --overwrite-conf mon create-initial

.将配置文件同步到各节点

[cephadmin@ST-K8S-01 ceph-cluster]$ ceph-deploy --overwrite-conf config push ST-K8S-01 ST-K8S-02 ST-K8S-03

确保你对 ceph.client.admin.keyring 有正确的操作权限

sudo cp ceph.client.admin.keyring /etc/ceph/
sudo chmod +r /etc/ceph/ceph.client.admin.keyring

添加osd

[cephadmin@ST-K8S-01 ceph-cluster]$ ceph-deploy osd create ST-K8S-01 --data /dev/sdc
[cephadmin@ST-K8S-01 ceph-cluster]$ ceph-deploy osd create ST-K8S-02 --data /dev/sdc
[cephadmin@ST-K8S-01 ceph-cluster]$ ceph-deploy osd create ST-K8S-03 --data /dev/sdc

创建管理节点

[cephadmin@ST-K8S-01 ceph-cluster]$ ceph-deploy mgr create ST-K8S-01 ST-K8S-02 ST-K8S-03

四、ceph开启Dashboard

1、安装Dashboard

yum install -y ceph-mgr-dashboard -y

2、开启插件

ceph mgr module enable dashboard

3、禁用SSL

ceph config set mgr mgr/dashboard/ssl false

4、配置监听IP

ceph config set mgr mgr/dashboard/server_addr 0.0.0.0

温馨提醒
此处必须设置监控地址为0.0.0.0,而不能是直接IP地址，因为其监控的是所有本地地址包括IPV4和IPV6，同时也不能禁用IPV6地址。

5、配置监听端口

ceph config set mgr mgr/dashboard/server_port 9400

6、设置用户及密码

将密码写入到一个文件：

echo "123456" >password     ###此时admin的密码为：123456

然后执行

ceph dashboard ac-user-create admin -i password administrator

7、使用配置生效

ceph mgr module disable dashboard
ceph mgr module enable dashboard

8、通过查看ceph mgr services命令输出地址

[root@ceph-node01 ceph-deploy]# ceph mgr services
{
    "dashboard": "http://ceph-node01:8444/"
}

9、开启prometheus监控

[root@ceph-node01 ceph-deploy]# ceph mgr module enable prometheus
[root@ceph-node01 ceph-deploy]# ceph mgr services
{
    "dashboard": "http://ceph-node01:8444/",
    "prometheus": "http://ceph-node01:9283/"
}

prometheus

vim /opt/prometheus/prometheus1.yml 添加以下：

 - job_name: 'ceph'
    static_configs:
    - targets: ['192.168.8.100:9283']