极客时间运维进阶训练营第十一周作业
1、掌握对象存储的特点及使用场景
对象存储特性
沿用AWS S3 api标准,无需挂载
数据存在于平面地址空间内的同一级别,应用通过唯一地址来识别每个单独的数据对象
每个对象可包含有助于检索的元数据
通过restful接口实现数据的读写
eg:
rados GW对象存储网关简介:
RadosGW是对象存储-oss objetct storage service的一种访问实现,也成为ceph 对象网关、RadosGW、RGW
可使客户端能够利用标准对象存储api来方位ceph集群,支持AWS S3和Swift api
RadosGW存储特点
通过对象存储网关将数据存储为对象,每个对象出了包含数据,还包含数据自身的元数据
通过object id来检索,只能通过API来访问或第三方客户端
存储在偏平的命名空间中,S3将这个扁平的命名空间成为bucket,swift称为容器
命名空间不能嵌套创建
bucket需要被授权才能访问,一个账号可以多个bucket授权,权限可以不同
方便的横向扩展、快速检索数据
不支持客户端挂载且需要客户端访问的时候指定文件名称
适合1次写多次读的场景
ceph 使用bucket作为存储桶,实现对象数据的存储和多用户隔离,数据存储在bucket中,用户的权限也是针对bucket进行授权,可以设置用户对不同的bucket拥有不同的权限,实现权限管理
bucket特性
所有对象必须隶属于某个存储空间,可以设置和修改存储空间属性来控制地域、访问权限、生命周期等
同一个存储空间的内部是扁平的,没有文件系统的目录等概念,所有的对象都直接隶属于其对象的存储空间
每个用户可以有多个存储空间
存储空间的名称在oss范围内必须是全局唯一,一旦创建后无法修改名称
存储空间内存的对象数目没有限制
参考
S3 提供商了user bucket object 分别表示用户、存储通和对象,其中 bucket 隶属于 user, 可以针对user 设置不同 bucket 的明明空间的访问权限,不同用户允许访问相同的bucket
2、在两台主机部署 radowsgw 存储网关以实现高可用环境
端口7480 apt install -y radosgw Centos 安装命令 yum install ceph-redosgw ceph-deploy rgw create ceph-mgr2 ceph -s 部署负载均衡器 安装keepalived apt install -y keepalived find / -name "keep*" cp /usr/share/doc/keepalived/samples/keepalived.conf.vrrp /etc/keepalived/ tee /etc/keepalived/keepalived.conf << "EOF" ! Configuration File for keepalived global_defs { notification_email { acassen } notification_email_from Alexandre.Cassen@firewall.loc smtp_server 192.168.200.1 smtp_connect_timeout 30 router_id LVS_DEVEL } vrrp_instance VI_1 { state MASTER interface eth0 garp_master_delay 10 smtp_alert virtual_router_id 51 priority 100 advert_int 1 authentication { auth_type PASS auth_pass 1111 } virtual_ipaddress { 172.31.6.188 dev eth0 label eth0:0 } } EOF systemctl restart keepalived.service systemctl enable keepalived.service ip a ping 172.31.6.188 安装 haproxy apt install -y haproxy tee -a /etc/haproxy/haproxy.cfg << "EOF" listen ceph-rgw-7480 bind 172.31.6.188:80 mode tcp server rgw1 172.31.6.103:7480 check inter 2s fall 3 rise 3 server rgw2 172.31.6.104:7480 check inter 2s fall 3 rise 3 EOF haproxy -f /etc/haproxy/haproxy.cfg systemctl restart haproxy.service systemctl enable haproxy.service netstat -ntlp curl http://172.31.6.188 curl http://rgw.iclinux.com
3、基于 s3cmd 实现 bucket 的管理及数据的上传和下载
将ceph配置进行还原,client 使用部分配置如下 [client.rgw.ceph-mgr1] rgw_host = ceph-mgr1 rgw_frontends = civetweb port=9900 rgw_dns_name = rgw.iclinux.com [client.rgw.ceph-mgr2] rgw_host = ceph-mgr2 rgw_frontends = civetweb port=9900 rgw_dns_name = rgw.iclinux.com systemctl restart ceph-radosgw@rgw.ceph-mgr1.service netstat -ntlp deploy 节点安装agent sudo apt-cache madison s3cmd sudo apt install s3cmd 验证 s3cmd --version telnet rgw.iclinux.com 80 配置s3cmd s3cmd --configure New settings: Access Key: N6FH9IFQXZY0PLTWDX76 Secret Key: E05PpMdNhYqxV21swGggVkAlIdPLrWtUjG0w70Ov Default Region: US S3 Endpoint: rgw.iclinux.com DNS-style bucket+hostname:port template for accessing a bucket: rgw.iclinux.com/%(bucket) Encryption password: Path to GPG program: /usr/bin/gpg Use HTTPS protocol: False HTTP Proxy server name: HTTP Proxy server port: 0 s3cmd 基本操作 列出所有bucket s3cmd la 创建bucket s3cmd mb s3://magedu s3cmd mb s3://css s3cmd mb s3://images 上传测试文件 cd /tmp && curl -O https://img1.jcloudcs.com/portal/brand/2021/fl1-2.jpg s3cmd put fl1-2.jpg s3://images s3cmd put fl1-2.jpg s3://images/jpg s3cmd ls s3://images 下载文件 mkdir /tmp/123 cd /tmp/123 s3cmd get s3://images/fl1-2.jpg /tmp/123 删除bucket 首先删除bucket中的所有内容 s3cmd rm s3://images/* s3cmd rb s3://images
4、基于 Nginx+RGW 的动静分离及短视频案例
rgw授权 https://docs.aws.amazon.com/zh_cn/AmazonS3/latest/userguide/example-bucket-policies.html https://docs.aws.amazon.com/zh_cn/AmazonS3/latest/API/API_Operations.html 查看权限 s3cmd ls s3:// s3cmd mb s3://videos s3cmd mb s3://images s3cmd info s3://videos 授权匿名用户只读权限 编写json配置文件 tee /tmp/mybucket-single_policy << "EOF" { "Version": "2012-10-17", "Statement": [{ "Effect": "Allow", "Principal": "","Action": "s3:GetObject","Resource": ["arn:aws:s3:::images/" ] }] } EOF s3cmd setpolicy /tmp/mybucket-single_policy s3://images 成功执行后就可以匿名用户就可以访问了 http://rgw.iclinux.com/images/fl1-2.jpg http://172.31.6.105:9900/images/fl1-2.jpg 授权videos匿名访问 tee /tmp/mybucket-single_policy_videos << "EOF" { "Version": "2012-10-17", "Statement": [{ "Effect": "Allow", "Principal": "","Action": "s3:GetObject","Resource": ["arn:aws:s3:::videos/" ] }] } EOF s3cmd setpolicy /tmp/mybucket-single_policy_videos s3://videos cd /tmp && curl -o 123.mp4 https://vod.300hu.com/4c1f7a6atransbjngwcloud1oss/5ff754f8381492940550189057/v.f30.mp4?source=1&h265=v.f1022_h265.mp4 s3cmd put /tmp/123.mp4 s3://videos 创建bucket video s3cmd mb s3://video tee /tmp/mybucket-single_policy_video << "EOF" { "Version": "2012-10-17", "Statement": [{ "Effect": "Allow", "Principal": "","Action": "s3:GetObject","Resource": ["arn:aws:s3:::video/" ] }] } EOF s3cmd setpolicy /tmp/mybucket-single_policy_video s3://video s3cmd put /tmp/123.mp4 s3://video 安装nginx ubuntu 1804 -203 apt update && apt install -y iproute2 ntpdate tcpdump telnet traceroute nfs-kernel-server nfs-common lrzsz tree openssl libssl-dev libpcre3 libpcre3-dev zlib1g-dev ntpdate tcpdump telnet traceroute gcc openssh-server lrzsz tree openssl libssl-dev libpcre3 libpcre3-dev zlib1g-dev ntpdate tcpdump telnet traceroute iotop unzip zip make && apt-get clean cd /usr/local/src && curl -O https://nginx.org/download/nginx-1.21.6.tar.gz && tar xzf nginx-1.21.6.tar.gz && cd /usr/local/src/nginx-1.21.6 && ./configure --prefix=/apps/nginx --user=nginx --group=nginx --with-http_ssl_module --with-http_v2_module --with-http_realip_module --with-http_stub_status_module --with-http_gzip_static_module --with-pcre --with-stream --with-stream_ssl_module --with-stream_realip_module && make && make install && ln -sv /apps/nginx/sbin/nginx /usr/bin && rm -rf /usr/local/src/nginx-1.21.6 && groupadd -g 2088 nginx && useradd -g nginx -s /usr/sbin/nologin -u 2088 nginx && chown -R nginx.nginx /apps/nginx FILENAME="/apps/nginx/conf/nginx.conf" if [[ -f ${FILENAME} ]];then cp ${FILENAME}{,.$(date +%s).bak} tee ${FILENAME} << "EOF" worker_processes 1; events { worker_connections 1024; } http { include mime.types; default_type application/octet-stream; sendfile on; keepalive_timeout 65; upstream videos { server 172.31.6.104:9900; server 172.31.6.105:9900; } upstream tomcat { server 172.31.6.202:8080; #server 172.31.6.105:9900; } server { listen 80; server_name rgw.iclinux.com rgw.iclinux.net; proxy_redirect off; proxy_set_header Host $host; proxy_set_header Remote_Addr $remote_addr; proxy_set_header X-REAL-IP $remote_addr; proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; location / { root html; index index.html index.htm; } location ~* \.(mp4|avi)$ { proxy_pass http://videos; } location /app1 { proxy_pass http://tomcat; } error_page 500 502 503 504 /50x.html; location = /50x.html { root html; } } } EOF fi 安装tomcat模拟后端服务-- 172.31.6.202 yum install -y tomcat systemctl restart tomcat mkdir /usr/share/tomcat/webapps/app1 tee /usr/share/tomcat/webapps/app1/index.jsp << "EOF" java app1 EOF systemctl restart tomcat 验证地址: http://172.31.6.202:8080/app1/
5、启用 ceph dashboard 并基于 prometheus 监控 ceph 集群运行状态
5.1 启用ceph dashboard
部署在 mgr 节点 两个节点均部署 apt update apt-cache madison ceph-mgr-dashboard apt install -y ceph-mgr-dashboard 部署节点查看可用模块 ceph mgr module ls | less 启用dashboard 组件 ceph mgr module enable dashboard ceph config set mgr mgr/dashboard/ssl false # 通常在nginx中启用 ceph config set mgr mgr/dashboard/ceph-mgr1/server_addr 172.31.6.104 ceph config set mgr mgr/dashboard/ceph-mgr1/server_port 9009 长时间无法启动端口,需要重启下mgr服务 systemctl restart ceph-mgr@ceph-mgr1.service 访问入口 http://172.31.6.104:9009/ 创建账号密码 echo "123456" > pass.txt ceph dashboard set-login-credentials jack -i pass.txt 启用证书 ceph dashboard create-self-signed-cert ceph config set mgr mgr/dashboard/ssl true ceph mgr services
5.2 基于 prometheus 监控 ceph 集群运行状态
4个node 安装node exporter BASE_DIR="/apps" install -d ${BASE_DIR} tar xzf /usr/local/src/node_exporter-1.5.0.linux-amd64.tar.gz -C ${BASE_DIR} ln -s /apps/node_exporter-1.5.0.linux-amd64/ /apps/node_exporter tee /etc/systemd/system/node-exporter.service << "EOF" [Unit] Description=Prometheus Node Exporter After=network.target [Service] ExecStart=/apps/node_exporter/node_exporter [Install] WantedBy=multi-user.target EOF systemctl daemon-reload && systemctl restart node-exporter && systemctl enable node-exporter 配置prometheus 收集数据 cp /etc/prometheus/prometheus.yml{,.bak} tee -a /etc/prometheus/prometheus.yml << "EOF" - job_name: "ceph-node-date" # metrics_path: '/metrics' # scheme defaults to 'http'. static_configs: - targets: ["172.31.6.106:9100","172.31.6.107:9100","172.31.6.108:9100","172.31.6.109:9100"] EOF promtool check config /etc/prometheus/prometheus.yml systemctl restart prometheus.service # ceph 开启 Prometheus 监控插件 部署节点执行 ceph mgr module enable prometheus 验证 http://172.31.6.105:9283 http://172.31.6.104:9283 haproxy(172.31.6.204) 修改haproxy 配置,实现负载均衡 tee -a /etc/haproxy/haproxy.cfg << "EOF" listen ceph-prometheus-9283 bind 172.31.6.188:9283 mode tcp server rgw1 172.31.6.104:9283 check inter 2s fall 3 rise 3 server rgw2 172.31.6.105:9283 check inter 2s fall 3 rise 3 EOF systemctl restart prometheus http://172.31.6.188:9283 配置Prometheus 实现数据的 tee -a /etc/prometheus/prometheus.yml << "EOF" - job_name: "ceph-clushter-date" static_configs: - targets: ["172.31.6.188:9283"] EOF systemctl restart prometheus ### grafana 模板 osd 监控 导入模板 17296 老版本可使用模板 5336 ceph 存储池 使用模板 5342 ceph cluser 使用模板 7056