【ceph】ceph osd blacklist cep黑名单|MDS问题分析

目录

blacklist 是什么

blacklist相关操作

Ceph MDS问题分析 

CephFS client evict子命令使用

概述

命令格式

1. 查看所有client/session

2. evict指定client

3. 查看ceph osd的blacklist

4. 尝试恢复evict client

5. evict所有的client

6. session kill命令

代码分析

相关参数

evict client后的处理

方法1:rm blacklist

方法2:wait 1小时


blacklist 是什么

(摘自:https://blog.51cto.com/wendashuai/2493435)

blacklist最常用于CephFS场景中,以防止滞后的元数据服务器对OSD上的数据进行不良更改。

当文件系统客户端无响应或行为异常时,可能有必要强制终止其对文件系统的访问。 此过程称为eviction(驱逐)。

驱逐CephFS客户端会阻止其与MDS daemons和OSD daemons进一步通信。 如果客户端正在对文件系统进行buffered IO,则所有未刷新的数据都将丢失。

blacklist也用于块存储中(openstack场景),某个卷存在watcher,无法删除该卷。假设客户端上watcher:watcher=10.37.192.139:0/1308721908存在异常。加入黑名单中,然后删除卷。

blacklist相关操作

(摘自<OSD相关命令>:https://www.bookstack.cn/read/zxj_ceph/osd-refer

查看 BlackList

$ ceph osd blacklist ls

or

$ ceph osd dump

6.  移除 BlackList

1、blacklist-用于管理客户端黑名单

1-1 把加入黑名单
可指定时间,从现在起秒

ceph osd blacklist add <EntityAddr> {<float[0.0-]>}

1-2 列出进黑名单的客户端

ceph osd blacklist ls

1-3 从黑名单里删除

ceph osd blacklist rm <EntityAddr>

例子:

把客户端加入黑名单

ceph osd blacklist add <EntityAddr> {<float[0.0-]>}
Eg:
ceph osd blacklist add 10.37.192.139:0/1308721908

注:默认时间为1小时,超过1小时自动释放;

查看黑名单中的客户端

# ceph osd blacklist ls
listed 1 entries
10.37.192.139:0/1308721908 2019-02-27 10:10:52.049084

删除黑名单中的客户端

ceph osd blacklist rm <EntityAddr>
Eg:
ceph osd blacklist rm 10.37.192.139:0/1308721908

查看rbd卷的watcher

[root@ceph1 ~]# rbd status dpool/test
Watchers: 10.37.192.139:0/1308721908 client.172221 cookie=1222222189908

未消化搜藏:

CephFS client evict子命令使用

(原文:http://www.yangguanjun.com/2018/09/28/cephfs-client-evict-intro/)

概述

在使用Ceph的CephFS时,每个client都会建立与MDS的连接,以获取CephFS的元数据信息。如果有多个Active的MDS,则一个client可能会与多个MDS都建立连接。

Ceph提供了client/session子命令来查询和管理这些连接。在这些子命令中,有一个命令来处理CephFS的client有问题时,如何手动来断开这些client的连接,比如执行命令:# ceph tell mds.2 client evict,则会把与mds rank 2 连接的所有clients都断开。

那么执行client evict的影响是什么?是否可以恢复呢?本文将重点介绍一下这些。

命令格式

参考:Ceph file system client eviction — Ceph Documentation

测试环境:Ceph Mimic 13.2.1

1. 查看所有client/session

可以通过命令 client/session ls查看与ms rank [id] 建立connection的所有clients;

# ceph tell mds.0 client ls
2018-09-05 10:00:15.986 7f97f0ff9700  0 client.25196 ms_handle_reset on 192.168.0.26:6800/1856812761
2018-09-05 10:00:16.002 7f97f1ffb700  0 client.25199 ms_handle_reset on 192.168.0.26:6800/1856812761
[
    {
        "id": 25085,
        "num_leases": 0,
        "num_caps": 5,
        "state": "open",
        "replay_requests": 0,
        "completed_requests": 0,
        "reconnecting": false,
        "inst": "client.25085 192.168.0.26:0/265326503",
        "client_metadata": {
            "ceph_sha1": "5533ecdc0fda920179d7ad84e0aa65a127b20d77",
            "ceph_version": "ceph version 13.2.1 (5533ecdc0fda920179d7ad84e0aa65a127b20d77) mimic (stable)",
            "entity_id": "admin",
            "hostname": "mimic3",
            "mount_point": "/mnt/cephfuse",
            "pid": "44876",
            "root": "/"
        }
    }
]

比较重要的信息有:

  • id:client唯一id
  • num_caps:client获取的caps
  • inst:client端的ip和端口链接信息
  • ceph_version:client端的ceph-fuse版本,若使用kernel client,则为kernel_version
  • hostname:client端的主机名
  • mount_point:client在主机上对应的mount point
  • pid:client端ceph-fuse进程的pid

2. evict(驱逐)指定client

可以通过指定id来evict(驱逐)特定的client链接;

若有多个Active MDS,单个MDS Rank的evict也会传播到别的Active MDS

# ceph tell mds.0 client evict id=25085

evict client后,在对应的host上检查client的mountpoint已经不能访问:

root@mimic3:/mnt/cephfuse# ls
ls: cannot open directory '.': Cannot send after transport endpoint shutdown
 
 
root@mimic3:~# vim /var/log/ceph/ceph-client.admin.log
...
2018-09-05 10:02:54.829 7fbe732d7700 -1 client.25085 I was blacklisted at osd epoch 519

3. 查看ceph osd的blacklist

evict client后,会把client加入到osd blacklist中(后续有代码分析);

root@mimic1:~# ceph osd blacklist ls
listed 1 entries
192.168.0.26:0/265326503 2018-09-05 11:02:54.696345

加入到osd blacklist后,防止evict client的in-flight数据写下去,影响数据一致性;有效时间为1个小时;

4. 尝试恢复evict client

把ceph osd blacklist里与刚才evict client相关的记录删除;

root@mimic1:~# ceph osd blacklist rm 192.168.0.26:0/265326503
un-blacklisting 192.168.0.26:0/265326503

在对应的host上检查client是否正常?发现client变得正常了!!

root@mimic3:~# cd /mnt/cephfuse
root@mimic3:/mnt/cephfuse# ls
perftest

而测试 Ceph Luminous 12.2.7 版本时,evcit client后无法立刻恢复,等一段时间后恢复!!

( “mds_session_autoclose”: “300.000000”,)

root@luminous2:~# ceph osd blacklist rm 192.168.213.25:0/1534097905
un-blacklisting 192.168.213.25:0/1534097905

root@luminous2:~# ceph osd blacklist ls
listed 0 entries

root@luminous2:/mnt/cephfuse# ls
ls: cannot open directory '.': Cannot send after transport endpoint shutdown

等待一段时间(300s)后,session变得正常!

root@luminous2:/mnt/cephfuse# ls
perftest

测试cephfs kernel client的evcit,client无法恢复!!

root@mimic3:~# cd /mnt/cephfs
-bash: cd: /mnt/cephfs: Permission denied

5. evict所有的client

若在evict命令后不指定具体的client id,则会把与该MDS Rank链接的所有client evict掉;

若有多个Active MDS,单个MDS Rank的evict也会传播到别的Active MDS

# ceph tell mds.0 client evict

6. session kill(命令这个命令慎用,也一定不要误用,影响比较大!!!

session子命令里还有一个kill命令,它比evict命令更彻底;

root@mimic1:~# ceph tell mds.0 session kill 104704
2018-09-05 15:57:45.897 7ff2157fa700  0 client.25742 ms_handle_reset on 192.168.0.26:6800/1856812761
2018-09-05 15:57:45.917 7ff2167fc700  0 client.25745 ms_handle_reset on 192.168.0.26:6800/1856812761
 
 
root@mimic1:~# ceph tell mds.0 session ls
2018-09-05 15:57:50.709 7f44eeffd700  0 client.95370 ms_handle_reset on 192.168.0.26:6800/1856812761
2018-09-05 15:57:50.725 7f44effff700  0 client.95376 ms_handle_reset on 192.168.0.26:6800/1856812761
[]
 
 
root@mimic1:~# ceph osd blacklist ls
listed 1 entries
192.168.0.26:0/1613295381 2018-09-05 16:57:45.920138

删除 osd blacklist entry:

root@mimic1:~# ceph osd blacklist rm 192.168.0.26:0/1613295381
un-blacklisting 192.168.0.26:0/1613295381
root@mimic1:~# ceph osd blacklist ls
listed 0 entries

之后client链接没有再恢复!!!

root@mimic3:~# cd /mnt/cephfuse
root@mimic3:/mnt/cephfuse# ls
ls: cannot open directory '.': Cannot send after transport endpoint shutdown

session kill后,这个session无法再恢复!!!也要慎用!!!

代码分析

基于Ceph Mimic 13.2.1代码;

执行client evict的代码如下,可以看出里面会添加osd blacklist:

bool MDSRank::evict_client(int64_t session_id,
                           bool wait, bool blacklist, std::stringstream& err_ss,
                           Context *on_killed)
{
	...
    // 获取指定id的session
    Session *session = sessionmap.get_session(
                           entity_name_t(CEPH_ENTITY_TYPE_CLIENT, session_id));
    
	// 定义kill mds session的函数
    auto kill_mds_session = [this, session_id, on_killed]() {
        assert(mds_lock.is_locked_by_me());
        Session *session = sessionmap.get_session(
                               entity_name_t(CEPH_ENTITY_TYPE_CLIENT, session_id));
        if (session) {
            if (on_killed) {
                server->kill_session(session, on_killed);
            } else {
                C_SaferCond on_safe;
                server->kill_session(session, &on_safe);
                mds_lock.Unlock();
                on_safe.wait();
                mds_lock.Lock();
            }
        }
        ...
    };
 
 	// 定义添加OSD blacklist的函数
    auto background_blacklist = [this, session_id, cmd](std::function<void ()> fn) {
        ...
        Context *on_blacklist_done = new FunctionContext([this, session_id, fn](int r) {
            objecter->wait_for_latest_osdmap(
                new C_OnFinisher(
            		new FunctionContext(...), finisher)
            );
        });
		...
        monc->start_mon_command(cmd, {}, nullptr, nullptr, on_blacklist_done);
    };
 
 
    auto blocking_blacklist = [this, cmd, &err_ss, background_blacklist]() {
        C_SaferCond inline_ctx;
        background_blacklist([&inline_ctx]() {
            inline_ctx.complete(0);
        });
        mds_lock.Unlock();
        inline_ctx.wait();
        mds_lock.Lock();
    };
 
 	// 根据参数执行kill mds session和添加OSD的blacklist
    if (wait) {
        if (blacklist) {
            blocking_blacklist();
        }
 
        // We dropped mds_lock, so check that session still exists
        session = sessionmap.get_session(entity_name_t(CEPH_ENTITY_TYPE_CLIENT,
                                         session_id));
		...
        kill_mds_session();
    } else {
        if (blacklist) {
            background_blacklist(kill_mds_session);
        } else {
            kill_mds_session();
        }
    }
	...
}

调用该函数的地方有:

Cscope tag: evict_client
   #   line  filename / context / line
   1   1965  mds/MDSRank.cc <<handle_asok_command>>
             bool evicted = evict_client(strtol(client_id.c_str(), 0, 10), true,
   2   2120  mds/MDSRank.cc <<evict_clients>>
             evict_client(s->info.inst.name.num(), false,
   3    782  mds/Server.cc <<find_idle_sessions>>
             mds->evict_client(session->info.inst.name.num(), false, true,
   4   1058  mds/Server.cc <<reconnect_tick>>
             mds->evict_client(session->info.inst.name.num(), false, true, ss,

1、handle_asok_command:命令行处理client evict

2、evict_clients:批量evict clients

3、find_idle_sessions:对于stale状态的session,执行evict client

4、reconnect_tick:mds恢复后等待client reconnect,45s超时后evict clients

相关参数

于mds session相关的配置参数有:

# ceph daemon mgr.luminous2 config show | grep mds_session_
    "mds_session_autoclose": "300.000000",
    "mds_session_blacklist_on_evict": "true",
    "mds_session_blacklist_on_timeout": "true",
    "mds_session_timeout": "60.000000",

还有一些client相关的:

"client_reconnect_stale": "false",
"client_tick_interval": "1.000000",
"mon_client_ping_interval": "10.000000",
"mon_client_ping_timeout": "30.000000",

evict client后的处理

从上面的实践可以看出,evcit client后,client会被添加到osd blacklist里,超时时间为1小时;在这个时间段内,client是不能访问CephFS的;

但是通过命令:ceph osd blacklist rm <entry> 删除osd的blacklist后,client端立刻就能继续访问CephFS,一切都跟之前正常时候一样!

方法1:rm blacklist

root@mimic1:~# ceph tell mds.0 client evict id=25085
2018-09-05 11:07:43.580 7f80d37fe700  0 client.25364 ms_handle_reset on 192.168.0.26:6800/1856812761
2018-09-05 11:07:44.292 7f80e8ff9700  0 client.25370 ms_handle_reset on 192.168.0.26:6800/1856812761
 
 
root@mimic1:~# ceph tell mds.0 client ls
2018-09-05 11:05:23.527 7f5005ffb700  0 client.25301 ms_handle_reset on 192.168.0.26:6800/1856812761
2018-09-05 11:05:23.539 7f5006ffd700  0 client.94941 ms_handle_reset on 192.168.0.26:6800/1856812761
[]
 
 
root@mimic1:~# ceph osd blacklist rm 192.168.0.26:0/265326503
un-blacklisting 192.168.0.26:0/265326503
 
 
root@mimic1:~# ceph tell mds.0 client ls
2018-09-05 11:07:57.884 7fe07b7f6700  0 client.95022 ms_handle_reset on 192.168.0.26:6800/1856812761
2018-09-05 11:07:57.900 7fe07c7f8700  0 client.25400 ms_handle_reset on 192.168.0.26:6800/1856812761
[]

然后在client host重新访问以下挂载点目录后,session变为正常

root@mimic1:~# ceph tell mds.0 client ls
2018-09-05 11:06:31.484 7f6c6bfff700  0 client.94971 ms_handle_reset on 192.168.0.26:6800/1856812761
2018-09-05 11:06:31.496 7f6c717fa700  0 client.94977 ms_handle_reset on 192.168.0.26:6800/1856812761
[
    {
        "id": 25085,
        "num_leases": 0,
        "num_caps": 4,
        "state": "open",
        "replay_requests": 0,
        "completed_requests": 0,
        "reconnecting": false,
        "inst": "client.25085 192.168.0.26:0/265326503",
        "client_metadata": {
            "ceph_sha1": "5533ecdc0fda920179d7ad84e0aa65a127b20d77",
            "ceph_version": "ceph version 13.2.1 (5533ecdc0fda920179d7ad84e0aa65a127b20d77) mimic (stable)",
            "entity_id": "admin",
            "hostname": "mimic3",
            "mount_point": "/mnt/cephfuse",
            "pid": "44876",
            "root": "/"
        }
    }
]

方法2:wait 1小时

默认evict client后,添加osd blacklist的超时时间为1小时,考察1小时过后,session可以变为正常:

root@mimic1:~# ceph osd blacklist ls
listed 0 entries

然后在client host重新访问以下挂载点目录后,session变为正常

root@mimic3:~# cd /mnt/cephfuse/
root@mimic3:/mnt/cephfuse# ls
perftest

查看mds的sessions:

root@mimic1:~# ceph tell mds.0 session ls
2018-09-05 13:56:26.630 7fae7f7fe700  0 client.95118 ms_handle_reset on 192.168.0.26:6801/1541744746
2018-09-05 13:56:26.642 7fae94ff9700  0 client.25496 ms_handle_reset on 192.168.0.26:6801/1541744746
[
    {
        "id": 25085,
        "num_leases": 0,
        "num_caps": 1,
        "state": "open",
        "replay_requests": 0,
        "completed_requests": 0,
        "reconnecting": false,
        "inst": "client.25085 192.168.0.26:0/265326503",
        "client_metadata": {
            "ceph_sha1": "5533ecdc0fda920179d7ad84e0aa65a127b20d77",
            "ceph_version": "ceph version 13.2.1 (5533ecdc0fda920179d7ad84e0aa65a127b20d77) mimic (stable)",
            "entity_id": "admin",
            "hostname": "mimic3",
            "mount_point": "/mnt/cephfuse",
            "pid": "44876",
            "root": "/"
        }
    }
]

Ceph MDS问题分析 

(原文:https://blog.csdn.net/weixin_44389885/article/details/86621701)

1. 问题背景


1.1 客户端缓存问题
$ ceph -s
health HEALTH_WARN
mds0: Client xxx-online00.gz01 failing to respond to cache pressure

官方解释

消息: “Client name failing to respond to cache pressure”
代码: MDS_HEALTH_CLIENT_RECALL, MDS_HEALTH_CLIENT_RECALL_MANY

描述:
客户端有各自的元数据缓存,客户端缓存中的条目(比如索引节点)也会存在于 MDS 缓存中,所以当 MDS 需要削减其缓存时(保持在 mds_cache_size 以下),它也会发消息给客户端让它们削减自己的缓存。如果有客户端没响应或者有缺陷,就会妨碍 MDS 将缓存保持在 mds_cache_size 以下, MDS 就有可能耗尽内存而后崩溃。如果某个客户端的响应时间超过了 mds_recall_state_timeout (默认为 60s ),这条消息就会出现。

1.2 服务端内存不释放
同上参考1.1 客户端缓存问题

1.3 mds session的inode过多
客户端session的inode太多,导致内存很高,从而也导致主从mds切换加载inode慢,严重影响服务的可用性。

1.4 mds夯住问题或慢查询
客户端搜索遍历查找文件(不可控)
session的 inode太大导致mds负载过高
日志级别开的太大,从而导致mds负载高

2. 分析思路
上面的几个问题都是有一定的联系,互相影响的。所以,我们先从已知的方向逐步深入分析问题,从而优化解决问题。

2.1 组件通信流程图


Client <–> MDS
元数据操作和capalities
Client <–> OSD
数据IO
Client <–> Monitor
认证,集群map信息等
MDS <–> Monitor
心跳,集群map信息等
MDS <–> OSD
元数据IO
Monitor <–> OSD
心跳,集群map信息等
2.2 查看客户端session
$ ceph --admin-daemon  /var/run/ceph/ceph-mds.ceph-xxxx-osd02.py.asok session ls
[
    {
        "id": 5122511,
        "num_leases": 0,
        "num_caps": 655,
        "state": "open",
        "replay_requests": 0,
        "completed_requests": 1,
        "reconnecting": false,
        "inst": "client.5122511 192.168.1.2:0\/2026289820",
        "client_metadata": {
            "ceph_sha1": "b1e0532418e4631af01acbc0cedd426f1905f4af",
            "ceph_version": "ceph version 0.94.10 (b1e0532418e4631af01acbc0cedd426f1905f4af)",
            "entity_id": "log_xxx_cephfs",
            "hostname": "ceph-test-osd02",
            "mount_point": "\/mnt\/log"
        }
    }
]
说明:

id:client唯一id
num_caps:client获取的caps
inst:client端的ip和端口链接信息
ceph_version:client端的ceph-fuse版本,若使用kernel client,则为kernel_version
hostname:client端的主机名
mount_point:client在主机上对应的mount point
pid:client端ceph-fuse进程的pid
2.3 查看客户端的inode数量
跟踪代码发现session里面的num_caps就是统计的客户端的inode数量, 大概统计了下已经打开的inode数量在400w左右。

image.png
总结:
可以查看客户端的session信息,包含host、mount、inode等信息
可以统计所有客户端session的inode数量。

2.4 尝试mds主从切换
2.4.1 执行过程如下
2018-04-27 19:24:03.923349 7f53015d7700  1 mds.0.2738 handle_mds_map state change up:boot --> up:replay
2018-04-27 19:24:03.923356 7f53015d7700  1 mds.0.2738 replay_start
2018-04-27 19:24:03.923360 7f53015d7700  1 mds.0.2738  recovery set is
2018-04-27 19:24:03.923365 7f53015d7700  1 mds.0.2738  waiting for osdmap 6339 (which blacklists prior instance)
2018-04-27 19:24:03.948526 7f52fc2ca700  0 mds.0.cache creating system inode with ino:100
2018-04-27 19:24:03.948675 7f52fc2ca700  0 mds.0.cache creating system inode with ino:1
2018-04-27 19:24:04.238128 7f52fa2b8700  1 mds.0.2738 replay_done
2018-04-27 19:24:04.238143 7f52fa2b8700  1 mds.0.2738 making mds journal writeable
2018-04-27 19:24:04.924352 7f53015d7700  1 mds.0.2738 handle_mds_map i am now mds.0.2738
2018-04-27 19:24:04.924357 7f53015d7700  1 mds.0.2738 handle_mds_map state change up:replay --> up:reconnect
2018-04-27 19:24:04.924370 7f53015d7700  1 mds.0.2738 reconnect_start
2018-04-27 19:24:04.924371 7f53015d7700  1 mds.0.2738 reopen_log
2018-04-27 19:24:04.924380 7f53015d7700  1 mds.0.server reconnect_clients -- 19 sessions
2018-04-27 19:24:04.926357 7f53015d7700  0 log_channel(cluster) log [DBG] : reconnect by client.4375 192.168.1.3:0/1796553051 after 0.001950
2018-04-27 19:24:04.926429 7f53015d7700  0 log_channel(cluster) log [DBG] : reconnect by client.4403 192.168.1.3:0/1032897847 after 0.002036
2018-04-27 19:24:15.228507 7f53015d7700  1 mds.0.2738 reconnect_done
2018-04-27 19:24:15.984143 7f53015d7700  1 mds.0.2738 handle_mds_map i am now mds.0.2738
2018-04-27 19:24:15.984148 7f53015d7700  1 mds.0.2738 handle_mds_map state change up:reconnect --> up:rejoin
2018-04-27 19:24:15.984156 7f53015d7700  1 mds.0.2738 rejoin_start
2018-04-27 19:25:15.987531 7f53015d7700  1 mds.0.2738 rejoin_joint_start
2018-04-27 19:27:40.105134 7f52fd4ce700  1 mds.0.2738 rejoin_done
2018-04-27 19:27:42.206654 7f53015d7700  1 mds.0.2738 handle_mds_map i am now mds.0.2738
2018-04-27 19:27:42.206658 7f53015d7700  1 mds.0.2738 handle_mds_map state change up:rejoin --> up:active
2018-04-27 19:27:42.206666 7f53015d7700  1 mds.0.2738 recovery_done -- successful recovery!
主从切换流程:

handle_mds_map state change up:boot --> up:replay
handle_mds_map state change up:replay --> up:reconnect
handle_mds_map state change up:reconnect --> up:rejoin
handle_mds_map state change up:rejoin --> up:active
up:boot

此状态在启动期间被广播到CEPH监视器。这种状态是不可见的,因为监视器立即将MDS分配给可用的秩或命令MDS作为备用操作。这里记录了完整性的状态。
up:replay

日志恢复阶段,他将日志内容读入内存后,在内存中进行回放操作。
up:reconnect

恢复的mds需要与之前的客户端重新建立连接,并且需要查询之前客户端发布的文件句柄,重新在mds的缓存中创建一致性功能和锁的状态。
mds不会同步记录文件打开的信息,原因是需要避免在访问mds时产生多余的延迟,并且大多数文件是以只读方式打开。
up:rejoin

把客户端的inode加载到mds cache。(耗时最多的地方)
为什么mds切换耗时比较高?

分析日志(发现执行rejoin_start,rejoin_joint_start动作耗时比较高)
2018-04-27 19:24:15.984156 7f53015d7700  1 mds.0.2738 rejoin_start
2018-04-27 19:25:15.987531 7f53015d7700  1 mds.0.2738 rejoin_joint_start
2018-04-27 19:27:40.105134 7f52fd4ce700  1 mds.0.2738 rejoin_done
2018-04-27 19:27:42.206654 7f53015d7700  1 mds.0.2738 handle_mds_map i am now mds.0.2738
2018-04-27 19:27:42.206658 7f53015d7700  1 mds.0.2738 handle_mds_map state change up:rejoin --> up:active
跟踪代码分析(在执行process_imported_caps超时了, 这个函数主要是打开inodes 加载到cache中)

image.png
总结:

主从切换时mds详细状态
主从切换时主要耗时的阶段rejoin_start,加载客户端session的inode信息
2.5 释放客户端inode
2.5.1 模拟客户端session inode过多
查看客户端session信息
#查看客户端session的inode数量, num_caps:7
$ ceph daemon mds.ceph-xxx-osd01.ys session ls
[
    {
        "id": 554418,
        "num_leases": 0,
        "num_caps": 7,
        "state": "open",
        "replay_requests": 0,
        "completed_requests": 0,
        "reconnecting": false,
        "inst": "client.554418 192.168.1.2:0/1285681097",
        "client_metadata": {
            "ceph_sha1": "fe3a2269d799a8b950404cb2de11af84c7af0ea4",
            "ceph_version": "didi_dss version 12.2.2.4 (fe3a2269d799a8b950404cb2de11af84c7af0ea4) luminous (stable)",
            "entity_id": "admin",
            "hostname": "ceph-xxx-osd01.ys",
            "mount_point": "/mnt",
            "pid": "2084",
            "root": "/"
        }
    }
]
客户端遍历所有文件
#遍历挂载目录下所有文件
$ tree /mnt/
 
#查看这个目录下面所有文件夹及文件数量
$ tree /mnt/ | wc -l
347
再次查看客户端session信息
#查看客户端session的inode数量, num_caps:346
$ ceph daemon mds.ceph-xxx-osd01.ys session ls
[
   {
        "id": 554418,
        "num_leases": 1,
        "num_caps": 346,
        "state": "open",
        "replay_requests": 0,
        "completed_requests": 2,
        "reconnecting": false,
        "inst": "client.554418 192.168.1.3:0/1285681097",
        "client_metadata": {
            "ceph_sha1": "fe3a2269d799a8b950404cb2de11af84c7af0ea4",
            "ceph_version": "didi_dss version 12.2.2.4 (fe3a2269d799a8b950404cb2de11af84c7af0ea4) luminous (stable)",
            "entity_id": "admin",
            "hostname": "ceph-xxx-osd01.ys",
            "mount_point": "/mnt",
            "pid": "2084",
            "root": "/"
        }
    }
]
结论:

客户端通过遍历挂载目录下所有文件,发现服务端的session num_caps跟客户端文件夹及文件梳理匹配
也就是说客户端读取过的文件句柄,都会在服务端记录下来。 (mds缓存了dentry,并且以lru算法的缓存淘汰方式把dentry缓存在了内存中)
2.5.2 释放客户端session inode
解决方案:

方案1:采用多活mds(目前12版 multi active不稳定)
方案2:evict client(主动踢出有问题的客户端)
方案3:client remount(有问题的客户端重新mount挂载)
方案4:drop_cache, limit_cache
mds limiting cache by memory https://github.com/ceph/ceph/pull/17711
(官方提供的mds 主动删除cache,补丁在review过程中个,目标版本是ceph-14.0.0) https://github.com/ceph/ceph/pull/21566

image.png
3. 深入分析
根据上面的分析,我们基本有一定的思路。 这里我们继续深入到方案2中。

3.1 剔除客户端session
3.1.1 查看客户端session信息
$ ceph daemon mds.ceph-xxx-osd01.ys session ls
[
    {
        "id": 554418,
        "num_leases": 0,
        "num_caps": 1589,
        "state": "open",
        "replay_requests": 0,
        "completed_requests": 2,
        "reconnecting": false,
        "inst": "client.554418 192.168.1.2:0/1285681097",
        "client_metadata": {
            "ceph_sha1": "fe3a2269d799a8b950404cb2de11af84c7af0ea4",
            "ceph_version": "didi_dss version 12.2.2.4 (fe3a2269d799a8b950404cb2de11af84c7af0ea4) luminous (stable)",
            "entity_id": "admin",
            "hostname": "ceph-xxx-osd01.ys",
            "mount_point": "/mnt",
            "pid": "2084",
            "root": "/"
        }
    }
]
3.1.2 剔除客户端session信息
$ ceph tell mds.ceph-xxx-osd01.ys client evict id=554418
3.1.3 查看osd的blacklist
#超时恢复的时间是1小时,剔除的时间是16:16:30, 恢复的时间是17:16:30
$ ceph osd blacklist ls
listed 1 entries
192.168.1.2:0/1285681097 2018-10-10 17:16:30.819201
3.1.4 查看客户端挂载目录(不能读写)
$ ll /mnt
ls: cannot access /mnt: No such file or directory
3.1.5 恢复剔除的客户端
$ ceph osd blacklist rm 192.168.1.2:0/1285681097
un-blacklisting 192.168.1.2:0/1285681097
3.1.6 查看客户端挂载目录(正常读写)
$ ll /mnt
total 147698
-rw-r--r-- 1 root root         4 Oct 10 15:25 aa.txt
...
3.1.7 osd黑名单的客户端超时时间
旧版本超时时间为1小时
新版本12.2.2 超时时间为300s
总结:

可以通过指令client evict 剔除指定的客户端
剔除的客户端会加入到osd黑名单中
加入到osd黑名单中的客户端都不能读写
恢复剔除的客户端需要删除osd黑名单中的客户端信息
删除osd黑名单中客户端信息,客户端立马能正常读写
fuse客户端可以恢复,kernel客户端无法恢复
经过试验证明:
剔除的用户虽然释放了inode
主mds的内存并未释放
主从切换后,备mds内存会释放
主从切换后,切换速度少了加载inode耗时的阶段,从而加快切换速度,秒级别
3.2 内存未释放分析
3.2.1 依赖软件
yum install google-perftools
3.2.2 查看mds内存
top - 13:14:06 up 63 days, 21:36,  1 user,  load average: 0.06, 0.08, 0.12
Tasks:   1 total,   0 running,   1 sleeping,   0 stopped,   0 zombie
%Cpu(s):  0.2 us,  0.1 sy,  0.0 ni, 99.7 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
KiB Mem : 13149521+total, 96957576 free, 10023744 used, 24513896 buff/cache
KiB Swap:        0 total,        0 free,        0 used. 11539159+avail Mem
  PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND
 4997 ceph      20   0 4081012 1.447g  11452 S   0.0  1.2   0:54.29 ceph-mds
3.2.3 启动剖析器
$ ceph tell mds.0 heap start_profiler
2018-10-12 13:15:35.979279 7f3430bfa700  0 client.5796596 ms_handle_reset on 192.168.1.2:6804/2252738073
2018-10-12 13:15:36.008686 7f34293fc700  0 client.5796599 ms_handle_reset on 192.168.1.2:6804/2252738073
mds.ceph-xxx-osd01.ys started profiler
3.2.4 转储堆栈信息
$ ceph tell mds.0 heap dump
2018-10-12 13:16:34.891671 7efd04bfa700  0 client.5796659 ms_handle_reset on 192.168.1.2:6804/2252738073
2018-10-12 13:16:34.922696 7efcfd3fc700  0 client.5796662 ms_handle_reset on 192.168.1.2:6804/2252738073
mds.ceph-xxx-osd01.ys dumping heap profile now.
------------------------------------------------
MALLOC:     1225155304 ( 1168.4 MiB) Bytes in use by application
MALLOC: +            0 (    0.0 MiB) Bytes in page heap freelist
MALLOC: +    289987072 (  276.6 MiB) Bytes in central cache freelist
MALLOC: +     11013456 (   10.5 MiB) Bytes in transfer cache freelist
MALLOC: +      7165384 (    6.8 MiB) Bytes in thread cache freelists
MALLOC: +      7598240 (    7.2 MiB) Bytes in malloc metadata
MALLOC:   ------------
MALLOC: =   1540919456 ( 1469.5 MiB) Actual memory used (physical + swap)
MALLOC: +    112582656 (  107.4 MiB) Bytes released to OS (aka unmapped)
MALLOC:   ------------
MALLOC: =   1653502112 ( 1576.9 MiB) Virtual address space used
MALLOC:
MALLOC:          94545              Spans in use
MALLOC:             16              Thread heaps in use
MALLOC:           8192              Tcmalloc page size
------------------------------------------------
Call ReleaseFreeMemory() to release freelist memory to the OS (via madvise()).
Bytes released to the OS take up virtual address space but no physical memory.
3.2.5 google-pprof分析内存堆栈
pprof --text bin/ceph-mds out/mds.a.profile.0001.heap
$ pprof --text bin/ceph-mds out/mds.a.profile.0008.heap
Using local file bin/ceph-mds.
Using local file out/mds.a.profile.0008.heap.
Total: 46.6 MB
    18.1  38.7%  38.7%     19.5  41.9% Server::prepare_new_inode
     6.2  13.3%  52.0%      6.2  13.3% std::_Rb_tree::_M_emplace_hint_unique (inline)
     5.0  10.7%  62.7%      5.8  12.3% CDir::add_null_dentry
     3.8   8.1%  70.8%      3.8   8.1% std::_Rb_tree::_Rb_tree_impl::_M_initialize (inline)
     3.6   7.7%  78.6%      3.6   7.7% ceph::logging::Log::create_entry
     3.1   6.7%  85.2%      3.1   6.7% Counter::_count (inline)
     2.6   5.5%  90.7%      2.6   5.5% ceph::buffer::raw_combined::create (inline)
     0.9   2.0%  92.8%      0.9   2.0% std::_Vector_base::_M_create_storage (inline)
     0.8   1.6%  94.4%      0.8   1.6% CDir::add_null_dentry (inline)
     0.6   1.2%  95.6%      0.6   1.2% CInode::add_client_cap (inline)
     0.5   1.1%  96.6%      0.5   1.1% std::string::_Rep::_S_create
     0.5   1.0%  97.6%      0.5   1.0% MDCache::add_inode (inline)
     0.2   0.5%  98.2%      0.3   0.6% decode_message
     0.2   0.4%  98.5%      0.2   0.5% MDCache::request_start (inline)
     0.1   0.2%  98.7%      0.1   0.3% CInode::project_inode
     0.1   0.2%  99.0%      0.1   0.2% std::_Rb_tree::_M_insert_unique (inline)
     0.1   0.1%  99.1%      0.1   0.1% std::string::_M_data (inline)
4. 总结
cephfs mds目前版本都是单活,对于复杂多变的客户端可能会带来一定的性能影响。
例如:在本地磁盘下去搜索一个文件,如果文件数过多的,对本机cpu以及负载也会带来一定的冲击,更何况是复杂多变的网络磁盘。

目前推荐的优化方案:

ceph-fuse客户端Qos限速,避免IO一瞬间涌进来导致mds抖动(从客户端限制IOPS,避免资源争抢,对系统资源带来冲击)
多活mds, 目录分片(最终解决方案,详见:多活MDS的性能测试 )
mds在主处理流程中使用了单线程,这导致了其单个MDS的性能受到了限制,最大单个MDS可达8k ops/s,CPU利用率达到的 140%左右。
如果mds负载过高或者内存过大,限制内存或者定期的回收cache(减轻mds的压力,提升吞吐 https://github.com/ceph/ceph/pull/17711/files)
剔除用户可以释放inode数量,但是不能减少内存,如果此时切换主从可以加快切换速度。

原文链接:https://blog.csdn.net/weixin_44389885/article/details/86621701

posted on 2022-10-04 01:21  bdy  阅读(320)  评论(0编辑  收藏  举报

导航