cephfs时我们产品依赖的主要分布式操作系统，但似乎很不给面子，压力测试的时候经常出问题。
背景
集群环境出现的问题: mds0: Many clients (191) failing to respond to cache pressure
背景：三个节点，100多个客户端mount，服务器可用内存仅剩100MB，ceph报错如下：
[root@node1 ceph]# ceph -s
cluster 1338affa-2d3d-416e-9251-4aa6e9c20eef
health HEALTH_WARN
mds0: Many clients (191) failing to respond to cache pressure
monmap e1: 3 mons at {node1=192.168.0.1:6789/0,node2=192.168.0.2:6789/0,node3=192.168.0.3:6789/0}
election epoch 22, quorum 0,1,2 node1,node2,node3
fsmap e924: 1/1/1 up {0=node1=up:active}, 2 up:standby
osdmap e71: 3 osds: 3 up, 3 in
flags sortbitwise,require_jewel_osds
pgmap v48336: 576 pgs, 3 pools, 82382 MB data, 176 kobjects
162 GB used, 5963 GB / 6126 GB avail
576 active+clean
client io 0 B/s rd, 977 kB/s wr, 19 op/s rd, 116 op/s wr
至今问题也没有解决。(我的意思是说没有弄清楚Capacity的机制，如果抱着解决不了问题，就解决提出问题的人的思路，可以参考第三部分。)
mds日志如下：
2019-11-12 16:00:17.679876 7fa6a5040700 0 log_channel(cluster) log [WRN] : 1 slow requests, 1 included below; oldest blocked for > 34.236623 secs
2019-11-12 16:00:17.679914 7fa6a5040700 0 log_channel(cluster) log [WRN] : slow request 34.236623 seconds old, received at 2019-11-12 15:59:43.326917: client_request(client.154893:13683 open #1000005cb77 2019-11-12 15:59:43.293037) currently failed to xlock, waiting
2019-11-12 16:03:27.614474 7fa6a5040700 0 log_channel(cluster) log [WRN] : 1 slow requests, 1 included below; oldest blocked for > 34.350555 secs
2019-11-12 16:03:27.614523 7fa6a5040700 0 log_channel(cluster) log [WRN] : slow request 34.350555 seconds old, received at 2019-11-12 16:02:53.263857: client_request(client.155079:5446 open #1000003e360 2019-11-12 16:02:54.011037) currently failed to xlock, waiting
2019-11-12 16:03:57.615297 7fa6a5040700 0 log_channel(cluster) log [WRN] : 1 slow requests, 1 included below; oldest blocked for > 64.351379 secs
2019-11-12 16:03:57.615322 7fa6a5040700 0 log_channel(cluster) log [WRN] : slow request 64.351379 seconds old, received at 2019-11-12 16:02:53.263857: client_request(client.155079:5446 open #1000003e360 2019-11-12 16:02:54.011037) currently failed to xlock, waiting
2019-11-12 16:03:58.181330 7fa6a5040700 0 log_channel(cluster) log [WRN] : client.155079 isn't responding to mclientcaps(revoke), ino 1000003e360 pending pAsxLsXsxFcb issued pAsxLsXsxFsxcrwb, sent 64.458260 seconds ago
后续的努力
自己找环境重现，用的一个测试服务器，安装了一个Ubuntu系统，然后进行测试。惊喜的发现，同一个客户端不管我mount多少个目录，与后端的连接始终都只有那两个。
但重现过程中还是出现类似的问题了。
mds0: Client ubuntu:guest failing to respond to capability release
静置一段时间之后出现了如下错误:
[root@ceph741 ~]# ceph -s
cluster 1338affa-2d3d-416e-9251-4aa6e9c20eef
health HEALTH_WARN
mds0: Client ubuntu:guest failing to respond to capability release
mds0: Client ubuntu:guest failing to advance its oldest client/flush tid
monmap e2: 3 mons at {ceph741=192.168.15.112:6789/0,ceph742=192.168.15.113:6789/0,ceph743=192.168.15.114:6789/0}
election epoch 38, quorum 0,1,2 ceph741,ceph742,ceph743
fsmap e8989: 1/1/1 up {0=ceph743=up:active}, 2 up:standby
osdmap e67: 3 osds: 3 up, 3 in
flags sortbitwise,require_jewel_osds
pgmap v847657: 576 pgs, 3 pools, 20803 MB data, 100907 objects
44454 MB used, 241 GB / 284 GB avail
576 active+clean
client io 59739 B/s rd, 3926 kB/s wr, 58 op/s rd, 770 op/s wr
临时的解决办法
临时的解决办法就是把出问题的客户端干掉。
步骤主要命令：
ceph tell mds.0 session ls
ceph tell mds.0 session evict id=249632
其中id是问题client的id。那么问题客户端比其他客户端哪里不同呢，实话说，我也不知道，大家可以看下：
参考：
https://www.jianshu.com/p/d1e0e32346ac
http://www.talkwithtrend.com/Article/242905
https://www.jianshu.com/p/fa49e40f6133
再后续的努力
找到一些资料，高版本的解决方法：
ceph daemon mds.node-mds01 config set mds_cache_memory_limit 10737418240
也就是调大mds_cache_memory_limit 的值；
但是低版本的ceph没有mds_cache_memory_limit这个参数怎么办？
临时的办法:修改mds cache size的值;但是此值是不能动态修改的；
所以只能修改配置文件重启mds
[mds]
mds cache size = 2000000
参考网址：
https://blog.csdn.net/zzboat0422/article/details/98771817
https://www.jianshu.com/p/b30ac22bf418
https://docs.ceph.com/en/nautilus/cephfs/cache-size-limits/#:~:text=By default%2C mds_health_cache_thresholdis 150%25 of the maximum cache,cause the MDS to exceed its cache size.
https://cloud.tencent.com/developer/article/1643559
https://www.cnblogs.com/zphj1987/p/13671961.html
http://cephdocs.s3-website.cern.ch/ops/cephfs_warnings.html#:~:text=Ceph logging [WRN] clients failing to respond to,recall request%2C then Ceph will log this warning.
https://www.cnblogs.com/Malphite/p/7742406.html
https://docs.ceph.com/en/latest/cephfs/client-config-ref/
https://blog.csdn.net/for_tech/article/details/72123223
