.net ServiceStack.Redis 性能调优

     最近在debug生产环境的问题时,发现了ServiceStack 4.0.60版本RedisClient存在一个非常严重的性能问题。在高并发下,PooledRedisClientManager.GetClientRedis.DisposeClient会导致High CPU,并且持续非常长的时间才能自动修复下面是Demo程序压测还原问题后,工具的分析结果。

 

通过分析源代码发现:原来获取RedisClient的逻辑中通过锁方式实现,并且当连接被占满后再获取连接时,需要循环遍历数组中所有的连接对象判断是否有可用连接,会非常消耗CPU。Dispose方法也存在循环遍历的问题。尝试了很多种修改方案后,都不尽人意,果断把这两段逻辑重写,下面是相关代码,已经经过压测。

 

 

 

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
PooledRedisClientManager.cs:<br><br>        private ConcurrentQueue<RedisClient> deactiveClientQueue = new ConcurrentQueue<RedisClient>();
        private static object lckObj = new object();
        private static object waitObj = new object();
        private int redisClientSize = 0;
        private int maxRedisClient = 500; //PooledRedisClientManager的构造函数中初始化此值:maxRedisClient = this.Config.MaxWritePoolSize;
<br>        //GetReadOnlyClient方法也可按此方式修改
        public IRedisClient GetClient()
        {
            RedisClient client = null;
            var poolTimedOut = false;
            DateTime startTime = DateTime.Now;
 
 
            while (true)
            {
                bool getResult = deactiveClientQueue.TryDequeue(out client);
 
                if (getResult == false)
                {
                    if (redisClientSize >= maxRedisClient)
                    {
                        Thread.Sleep(3);
                        if (PoolTimeout.HasValue)
                        {
                            // wait for a connection, cry out if made to wait too long
                            if ((DateTime.Now - startTime).TotalMilliseconds >= PoolTimeout.Value)
                            {
                                poolTimedOut = true;
                                break;
                            }
                        }
 
                    }
                    else
                    {
                        client = CreateRedisClient();
                        if (client != null)
                            return client;
                    }
                }
                else
                {
                    if (client != null)
                    {
                        InitClient(client);
                        return client;
                    }
                    else
                    {
                        client = CreateRedisClient();
                        if (client != null)
                            return client;
                    }
                }
            }
 
            if (poolTimedOut == true)
            {
                throw new TimeoutException(PoolTimeoutError);
            }
 
            return client;
        }
 
 
        private RedisClient CreateRedisClient()
        {
            if (redisClientSize >= maxRedisClient)
                return null;
            lock (lckObj)
            {
                if (redisClientSize >= maxRedisClient)
                    return null;
                Random dom = new Random((int)DateTime.Now.Ticks);
                var newClient = InitNewClient(RedisResolver.CreateMasterClient(dom.Next(100)));
                newClient.OnDispose += (isRecycle) =>
                {
                    if (isRecycle == true)
                    {
                        try
                        {
                            deactiveClientQueue.Enqueue(newClient);
                        }
                        catch
                        {
                            lock (lckObj)
                            {
                                redisClientSize--;
                            }
                        }
                    }
                    else
                    {
                        lock (lckObj)
                        {
                            redisClientSize--;
                        }
                    }
                };
                redisClientSize++;
 
                return newClient;
            }
        }
RedisClient.cs:
public event RedisClientDisposeEventHandler OnDispose; public override void Dispose() { if (OnDispose != null) OnDispose(this.HadExceptions == false); base.Dispose(); }
RedisClient.cs:    
public delegate void RedisClientDisposeEventHandler(bool isRecycle);

下面是修改前后的结果对比:

1.100个线程,每个线程完成2000次Redis调用,每次调用GetClient。 改造前12s,改造后8.5s,提升近50%。老版本CPU消耗稍高,并具有持续性。

 

2.200个线程,每个线程完成2000次Redis调用,每次调用GetClient。 改造前378s,改造后19s,提升提升近20倍。老版本CPU消耗非常高(解决100%),并具有持续性。新版本CPU占用了仅有原来的一半。

 

3.300个线程,每个线程完成2000次Redis调用,每次调用GetClient。 改造前1580s(26分钟),改造后29s,提升提升近55倍老版本CPU消耗非常高(解决100%),并具有持续性。新版本CPU占用了仅有原来的一半。

 

    通过上述三个场景的测试可以看出,当RedisClient访问压力持续增加时,原版本的响应时间呈现指数性增长,当达到一定压力时,RedisClient访问几乎阻塞,需要非常长时间才能缓解。重构后的RedisClient在性能上有大幅度提升,特别是在高并发下的性能表现,直接秒杀原版本!

posted @   凌晨三点半  阅读(3611)  评论(1编辑  收藏  举报
努力加载评论中...
点击右上角即可分享
微信分享提示