codis部署安装及性能测试
git clone https://github.com/CodisLabs/codis.git
1:下载go环境 官网：https://golang.org/doc/install
wget https://storage.googleapis.com/golang/go1.6.2.linux-amd64.tar.gz
tar -C /usr/local -xzf go1.6.2.linux-amd64.tar.gz
cd /usr/local/go
cat >> /etc/profile <<EOF
export GOROOT=/usr/local/go
export GOPATH=/workspace/golang
export PATH=\$PATH:\$GOROOT/bin:\$GOPATH/bin
export CODIS_HOME=$GOPATH/src/github.com/CodisLabs/codis
export GO15VENDOREXPERIMENT=0
EOF
source /etc/profile
2:下载安装codis 官网：https://github.com/CodisLabs/codis/blob/master/doc/tutorial_zh.md 根据官网推荐采用这种方式，而不是git clone ...codis
go get -u -d github.com/CodisLabs/codis
go get github.com/tools/godep
cd $GOPATH/src/github.com/CodisLabs/codis
make
make gotest
#mkdir -p /usr/local/codis/{logs,conf,scripts}
#mkdir -p /data/codis_server/{logs,conf,data}
cd $GOPATH/src/github.com/CodisLabs/codis/bin
3:安装zookeeper
wget http://mirror.bit.edu.cn/apache/zookeeper/zookeeper-3.4.6/zookeeper-3.4.6.tar.gz -P /opt
tar -C /opt -xvf /opt/zookeeper-3.4.6.tar.gz
cd /opt/zookeeper-3.4.6/conf
cp zoo_sample.cfg zoo.cfg
/opt/zookeeper-3.4.6/bin/zkServer.sh start
read -p "请输入zookeeper的地址: " zookeeper-ip
sed -i 's/192\.168\.0\.123/$zookeeper-ip/g' config.ini
###assets //dashboard的静态文件目录
###codis-config codis命令式配置管理组件
###codis-proxy codis 代理也就是核心组件
###codis-server codis基于redis2.8.21定制版redis
mkdir /workspace/golang/src/github.com/CodisLabs/codis/wk
cp /workspace/golang/src/github.com/CodisLabs/codis/config.ini /workspace/golang/src/github.com/CodisLabs/codis/wk/
4:启动 dashboard（codis-config上操作）
cd $CODIS_HOME && ./bin/codis-config -c $CODIS_HOME/wk/config.ini dashboard &
可以从浏览器打开http://192.168.16.239:18087/admin/
5:初始化 slot,该命令会在zookeeper上创建slot相关信息
cd $CODIS_HOME && ./bin/codis-config -c $CODIS_HOME/wk/config.ini slot init &
6:启动 Codis Redis , 和官方的Redis Server参数一样（codis-server上操作）
cp extern/redis-2.8.21/redis.conf wk/
cd $CODIS_HOME && ./codis-server ../extern/redis-2.8.21/redis.conf (端口6379)作为16.241的6380的master
./codis-server ../extern/redis-2.8.21/redis6380.conf （端口6380)作为16.241的6379的slave
7:添加 Redis Server Group , 每一个 Server Group 作为一个 Redis 服务器组存在, 只允许有一个 master,
可以有多个 slave, group id 仅支持大于等于1的整数（codis-config上操作）
./codis-config -c ../wk/config.ini server add 1 192.168.16.239:6379 master
./codis-config -c ../wk/config.ini server add 1 192.168.16.241:6380 slave
./codis-config -c ../wk/config.ini server add 2 192.168.16.241:6379 master
./codis-config -c ../wk/config.ini server add 2 192.168.16.239:6380 slave
8: 启动241上面的redis服务，作为slave
/codis-server ../extern/redis-2.8.21/redis.conf 作为16.239的6380 的master
/codis-server ../extern/redis-2.8.21/redis6380.conf 作为16.239的6379 的slave
****************注意，不需要从redis.conf 去配置slave，必须用codis-server 去启动redis***************
[root@shell bin]# 2016/04/28 14:16:48 dashboard.go:160: [INFO] dashboard listening on addr: :18087
2016/04/28 14:16:49 dashboard.go:234: [PANIC] create zk node failed
[error]: dashboard already exists: {"addr": "192.168.16.239:18087", "pid": 2280}
[stack]:
3 /workspace/golang/src/github.com/CodisLabs/codis/cmd/cconfig/dashboard.go:234
main.runDashboard
2 /workspace/golang/src/github.com/CodisLabs/codis/cmd/cconfig/dashboard.go:54
main.cmdDashboard
1 /workspace/golang/src/github.com/CodisLabs/codis/cmd/cconfig/main.go:85
main.runCommand
0 /workspace/golang/src/github.com/CodisLabs/codis/cmd/cconfig/main.go:152
main.main
... ...
解决：
./zkCli.sh
[zk: localhost:2181(CONNECTED) 11] rmr /zk/codis
原因：
2016/04/28 14:52:44 dashboard.go:143: [INFO] dashboard node created: /zk/codis/db_wk/dashboard, {"addr": "192.168.16.239:18087", "pid": 8578}
2016/04/28 14:52:44 dashboard.go:144: [WARN] ********** Attention **********
2016/04/28 14:52:44 dashboard.go:145: [WARN] You should use `kill {pid}` rather than `kill -9 {pid}` to stop me,
2016/04/28 14:52:44 dashboard.go:146: [WARN] or the node resisted on zk will not be cleaned when I'm quiting and you must remove it manually
9:设置 server group 服务的 slot 范围
Codis 采用 Pre-sharding 的技术来实现数据的分片, 默认分成 1024 个 slots (0-1023), 对于每个key来说, 通过以下公式确定所属的
Slot Id : SlotId = crc32(key) % 1024 每一个 slot 都会有一个且必须有一个特定的 server group id 来表示这个 slot 的数据由哪个 server group 来提供
[root@shell bin]# ./codis-config -c ../wk/config.ini slot -h
usage:
codis-config slot init [-f]
codis-config slot info <slot_id>
codis-config slot set <slot_id> <group_id> <status>
codis-config slot range-set <slot_from> <slot_to> <group_id> <status>
codis-config slot migrate <slot_from> <slot_to> <group_id> [--delay=<delay_time_in_ms>]
codis-config slot rebalance [--delay=<delay_time_in_ms>]
[root@shell bin]# ./codis-config -c ../wk/config.ini slot range-set 0 511 1 online
{
"msg": "OK",
"ret": 0
}
到浏览器上看看效果
或者
[root@shell bin]# ./codis-config -c ../wk/config.ini slot info 1
{
"group_id": 1,
"id": 1,
"product_name": "wk",
"state": {
"last_op_ts": "0",
"migrate_status": {
"from": -1,
"to": -1
},
"status": "online"
}
10：启动 codis-proxy
[root@shell bin]# ./codis-proxy -c ../wk/config.ini -L ./log/proxy.log --cpu=4 --addr=0.0.0.0:19000 --http-addr=0.0.0.0:11000 &
_____ ____ ____/ / (_) _____
/ ___/ / __ \ / __ / / / / ___/
/ /__ / /_/ / / /_/ / / / (__ )
\___/ \____/ \__,_/ /_/ /____/
在241上启动第二个代理服务器
./codis-proxy -c ../wk/config.ini -L ./log/proxy.log --cpu=4 --addr=0.0.0.0:19000 --http-addr=0.0.0.0:11000 &
在浏览器上看到proxy_2是offile状态，如果用命令去启动proxy_2 online状态，则提示下面错误，需要去启动dashboard,我们直接在界面上启动为online状态
[root@zabbix bin]# ./codis-config -c ../wk/config.ini proxy online proxy_2
2016/05/03 13:43:12 utils.go:49: [ERROR] can't connect to dashboard, please check 'dashboard_addr' is corrent in config file
[stack]:
11：用单个proxy进行测试，16.239proxy,可以用redis客户端连接proxy 的端口
[root@shell src]# ./redis-cli -p 19000
127.0.0.1:19000> info
...
# Replication
role:master
connected_slaves:1
slave0:ip=192.168.16.241,port=6380,state=online,offset=722094,lag=1
master_repl_offset:722094
repl_backlog_active:1
repl_backlog_size:1048576
repl_backlog_first_byte_offset:23019
repl_backlog_histlen:699076
...
插入数据
[root@shell opt]# vim redis-setkey.sh
#!/bin/bash
n=100000
for (( i=n ;i>=1 ; i-- ))
do
echo $i
redis-cli -p 19000 set zhao$i aa
done
根据下面脚本查看key的分布情况
#!/bin/bash
n=100000
for ((i=n;i>=0;i--))
do
echo $i >> /opt/a.txt
redis-cli -h 192.168.16.241 -p 6379 get zhao$i >>/opt/a.txt
done
sed 'N;s#\n# #g' a.txt | awk '{if($2=="aa") print $1}' | wc -l
结果为：50029
result=`redis-cli -h 192.168.16.241 -p 6379 get zhao$i`
if [ ! -z $result ];then
echo $i >>/opt/b.txt
fi
[root@shell opt]# cat b.txt | wc -l
50029
结果都一样，发现192.168.16.241 -p 6379 存了500029条数据，再查看下16.239上的6379
[root@shell opt]# cat b.txt | wc -l
49971
正好结果相加是对的
查看主从复制及数据是否一致：
result=`redis-cli -h 192.168.16.239 -p 6380 get zhao$i`
if [ ! -z $result ];then
echo $i >>/opt/b.txt
fi
[root@shell opt]# cat b.txt | wc -l
50029
结果跟16.241 6379上的数据保持一致。
12：性能测试：
1）10000请求，200并发，上是codis，下是单机
[root@shell src]# ./redis-benchmark -p 19000 -n 10000 -c 200 -q
PING_INLINE: 52083.33 requests per second
PING_BULK: 52083.33 requests per second
SET: 38314.18 requests per second
GET: 41666.67 requests per second
INCR: 37174.72 requests per second
LPUSH: 36496.35 requests per second
LPOP: 38610.04 requests per second
SADD: 37313.43 requests per second
SPOP: 36101.08 requests per second
LPUSH (needed to benchmark LRANGE): 34843.21 requests per second
LRANGE_100 (first 100 elements): 14430.01 requests per second
LRANGE_300 (first 300 elements): 5241.09 requests per second
LRANGE_500 (first 450 elements): 3713.33 requests per second
LRANGE_600 (first 600 elements): 2627.43 requests per second
MSET (10 keys): 16556.29 requests per second
[root@shell src]# ./redis-benchmark -p 6379 -n 10000 -c 200 -q
PING_INLINE: 67567.57 requests per second
PING_BULK: 73529.41 requests per second
SET: 73529.41 requests per second
GET: 73529.41 requests per second
INCR: 59523.81 requests per second
LPUSH: 68493.15 requests per second
LPOP: 62893.08 requests per second
SADD: 76335.88 requests per second
SPOP: 63291.14 requests per second
LPUSH (needed to benchmark LRANGE): 68965.52 requests per second
LRANGE_100 (first 100 elements): 36496.35 requests per second
LRANGE_300 (first 300 elements): 15748.03 requests per second
LRANGE_500 (first 450 elements): 11441.65 requests per second
LRANGE_600 (first 600 elements): 8928.57 requests per second
MSET (10 keys): 40160.64 requests per second
2）10000请求，500并发，上是codis，下是单机
[root@shell src]# ./redis-benchmark -p 19000 -n 10000 -c 500 -q
PING_INLINE: 55865.92 requests per second
PING_BULK: 55248.62 requests per second
SET: 35087.72 requests per second
GET: 49751.24 requests per second
INCR: 38167.94 requests per second
LPUSH: 34246.57 requests per second
LPOP: 36101.08 requests per second
SADD: 33112.59 requests per second
SPOP: 42553.19 requests per second
LPUSH (needed to benchmark LRANGE): 38461.54 requests per second
LRANGE_100 (first 100 elements): 15151.51 requests per second
LRANGE_300 (first 300 elements): 5238.34 requests per second
LRANGE_500 (first 450 elements): 3716.09 requests per second
LRANGE_600 (first 600 elements): 2680.25 requests per second
MSET (10 keys): 15948.96 requests per second
[root@shell src]# ./redis-benchmark -p 6379 -n 10000 -c 500 -q
PING_INLINE: 70422.53 requests per second
PING_BULK: 69444.45 requests per second
SET: 58823.53 requests per second
GET: 70921.98 requests per second
INCR: 66666.66 requests per second
LPUSH: 65789.48 requests per second
LPOP: 63694.27 requests per second
SADD: 68493.15 requests per second
SPOP: 67114.09 requests per second
LPUSH (needed to benchmark LRANGE): 66225.17 requests per second
LRANGE_100 (first 100 elements): 35714.29 requests per second
LRANGE_300 (first 300 elements): 15847.86 requests per second
LRANGE_500 (first 450 elements): 11918.95 requests per second
LRANGE_600 (first 600 elements): 9107.47 requests per second
MSET (10 keys): 38314.18 requests per second
3）10000请求，1000并发，上是codis，下是单机
[root@shell src]# ./redis-benchmark -p 19000 -n 10000 -c 1000 -q
PING_INLINE: 47846.89 requests per second
PING_BULK: 41493.78 requests per second
SET: 30769.23 requests per second
GET: 40322.58 requests per second
INCR: 38910.51 requests per second
LPUSH: 39370.08 requests per second
LPOP: 37453.18 requests per second
SADD: 40000.00 requests per second
SPOP: 42553.19 requests per second
LPUSH (needed to benchmark LRANGE): 36764.70 requests per second
LRANGE_100 (first 100 elements): 15384.62 requests per second
LRANGE_300 (first 300 elements): 5353.32 requests per second
LRANGE_500 (first 450 elements): 3776.44 requests per second
LRANGE_600 (first 600 elements): 2677.38 requests per second
MSET (10 keys): 14641.29 requests per second
[root@shell src]# ./redis-benchmark -p 6379 -n 10000 -c 1000 -q
PING_INLINE: 65789.48 requests per second
PING_BULK: 65789.48 requests per second
SET: 63694.27 requests per second
GET: 67114.09 requests per second
INCR: 66666.66 requests per second
LPUSH: 64516.13 requests per second
LPOP: 66666.66 requests per second
SADD: 67114.09 requests per second
SPOP: 65789.48 requests per second
LPUSH (needed to benchmark LRANGE): 63694.27 requests per second
LRANGE_100 (first 100 elements): 34843.21 requests per second
LRANGE_300 (first 300 elements): 15243.90 requests per second
LRANGE_500 (first 450 elements): 11682.24 requests per second
LRANGE_600 (first 600 elements): 9082.65 requests per second
MSET (10 keys): 37174.72 requests per second
4）10000 请求 2000并发
[root@shell src]# ./redis-benchmark -p 19000 -n 10000 -c 2000 -q
PING_INLINE: 37313.43 requests per second
PING_BULK: 42735.04 requests per second
SET: 29940.12 requests per second
GET: 31250.00 requests per second
INCR: 29411.76 requests per second
LPUSH: 29585.80 requests per second
LPOP: 23752.97 requests per second
SADD: 21978.02 requests per second
SPOP: 34129.69 requests per second
LPUSH (needed to benchmark LRANGE): 6600.66 requests per second
LRANGE_100 (first 100 elements): 10582.01 requests per second
LRANGE_300 (first 300 elements): 2314.81 requests per second
LRANGE_500 (first 450 elements): 2180.07 requests per second
LRANGE_600 (first 600 elements): 2630.19 requests per second
MSET (10 keys): 8481.76 requests per second
[root@shell src]# ./redis-benchmark -p 6379 -n 10000 -c 2000 -q
PING_INLINE: 49504.95 requests per second
PING_BULK: 64102.56 requests per second
SET: 58823.53 requests per second
GET: 64102.56 requests per second
INCR: 51020.41 requests per second
LPUSH: 43859.65 requests per second
LPOP: 15948.96 requests per second
SADD: 55248.62 requests per second
SPOP: 62500.00 requests per second
LPUSH (needed to benchmark LRANGE): 61349.69 requests per second
LRANGE_100 (first 100 elements): 30211.48 requests per second
LRANGE_300 (first 300 elements): 16000.00 requests per second
LRANGE_500 (first 450 elements): 11123.47 requests per second
LRANGE_600 (first 600 elements): 8271.30 requests per second
MSET (10 keys): 31746.03 requests per second
