使用cgroups来控制磁盘IO带宽
磨砺技术珠矶，践行数据之道，追求卓越价值
回到上一级页面：PostgreSQL内部结构与源代码研究索引页 回到顶级页面：PostgreSQL索引页
[作者 高健@博客园 luckyjackgao@gmail.com]
可资参考的资料：
http://kaivanov.blogspot.com/2012/07/setting-up-linux-cgroups-control-groups.html
首先用 hdparm测试磁盘读取的最大带宽:
hdparm --direct -t /dev/sda
测试结果为200MB/s以上。
然后给用户postgres设置参数，保持磁盘读写I/O为10MB/秒以下（此用户名下所有进程总和）。
# Configuration file generated by cgsnapshot
mount {
cpuset = /cgroup/cpuset;
cpu = /cgroup/cpu;
cpuacct = /cgroup/cpuacct;
memory = /cgroup/memory;
devices = /cgroup/devices;
freezer = /cgroup/freezer;
net_cls = /cgroup/net_cls;
blkio = /cgroup/blkio;
}
group io-test {
perm {
task{
uid=postgres;
gid=postgres;
}
admin{
uid=root;
gid=root;
}
} blkio {
blkio.throttle.write_iops_device="";
blkio.throttle.read_iops_device="";
blkio.throttle.write_bps_device="8:0 10485760";
blkio.throttle.read_bps_device="8:0 10485760";
blkio.reset_stats="";
blkio.weight="500";
blkio.weight_device="";
}
}
[postgres@cent6 Desktop]$
再看
[postgres@cent6 Desktop]$ cat /etc/cgrules.conf
# /etc/cgrules.conf
#
#Each line describes a rule for a user in the forms:
#
#<user> <controllers> <destination>
#<user>:<process name> <controllers> <destination>
#
#Where:
# <user> can be:
# - an user name
# - a group name, with @group syntax
# - the wildcard *, for any user or group.
# - The %, which is equivalent to "ditto". This is useful for
# multiline rules where different cgroups need to be specified
# for various hierarchies for a single user.
#
# <process name> is optional and it can be:
# - a process name
# - a full command path of a process
#
# <controller> can be:
# - comma separated controller names (no spaces)
# - * (for all mounted controllers)
#
# <destination> can be:
# - path with-in the controller hierarchy (ex. pgrp1/gid1/uid1)
#
# Note:
# - It currently has rules based on uids, gids and process name.
#
# - Don't put overlapping rules. First rule which matches the criteria
# will be executed.
#
# - Multiline rules can be specified for specifying different cgroups
# for multiple hierarchies. In the example below, user "peter" has
# specified 2 line rule. First line says put peter's task in test1/
# dir for "cpu" controller and second line says put peter's tasks in
# test2/ dir for memory controller. Make a note of "%" sign in second line.
# This is an indication that it is continuation of previous rule.
#
#
#<user> <controllers> <destination>
#
#john cpu usergroup/faculty/john/
#john:cp cpu usergroup/faculty/john/cp
#@student cpu,memory usergroup/student/
#peter cpu test1/
#% memory test2/
#@root * admingroup/
#* * default/
postgres blkio io-test/
# End of file
#
#
#
[postgres@cent6 Desktop]$
先用dd命令实际测试看看：
[postgres@cent6 Desktop]$ dd if=/dev/zero of=testfile2 bs=4K count=1024 oflag=direct
1024+0 records in
1024+0 records out
4194304 bytes (4.2 MB) copied, 0.372953 s, 11.2 MB/s
[postgres@cent6 Desktop]$ dd if=/dev/zero of=testfile2 bs=8K count=1024 oflag=direct
1024+0 records in
1024+0 records out
8388608 bytes (8.4 MB) copied, 0.733823 s, 11.4 MB/s
[postgres@cent6 Desktop]$ dd if=/dev/zero of=testfile2 bs=8K count=1024 oflag=direct
1024+0 records in
1024+0 records out
8388608 bytes (8.4 MB) copied, 0.733256 s, 11.4 MB/s
[postgres@cent6 Desktop]$
[postgres@cent6 Desktop]$ dd if=/dev/zero of=testfile2 bs=16K count=1024 oflag=direct
1024+0 records in
1024+0 records out
[postgres@cent6 Desktop]$
当然，如果dd执行时，开数据块太多，又是这种完全空的块，误差就会增大：
[postgres@cent6 Desktop]$ dd if=/dev/zero of=testfile1 bs=10240 count=300000
300000+0 records in
300000+0 records out
3072000000 bytes (3.1 GB) copied, 57.6779 s, 53.3 MB/s
[postgres@cent6 Desktop]$
用实际的下载动作进行测试：
[postgres@cent6 Desktop]$ wget http://centos.arcticnetwork.ca/6.4/isos/x86_64/CentOS-6.4-x86_64-LiveCD.iso
再看看IO状态：
[postgres@cent6 Desktop]$ iostat -x 20
Linux 2.6.32-279.el6.x86_64 (cent6.gao) 09/10/2013 _x86_64_ (1 CPU)
avg-cpu: %user %nice %system %iowait %steal %idle
1.38 0.00 3.55 12.16 0.00 82.91
Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s avgrq-sz avgqu-sz await svctm %util
sda 7.50 1747.20 7.77 24.88 535.75 14224.09 452.17 12.54 384.25 4.41 14.40
dm-0 0.00 0.00 13.79 1771.99 524.65 14224.07 8.26 1975.37 1106.16 0.09 16.73
dm-1 0.00 0.00 0.38 0.00 3.03 0.00 8.00 0.00 3.68 2.34 0.09
avg-cpu: %user %nice %system %iowait %steal %idle
3.65 0.00 2.14 0.73 0.00 93.48
Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s avgrq-sz avgqu-sz await svctm %util
sda 0.00 1.82 1.09 1.56 49.64 25.44 28.24 0.03 11.78 3.57 0.95
dm-0 0.00 0.00 1.04 3.18 48.80 25.44 17.58 0.03 7.35 2.15 0.91
dm-1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
avg-cpu: %user %nice %system %iowait %steal %idle
1.65 0.00 1.65 0.98 0.00 95.72
Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s avgrq-sz avgqu-sz await svctm %util
sda 0.00 60.28 0.00 1.44 0.00 492.12 341.14 0.02 11.79 8.86 1.28
dm-0 0.00 0.00 0.00 61.51 0.00 492.12 8.00 0.59 9.54 0.21 1.28
dm-1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
除了一开始的峰值，util长期在2%-3%之间，基本达到效果。
[作者 高健@博客园 luckyjackgao@gmail.com]
磨砺技术珠矶，践行数据之道，追求卓越价值
回到上一级页面：PostgreSQL内部结构与源代码研究索引页 回到顶级页面：PostgreSQL索引页
