Open-falcon监控
https://book.open-falcon.org/zh_0_2/
本文档记录了CentOS7.4下open-falcon-v2监控系统的部署流程，以及一些需要注意的地方。
环境准备
安装Redis
0.InstallRedis3.2.sh redis-3.2.3.tar.gz
安装mysql
略
将root让全部访问设置为%，将/tmp/mysql.sock链接到/app/mysqldata/3306/mysql.sock，将参数bind_address=127.0.0.1
初始化MySQL表结构
cd /tmp/ && git clone git://github.com/open-falcon/falcon-plus.git
cd /tmp/falcon-plus/scripts/mysql/db_schema/
mysql -h 127.0.0.1 -uroot -pmsds007 < 1_uic-db-schema.sql
mysql -h 127.0.0.1 -uroot -pmsds007 < 2_portal-db-schema.sql
mysql -h 127.0.0.1 -uroot -pmsds007 < 3_dashboard-db-schema.sql
mysql -h 127.0.0.1 -uroot -pmsds007 < 4_graph-db-schema.sql
mysql -h 127.0.0.1 -uroot -pmsds007 < 5_alarms-db-schema.sql
rm -rf /tmp/falcon-plus/
启动后端
open-falcon-v0.2.1.tar.gz 二进制包
创建工作目录
export FALCON_HOME=/home/work
export WORKSPACE=$FALCON_HOME/open-falcon
mkdir -p $WORKSPACE
解压二进制包
tar -xzvf open-falcon-v0.2.1.tar.gz -C $WORKSPACE
在一台机器上启动所有的后端组件
修改配置文件，确认配置文件中数据库账号密码与实际相同
# cd $WORKSPACE
# grep -Ilr 3306 ./
./nodata/config/cfg.json
./graph/config/cfg.json
./hbs/config/cfg.json
./alarm/config/cfg.json
./aggregator/config/cfg.json
./api/config/cfg.json
部分模块依赖连接数据库，需要修改配置文件
vim /home/work/open-falcon/aggregator/config/cfg.json
vim /home/work/open-falcon/graph/config/cfg.json
vim /home/work/open-falcon/hbs/config/cfg.json
vim /home/work/open-falcon/nodata/config/cfg.json
vim /home/work/open-falcon/api/config/cfg.json
vim /home/work/open-falcon/alarm/config/cfg.json
启动
cd $WORKSPACE
./open-falcon start
# 检查所有模块的启动状况
./open-falcon check
falcon-graph UP 3962
falcon-hbs UP 3970
falcon-judge UP 3978
falcon-transfer UP 3984
falcon-nodata UP 3990
falcon-aggregator UP 3996
falcon-agent UP 4003
falcon-gateway UP 4009
falcon-api UP 4016
falcon-alarm UP 4024
安装前端
python2.7安装pip
# unzip setuptools-38.5.1.zip
# cd setuptools-38.5.1
# python setup.py install
# tar -zxvf pip-9.0.1.tar.gz
# cd pip-9.0.1
# python setup.py install
克隆前端组件代码
cd $WORKSPACE
git clone git://github.com/open-falcon/dashboard.git
安装依赖包
yum install -y python-virtualenv
yum install -y python-devel
yum install -y openldap-devel
yum install -y mysql-devel
yum groupinstall "Development tools" -y
cd $WORKSPACE/dashboard/
virtualenv ./env
pip install -r pip_requirements.txt
ln -s /usr/local/mysql/lib/libmysqlclient.so.20 /usr/lib64/libmysqlclient.so.20
MySQL-python-1.2.5.zip会用到
修改配置
dashboard的配置文件为： '/home/work/open-falcon/dashboard/rrd/config.py'，请根据实际情况修改
## API_ADDR 表示后端api组件的地址
API_ADDR = "http://127.0.0.1:8080/api/v1"
## 根据实际情况，修改PORTAL_DB_*, 默认用户名为root，默认密码为""
## 根据实际情况，修改ALARM_DB_*, 默认用户名为root，默认密码为""
在生产环境启动
bash control start
bash control tail
http://127.0.0.1:8081
系统初始化
前端用户需要注册，第一个注册的root账户就是管理员，所以第一件事就是先注册root用户密码123456
注册完root用户，最好把注册功能关闭，可以通过root创建其他用户。需要修改api模块的配置文件，signup_disable改为true就可以了
部署agent
将agent目录复制到要监控的主机上
scp -r /home/work/open-falcon/agent/ mydb2:/home/work/open-falcon
scp -r /home/work/open-falcon/agent/ mydb3:/home/work/open-falcon
将open-falcon复制到要监控的主机上
scp /home/work/open-falcon/open-falcon mydb2:/home/work/open-falcon
scp /home/work/open-falcon/open-falcon mydb3:/home/work/open-falcon
编辑配置文件
# cat cfg.json
{
"debug": true,
"hostname": "mydb2",
"ip": "192.168.1.102",
"plugin": {
"enabled": false,
"dir": "./plugin",
"git": "https://github.com/open-falcon/plugin.git",
"logs": "./logs"
},
"heartbeat": {
"enabled": true,
"addr": "192.168.1.101:6030",
"interval": 60,
"timeout": 1000
},
"transfer": {
"enabled": true,
"addrs": [
"192.168.1.101:8433"
],
"interval": 60,
"timeout": 1000
},
"http": {
"enabled": true,
"listen": ":1988",
"backdoor": false
},
"collector": {
"ifacePrefix": ["eth", "em"],
"mountPoint": []
},
"default_tags": {
},
"ignore": {
"cpu.busy": true,
"df.bytes.free": true,
"df.bytes.total": true,
"df.bytes.used": true,
"df.bytes.used.percent": true,
"df.inodes.total": true,
"df.inodes.free": true,
"df.inodes.used": true,
"df.inodes.used.percent": true,
"mem.memtotal": true,
"mem.memused": true,
"mem.memused.percent": true,
"mem.memfree": true,
"mem.swaptotal": true,
"mem.swapused": true,
"mem.swapfree": true
}
}
/home/work/open-falcon/open-falcon start agent 启动进程
/home/work/open-falcon/open-falcon stop agent 停止进程
/home/work/open-falcon/open-falcon monitor agent 查看日志
mydb1是单MySQL
mydb2,mydb3是复制
监控MySQL
# tar -zxvf mymon.tar.gz
修改配置文件
# cat mon.cfg
[default]
log_file=/soft/mymon.log
# Panic 0
# Fatal 1
# Error 2
# Warn 3
# Info 4
# Debug 5
log_level=5
falcon_client=http://127.0.0.1:1988/v1/push
#自定义endpoint
endpoint=mydb2
[mysql]
user=root
password=msds007
host=127.0.0.1
port=3306
# cat slavestatus.sh
#!/bin/bash
source /etc/profile
ts=`date +%s`;
MySql_CMD="/usr/local/mysql/bin/mysql"
User=root
passwd=msds007
LogLevel="error"
#LogLevel="debug"
Endpoint=`hostname`
Port=3306
Slave_IO_Metric="Slave_IO_Status"
Slave_SQL_Metric="Slave_SQL_Status"
Normal_Value=1
NonNormal_Value=0
CurrentPath=$(dirname $0)
Logfile=$CurrentPath/out.log
#Ip_Address=`ifconfig -a|grep "inet addr"|grep 10|awk '{print $2}'|cut -d : -f 2`
Ip_Address=127.0.0.1
IOSTATUS=`$MySql_CMD -u$User -p$passwd -S /app/mysqldata/3306/mysql.sock -e "show slave status \G "|grep Slave_IO_Running|cut -d ':' -f 2|sed -e 's/ //g'`
SQLSTATUS=`$MySql_CMD -u$User -p$passwd -S /app/mysqldata/3306/mysql.sock -e "show slave status \G "|grep Slave_SQL_Running:|cut -d ':' -f 2|sed -e 's/ //g'`
echo $IOSTATUS
echo $SQLSTATUS
if [ "$LogLevel" == "debug" ]; then
if [ "$IOSTATUS" == ""Yes ]; then
echo "curl -X POST -d "[{\"metric\": \"$Slave_IO_Metric\", \"endpoint\": \"$Endpoint\", \"timestamp\": $ts,\"step\": 60,\"value\": $Normal_Value,\"counterType\": \"GAUGE\",\"tags\": \"port=$Port\"}]" http://$Ip_Address:1988/v1/push" >> $Logfile
else
echo "curl -X POST -d "[{\"metric\": \"$Slave_IO_Metric\", \"endpoint\": \"$Endpoint\", \"timestamp\": $ts,\"step\": 60,\"value\": $NonNormal_Value,\"counterType\": \"GAUGE\",\"tags\": \"port=$Port\"}]" http://$Ip_Address:1988/v1/push" >> $Logfile
fi
if [ "$SQLSTATUS" == ""Yes ]; then
echo "curl -X POST -d "[{\"metric\": \"$Slave_SQL_Metric\", \"endpoint\": \"$Endpoint\", \"timestamp\": $ts,\"step\": 60,\"value\": $Normal_Value,\"counterType\": \"GAUGE\",\"tags\": \"port=$Port\"}]" http://$Ip_Address:1988/v1/push" >> $Logfile
else
echo "curl -X POST -d "[{\"metric\": \"$Slave_SQL_Metric\", \"endpoint\": \"$Endpoint\", \"timestamp\": $ts,\"step\": 60,\"value\": $NonNormal_Value,\"counterType\": \"GAUGE\",\"tags\": \"port=$Port\"}]" http://$Ip_Address:1988/v1/push" >> $Logfile
fi
else
if [ "$IOSTATUS" == ""Yes ]; then
curl -X POST -d "[{\"metric\": \"$Slave_IO_Metric\", \"endpoint\": \"$Endpoint\", \"timestamp\": $ts,\"step\": 60,\"value\": $Normal_Value,\"counterType\": \"GAUGE\",\"tags\": \"port=$Port\"}]" http://$Ip_Address:1988/v1/push
else
curl -X POST -d "[{\"metric\": \"$Slave_IO_Metric\", \"endpoint\": \"$Endpoint\", \"timestamp\": $ts,\"step\": 60,\"value\": $NonNormal_Value,\"counterType\": \"GAUGE\",\"tags\": \"port=$Port\"}]" http://$Ip_Address:1988/v1/push
fi
if [ "$SQLSTATUS" == ""Yes ]; then
curl -X POST -d "[{\"metric\": \"$Slave_SQL_Metric\", \"endpoint\": \"$Endpoint\", \"timestamp\": $ts,\"step\": 60,\"value\": $Normal_Value,\"counterType\": \"GAUGE\",\"tags\": \"port=$Port\"}]" http://$Ip_Address:1988/v1/push
else
curl -X POST -d "[{\"metric\": \"$Slave_SQL_Metric\", \"endpoint\": \"$Endpoint\", \"timestamp\": $ts,\"step\": 60,\"value\": $NonNormal_Value,\"counterType\": \"GAUGE\",\"tags\": \"port=$Port\"}]" http://$Ip_Address:1988/v1/push
fi
fi
crontab中
* * * * * /soft/mymon -c /soft/etc/mon.cfg
* * * * * sh /soft/etc/slavestatus.sh
添加模板，添加主机组，添加screen
模板
主机组
Screen->Graph部分
Com_delete/port=3309
Com_insert/port=3309
Com_select/port=3309
Com_update/port=3309
Innodb_row_lock_current_waits/port=3309
Innodb_row_lock_time/port=3309
Innodb_row_lock_time_avg/port=3309
Innodb_row_lock_time_max/port=3309
Innodb_row_lock_waits/port=3309
Innodb_rows_deleted/port=3309
Innodb_rows_inserted/port=3309
Innodb_rows_updated/port=3309
Queries/port=3309
Questions/port=3309
Seconds_Behind_Master/port=3309
Slave_IO_Status/port=3309
Slave_SQL_Status/port=3309
Slow_queries/port=3309
Threads_connected/port=3309
Threads_running/port=3309
cpu.idle
disk.io.read_bytes/device=sdb
disk.io.read_requests/device=sdb
disk.io.util/device=sdb
disk.io.write_bytes/device=sdb
disk.io.write_requests/device=sdb
innodb_autoinc_lock_mode/port=3309
innodb_lock_wait_timeout/port=3309
mem.memfree.percent
mem.swapfree.percent
mem.swapused.percent
Innodb_row_lock_current_waits：当前正在等待锁定的数量；
Innodb_row_lock_time ：从系统启动到现在锁定的总时间长度，单位ms；
Innodb_row_lock_time_avg ：每次等待所花平均时间；
Innodb_row_lock_time_max：从系统启动到现在等待最长的一次所花的时间；
Innodb_row_lock_waits ：从系统启动到现在总共等待的次数
监控Redis
下载相关脚本
测试运行
根据实际部署情况，修改有注释位置附近的配置
测试： python redis-monitor.py
将脚本加入crontab执行即可
* * * * * /usr/bin/python /soft/redis-monitor.py
注意redis-cli全路径，修改redis-monitor.py
注意redis.conf全路径，修改redis-monitor.py
启动redis
/usr/local/redis/bin/redis-server /etc/redis/6379.conf
监控MongoDB
Python 2.6
PyYAML > 3.10
python-requests > 0.11
pip install pyyaml
pip install requests
pip install pymongo
解压目录到/soft/mongomon
配置当前服务器的MongoDB多实例(mongod,配置节点,mongos)信息，/soft/mongomon/conf/mongomon.conf 每行记录一个实例: 端口，用户名，密码
{port: 27017, user: "root",password: "abc123"}
配置crontab, 修改mongomon/conf/mongomon_cron文件中mongomon安装path; cp mongomon_cron /etc/cron.d/
几分钟后，可从open-falcon的dashboard中查看MongoDB metric
endpoint默认是hostname
启动MongoDB
/usr/local/mongodb/bin/mongod -f /app/mongodb/27017/mongodb.config
