【RSYSLOG】rsyslog作为日志采集器安装配置说明
RSYSLOG is the rocket-fast system for log processing.
About
由于环境基于CentOS 6.7 x64，rsyslog本身就是OS的组件，由于本文使用rsyslog作为统一日志采集器，需要与kafka进行数据交付，而原始支持kafka组件是在v8.7.0以后版本才支持，而系统自带的rsyslog是版本是v5的，因此需要对rsyslog版本进行升级。
有网的方法比较简单，通过yum命令即可，官网参考官网 http://www.rsyslog.com/rhelcentos-rpms/ 即可安装指定版本。
安装说明
由于鉴于内网环境无Internet无法使用yum进行安装，因此选择离线rpm方式升级rsyslog（rpm包下载地址：http://rpms.adiscon.com/ ）。
此处有坑：按习惯，一般使用官网提供的最新的稳定版/v8-stable/epel-7，然而由于CentOS 6.7 glibc版本为2.12而epel-7依赖的libgt-0.3.11-1.el7.x86_64、libfastjson-0.99.2-1.el7.x86_64、libestr-0.1.10-1.el7.x86_64几个组件需要 libc.so.6: version `GLIBC_2.14，CentOS 6.7升级后会提示bash版本有问题，因此此处选用/v8-stable/epel-6版本（ps：此坑是个人问题，centos6.x本来就应该使用epel-6，因此此坑可忽略）
需要下载的rpm
可在http://rpms.adiscon.com/v8-stable/epel-6/x86_64/RPMS/ 获取
- json-c-0.11-4.el6.x86_64.rpm
- libfastjson-0.99.2-1.el6.x86_64.rpm
- libgt-0.3.11-1.el6.x86_64.rpm
- libestr-0.1.10-1.el6.x86_64.rpm
- liblogging-1.0.5-1.el6.x86_64.rpm
- librdkafka1-0.8.5-0.x86_64.rpm
- rsyslog-kafka-8.19.0-1.el6.x86_64.rpm
- rsyslog-8.19.0-1.el6.x86_64.rpm
rpm安装
执行安装前 先给没个文件加上运行属性 chmod +x
rpm -ivh json-c-0.11-4.el6.x86_64.rpm
rpm -ivh libfastjson-0.99.2-1.el6.x86_64.rpm
rpm -ivh libgt-0.3.11-1.el6.x86_64.rpm
rpm -ivh libestr-0.1.10-1.el6.x86_64.rpm
rpm -ivh liblogging-1.0.5-1.el6.x86_64.rpm
rpm -ivh librdkafka1-0.8.5-0.x86_64.rpm
#这边使用 -U 参数是为了直接升级本地rsyslog
rpm -Uvh rsyslog-8.19.0-1.el6.x86_64.rpm
#rsyslog-kafka组件安装依赖 rsyslog-8.19.0及librdkafka，该命令最后执行
rpm -ivh rsyslog-kafka-8.19.0-1.el6.x86_64.rpm
验证安装
- 安装完后执行
rsyslogd -v, 验证rsyslog是否升级成功。
# rsyslogd -v
rsyslogd 8.19.0, compiled with:
PLATFORM: x86_64-redhat-linux-gnu
PLATFORM (lsb_release -d):
FEATURE_REGEXP: Yes
GSSAPI Kerberos 5 support: No
FEATURE_DEBUG (debug build, slow code): No
32bit Atomic operations supported: Yes
64bit Atomic operations supported: Yes
memory allocator: system default
Runtime Instrumentation (slow code): No
uuid support: Yes
Number of Bits in RainerScript integers: 64
See http://www.rsyslog.com for more information.
- 查看/lib64/rsyslog/（32位系统为 /lib/rsyslog） 中 是否存在 omkafka.so，验证rsyslog-kafka是否安装成功。
配置说明
rsyslog官网是个大坑，能路过尽量路过。
接下来说明下配置，首先声明下本文使用rsyslog的目的是作为应用的log collector，应用使用log4x(log4net:log4net.Appender.RemoteSyslogAppender, log4j:org.apache.log4j.net.SyslogAppender)，通过UDP协议将日志传输到rsyslog。
rsyslo的默认的配置文件 /etc/rsyslog.config
# rsyslog configuration file
# note that most of this config file uses old-style format,
# because it is well-known AND quite suitable for simple cases
# like we have with the default config. For more advanced
# things, RainerScript configuration is suggested.
# For more information see /usr/share/doc/rsyslog-*/rsyslog_conf.html
# If you experience problems, see http://www.rsyslog.com/doc/troubleshoot.html
#### MODULES ####
module(load="imuxsock") # provides support for local system logging (e.g. via logger command)
module(load="imklog") # provides kernel logging support (previously done by rklogd)
#module(load"immark") # provides --MARK-- message capability
# Provides UDP syslog reception
# for parameters see http://www.rsyslog.com/doc/imudp.html
#module(load="imudp") # needs to be done just once
#input(type="imudp" port="514")
# Provides TCP syslog reception
# for parameters see http://www.rsyslog.com/doc/imtcp.html
#module(load="imtcp") # needs to be done just once
#input(type="imtcp" port="514")
#### GLOBAL DIRECTIVES ####
# Use default timestamp format
$ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat
# File syncing capability is disabled by default. This feature is usually not required,
# not useful and an extreme performance hit
#$ActionFileEnableSync on
# Include all config files in /etc/rsyslog.d/
$IncludeConfig /etc/rsyslog.d/*.conf
#### RULES ####
# Log all kernel messages to the console.
# Logging much else clutters up the screen.
#kern.* /dev/console
# Log anything (except mail) of level info or higher.
# Don't log private authentication messages!
*.info;mail.none;authpriv.none;cron.none /var/log/messages
# The authpriv file has restricted access.
authpriv.* /var/log/secure
# Log all the mail messages in one place.
mail.* /var/log/maillog
# Log cron stuff
cron.* /var/log/cron
# Everybody gets emergency messages
*.emerg :omusrmsg:*
# Save news errors of level crit and higher in a special file.
uucp,news.crit /var/log/spooler
# Save boot messages also to boot.log
local7.* /var/log/boot.log
# ### begin forwarding rule ###
# The statement between the begin ... end define a SINGLE forwarding
# rule. They belong together, do NOT split them. If you create multiple
# forwarding rules, duplicate the whole block!
# Remote Logging (we use TCP for reliable delivery)
#
# An on-disk queue is created for this action. If the remote host is
# down, messages are spooled to disk and sent when it is up again.
#$WorkDirectory /var/lib/rsyslog # where to place spool files
#$ActionQueueFileName fwdRule1 # unique name prefix for spool files
#$ActionQueueMaxDiskSpace 1g # 1gb space limit (use as much as possible)
#$ActionQueueSaveOnShutdown on # save messages to disk on shutdown
#$ActionQueueType LinkedList # run asynchronously
#$ActionResumeRetryCount -1 # infinite retries if host is down
# remote host is: name/ip:port, e.g. 192.168.0.1:514, port optional
#*.* @@remote-host:514
# ### end of the forwarding rule ###
接下来说明配置修改：
配置 input
- 开启接收UDP消息
module(load="imudp") # 加载imudp模块
input(type="imudp" port="514") # 配置UDP监听的端口号
配置 output
- 开启数据转发kafka
module(load="omkafka") # 配置加载omkafka模块
# 转发到kafka消息格式
template(name="rsyslogToKafka" type="list" option.json="on"){
constant(value="{")
#constant(value="\"timestamp\":\"") property(name="timereported" dateFormat="rfc3339")
constant(value="\"message\":\"") property(name="msg" format="json")
constant(value="\",\"host\":\"") property(name="hostname")
constant(value="\",\"severity\":\"") property(name="syslogseverity")
constant(value="\",\"severity-text\":\"") property(name="syslogseverity-text")
constant(value="\",\"facility\":\"") property(name="syslogfacility-text")
constant(value="\",\"syslogtag\":\"") property(name="syslogtag")
constant(value="\",\"type\":\"application\"")
constant(value=",\"recieve-from\":\"") property(name="fromhost-ip")
constant(value="\",\"recieve-at\":\"") property(name="timegenerated" dateFormat="rfc3339")
constant(value="\",\"report-at\":\"") property(name="timereported" dateFormat="rfc3339")
constant(value="\"}")
}
# 配置kafka相关信息及模板
action( type="omkafka"
topic="test"
broker="11.4.74.26:9092"
template="rsyslogToKafka"
)
- 另外一个坑:关于EscapeControlCharacters转换的问题
rsyslog接收的时候会对一些回车换行等敏感字符进行转换 比如 \r\n 会被转换为#015#012，而通过log4x提交的日志如果包含Exception在kibana显示就不能正常的换行，因此关闭了rsyslog中敏感字的转换
$EscapeControlCharactersOnReceive off
