Elastalert安装及使用

2018.02.08 10:21 字数 1497 阅读 2982评论 1喜欢 3

简介

Elastalert是用python2写的一个报警框架(目前支持python2.6和2.7，不支持3.x)，github地址为 https://github.com/Yelp/elastalert。他提供不同场景的规则配置，若觉得规则、告警不满足需求时，可以用python编写插件Adding a New Rule Type、Adding a New Alerter。
Elastalert内置的告警方式：

Email
JIRA
OpsGenie
Commands
HipChat
MS Teams
Slack
Telegram
AWS SNS
VictorOps
PagerDuty
Exotel
Twilio
Gitter

前置条件

Elasticsearch
Python 2.7
pip

pip 是 Python 包管理工具，该工具提供了对Python 包的查找、下载、安装、卸载的功能。
目前如果你在 python.org 下载最新版本的安装包，则是已经自带了该工具。
Python 2.7.9 + 或 Python 3.4+ 以上版本都自带 pip 工具。
pip 官网：https://pypi.org/project/pip/
你可以通过以下命令来判断是否已安装：
pip --version
如果你还未安装，则可以使用以下方法来安装：
$ curl https://bootstrap.pypa.io/get-pip.py -o get-pip.py   # 下载安装脚本
$ sudo python get-pip.py    # 运行安装脚本
注意：用哪个版本的 Python 运行安装脚本，pip 就被关联到哪个版本，如果是 Python3 则执行以下命令：
$ sudo python3 get-pip.py    # 运行安装脚本。
一般情况 pip 对应的是 Python 2.7，pip3 对应的是 Python 3.x。
部分 Linux 发行版可直接用包管理器安装 pip，如 Debian 和 Ubuntu：
sudo apt-get install python-pip

sudo easy_install -U setuptools
sudo yum install python-devel
python-pip python-dev libffi-dev libssl-dev

安装

直接使用pip安装：

$ pip install elastalert  直接装的官网发布的，不好用，还是下载源码安装

或者克隆源码安装：

$ git clone https://github.com/Yelp/elastalert.git

安装模块：

$ pip install "setuptools>=11.3" or sudo ....
$ python setup.py install

根据不同的Elasticsearch版本安装elasticsearch-py。
Elasticsearch 5.0+:

$ pip install "elasticsearch>=5.0.0"

Elasticsearch 2.X:

$ pip install "elasticsearch<3.0.0"

补充：
若出现如下问题：

error: six 1.9.0 is installed but six>=1.10.0 is required by set(['jira'])

解决办法：

$ pip install --upgrade six

更新的位置不是 python lib找的位置：/usr/lib/python2.7/site-packages/pkg_resources            /root/.local/lib/python2.7/site-packages
1，
sudo pip install --upgrade six
DEPRECATION: Python 2.7 will reach the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 won't be maintained after that date. A future version of pip will drop support for Python 2.7.
Requirement already up-to-date: six in /root/.local/lib/python2.7/site-packages (1.12.0)

2，
elastalert-create-index
Traceback (most recent call last):
  File "/usr/bin/elastalert-create-index", line 6, in <module>
    from pkg_resources import load_entry_point
  File "/usr/lib/python2.7/site-packages/pkg_resources/__init__.py", line 3241, in <module>
    @_call_aside
  File "/usr/lib/python2.7/site-packages/pkg_resources/__init__.py", line 3225, in _call_aside
    f(*args, **kwargs)
  File "/usr/lib/python2.7/site-packages/pkg_resources/__init__.py", line 3254, in _initialize_master_working_set
    working_set = WorkingSet._build_master()
  File "/usr/lib/python2.7/site-packages/pkg_resources/__init__.py", line 585, in _build_master
    return cls._build_from_requirements(__requires__)
  File "/usr/lib/python2.7/site-packages/pkg_resources/__init__.py", line 598, in _build_from_requirements
    dists = ws.resolve(reqs, Environment())
  File "/usr/lib/python2.7/site-packages/pkg_resources/__init__.py", line 791, in resolve
    raise VersionConflict(dist, req).with_context(dependent_req)

解决方法：

pip uninstall  six

pip install --upgrade six

卸载后重新安装就行。

然后继续安装即可。

如果在windows 64平台报错：执行

pip install python-magic-bin==0.4.14
修复https://stackoverflow.com/questions/18374103/exception-valuefailed-to-find-libmagic-check-your-installation-in-windows-7

E:\elastalert\example_rules> python -m elastalert.elastalert --verbose --rule config.yaml
Traceback (most recent call last):
  File "C:\Python27\lib\runpy.py", line 162, in _run_module_as_main
    "__main__", fname, loader, pkg_name)
  File "C:\Python27\lib\runpy.py", line 72, in _run_code
    exec code in run_globals
  File "C:\Python27\lib\site-packages\elastalert\elastalert.py", line 21, in <module>
    from alerts import DebugAlerter
  File "C:\Python27\lib\site-packages\elastalert\alerts.py", line 31, in <module>
    from thehive4py.api import TheHiveApi
  File "C:\Python27\lib\site-packages\thehive4py\api.py", line 7, in <module>
    import magic
  File "C:\Python27\lib\site-packages\magic.py", line 181, in <module>
    raise ImportError('failed to find libmagic.  Check your installation')
ImportError: failed to find libmagic.  Check your installation

重命名配置文件：

$ cp config.yaml.example config.yaml

打开配置文件config.yaml：

$ vim config.yaml

设置es_host、es_host指向你的Elasticsearch，然后保存退出。

配置文件解析

此处的配置文件是指config.yaml文件。
rules_folder：ElastAlert将加载规则配置文件的地方，它将尝试加载文件夹中的每个.yaml文件。
run_every：ElastAlert查询Elasticsearch的频率。
buffer_time：是查询窗口的大小，从每个查询运行的时间向后延伸。对于其中use_count_query或use_terms_query设置为true的规则，此值将被忽略。
es_host：是Elasticsearch集群的地址，ElastAlert将存储有关其状态、查询运行、警报和错误的数据。每个规则也可以设置不同的elasticsearch主机进行查询。
es_port：Elasticsearch对应的端口。
use_ssl：（可选的）是否使用TLS;连接到es_host；设置为True或False。
verify_certs：（可选的）是否验证TLS证书; 设置为True或False，默认是True。
client_cert：（可选的）PEM证书的路径。
client_key：（可选的）作为客户端密钥使用的私钥文件的路径。
ca_certs：（可选的）用于验证SSL连接的CA证书的路径。
es_username：（可选的）用于连接Elasticsearch的basic-auth用户名。
es_password：（可选的）用于连接Elasticsearch的密码。
es_url_prefix：（可选的） Elasticsearch端点的URL前缀。
es_send_get_body_as：（可选的）查询Elasticsearch方法- GET，POST或source，默认是GET。
writeback_index：是ElastAlert将存储数据的索引名称。
alert_time_limit：是失败警报的重试窗口。

创建Elasticsearch索引

elastalert-create-index这个命令会在elasticsearch创建索引，便于ElastAlert将有关其查询及其警报的信息和元数据保存回Elasticsearch。这不是必须的步骤，但是强烈建议创建。因为对于审计，测试很有用，并且重启elastalert不影响计数和发送alert。默认情况下，创建的索引叫 elastalert_status。

$ elastalert-create-index
New index name (Default elastalert_status)
Name of existing index to copy (Default None)
New index elastalert_status created
Done!

创建成功之后，我们可以在Elasticsearch中看到所创建的索引elastalert_status：

$ curl 'localhost:9200/_cat/indices?v'
health status index             uuid                   pri rep docs.count docs.deleted store.size pri.store.size
yellow open   .kibana           _4Op-xRCSzyC5nI2wZSPvw   1   1          2            0      8.4kb          8.4kb
yellow open   elastalert_status ZTetmp8mTS6F28aIBJmJOg   5   1          0            0       810b           810b

elastalert_status
ElastAlert 根据elastalert_status去确定首次启动的时候在什么时间范围内去查询，以避免重复查询。对于每个规则，它将从最近的结束时间开始查询。包括：

@timestamp：文件上传到Elasticsearch的时间。这是在运行查询并且已经处理结果之后。
rule_name：相应规则的名称。
starttime：查询的开始时间戳。
endtime：查询结束时间戳。
hits：查询结果的数量。
matches：处理命中后规则返回的匹配数。请注意，这并不一定意味着警报被触发。
time_taken：此查询运行所需的秒数。

测试Demo

本次elastalert安装目录为：/opt/elastalert。

配置SMTP

在/opt/elastalert目录下创建文件smtp_auth_file.yaml，内容如下：

#邮箱用户名
user: example@email.com
#不是邮箱密码，是设置的SMTP密码
password: XXXXX

配置规则

在example_rules目录下修改配置文件，如下：

$ cd /opt/elastalert/example_rules
$ vim example_frequency.yaml

修改后的配置文件：

# Alert when the rate of events exceeds a threshold

# (Optional)
# Elasticsearch host
# es_host: elasticsearch.example.com

# (Optional)
# Elasticsearch port
# es_port: 9200

# (OptionaL) Connect with SSL to Elasticsearch
#use_ssl: True

# (Optional) basic-auth username and password for Elasticsearch
#es_username: someusername
#es_password: somepassword

# (Required)
# Rule name, must be unique
 name: Example rule

# (Required)
# Type of alert.
# the frequency rule type alerts when num_events events occur with timeframe time
 type: frequency

# (Required)
# Index to search, wildcard supported
 index: logstash-2017.08.28

# (Required, frequency specific)
# Alert when this many documents matching the query occur within a timeframe
 num_events: 1

# (Required, frequency specific)
# num_events must occur within this amount of time to trigger an alert
 timeframe:
   minutes: 1

# (Required)
# A list of Elasticsearch filters used for find events
# These filters are joined with AND and nested in a filtered query
# For more info: http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/query-dsl.html

#匹配日志中field字段值为value的日志数据，若不想进行过滤可以写成“filter: []”（这种方式可以用在日志数量的告警中）
 filter:
 - query:
     query_string:
       query: "field: value"

#SMTP configration
 smtp_host: smtp.163.com
 smtp_port: 25

#SMTP auth
 smtp_auth_file: /opt/elastalert/smtp_auth_file.yaml
 email_reply_to: example@email.com
 from_addr: example@email.com

# (Required)
# The alert is use when a match is found
 alert:
 - "email"

# (required, email specific)
# a list of email addresses to send alerts to
 email:
 - "test@email.com"

保存修改并退出。
补充：
上述规则表示：在elastalert执行的一分钟内，出现一条有field值为value日志，则触发告警，并且告警通知将以email的形式从example@email.com邮箱发送给test@email.com。
规则配置解析：

es_host、es_port：应该指向我们要查询的Elasticsearch集群。
name：是这个规则的唯一名称。如果两个规则共享相同的名称，ElastAlert将不会启动。
type：每个规则都有不同的类型，可能会采用不同的参数。该frequency类型表示“在timeframe时间内匹配成功次数超过num_events发出警报”。有关其他类型的信息，请参阅规则类型。
index：要查询的索引的名称。
num_events：此参数特定于frequency类型，是触发警报时的阈值。
timeframe：timeframe是num_events必须发生的时间段。
filter：是用于过滤结果的Elasticsearch过滤器列表。有关
详细信息，请参阅编写过滤规则。
alert：警报。有关警报类型的更多信息，请参阅警报。电子邮件警报需要SMTP服务器才能发送邮件。默认情况下，它将尝试使用localhost。这可以通过smtp_host选项更改。
email：是要发送警报的地址列表。

还有许多其他可选配置选项，请参阅常见配置选项。

myrule.yaml和config.yaml

# From example_rules/example_frequency.yaml
es_host: xxx.xxx
es_port: 9200
name: Example rule
type: frequency
index: myindex
timestamp_field: mytime
num_events: 1
timeframe:
  minutes: 50
filter:
- term:
    "carColor" : "blue"
alert:
- "email"
email:
- "elastalert@example.com"

# This is the folder that contains the rule yaml files
# Any .yaml file will be loaded as a rule
rules_folder: example_rules

# How often ElastAlert will query Elasticsearch
# The unit can be anything from weeks to seconds
run_every:
  minutes: 1

# ElastAlert will buffer results from the most recent
# period of time, in case some log sources are not in real time
buffer_time:
  minutes: 15

# The Elasticsearch hostname for metadata writeback
# Note that every rule can have its own Elasticsearch host
es_host: 211.100.75.204

# The Elasticsearch port
es_port: 9200

# The AWS region to use. Set this when using AWS-managed elasticsearch
#aws_region: us-east-1

# The AWS profile to use. Use this if you are using an aws-cli profile.
# See http://docs.aws.amazon.com/cli/latest/userguide/cli-chap-getting-started.html
# for details
#profile: test

# Optional URL prefix for Elasticsearch
#es_url_prefix: elasticsearch

# Connect with TLS to Elasticsearch
#use_ssl: True

# Verify TLS certificates
#verify_certs: True

# GET request with body is the default option for Elasticsearch.
# If it fails for some reason, you can pass 'GET', 'POST' or 'source'.
# See http://elasticsearch-py.readthedocs.io/en/master/connection.html?highlight=send_get_body_as#transport
# for details
#es_send_get_body_as: GET

# Option basic-auth username and password for Elasticsearch
#es_username: someusername
#es_password: somepassword

# Use SSL authentication with client certificates client_cert must be
# a pem file containing both cert and key for client
#verify_certs: True
#ca_certs: /path/to/cacert.pem
#client_cert: /path/to/client_cert.pem
#client_key: /path/to/client_key.key

# The index on es_host which is used for metadata storage
# This can be a unmapped index, but it is recommended that you run
# elastalert-create-index to set a mapping
writeback_index: elastalert_status

# If an alert fails for some reason, ElastAlert will retry
# sending the alert until this time period has elapsed
alert_time_limit:
  days: 2

更全的示例：

# Alert when the rate of events exceeds a threshold

# (Optional)
# Elasticsearch host
es_host: 110.10.21.77

# (Optional)
# Elasticsearch port
es_port: 9200

# (OptionaL) Connect with SSL to Elasticsearch
#use_ssl: True

# (Optional) basic-auth username and password for Elasticsearch
#es_username: someusername
#es_password: somepassword

# (Required)
# Rule name, must be unique
name:  system error 

# (Required)
# Type of alert.
# the frequency rule type alerts when num_events events occur with timeframe time
type: frequency

# (Required)
# Index to search, wildcard supported
index:  task*

attach_related: true
# (Required, frequency specific)
# Alert when this many documents matching the query occur within a timeframe
num_events: 1

# (Required, frequency specific)
# num_events must occur within this amount of time to trigger an alert
timeframe:
  hours: 1
timestamp_field: "@timestamp"
# (Required)
# A list of Elasticsearch filters used for find events
# These filters are joined with AND and nested in a filtered query
# For more info: http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/query-dsl.html
filter:
- terms:
    "Level": ["fatal", "error"]

#use_kibana_dashboard: afAlertDashboard
use_kibana4_dashboard: https://nodejsgbl.italkbb.com/kibana/app/kibana#/dashboard/3bde48d0-9880-11e9-b5d5-2df46b09dea6
# (Required)
# The alert is use when a match is found
alert:

- "email"

# (required, email specific)
# a list of email addresses to send alerts to
email:
- "xxxx@net263.com"
- "xxxx@163.com"

smtp_host: smtp.263.net
smtp_port: 25
smtp_auth_file: ../smtp_auth_file.yaml
email_reply_to: No.reply@net263.com
from_addr: ElastAlert@net263.com
#cc: aaa@net263.com
email_format: html
#alert_text_type: exclude_fields #alert_text_only
alert_subject: "Alert: System {0} occurred {1} times."
alert_subject_args:
- Level #来自index里面json内容
- "num_hits"
alert_text: "<pre> You can access all of error log on Kibana: <a href={0}> Aijia server alert dashboard</a><br>"
alert_text_args:
- kibana_link

{"indexId":"1000001","mytime": "2019-01-14T17:40:17+08:00","carColor":"blue","@timestamp":"2018-12-18T07:48:17","cost":1001,"indexName":"default_log_index" }

{ "indexName" : "af-test",     "code" : "1",          "msg" : "testMsg",          "position" : "AHC.User.ReadService",          "eventType" : "alarm","eventTimestamp" : "2019-01-14T17:40:17+08:00",          "@timestamp" : "2019-01-14T17:44:24.6172215+08:00"}

key要区分大小写，value 写过滤条件时候，全部小写，否则查询不到。如 term: "msg":"testmsg" 才行

测试规则

命令如下：

$ elastalert-test-rule example_rules/example_frequency.yaml

能够检查规则是否正确，有关详细信息测试部分。

运行

运行elastalert：

$ cd /opt/elastalert
$ python -m elastalert.elastalert --verbose --rule example_frequency.yaml

或者运行指定目录下所有yaml文件：
 elastalert --verbose --config ./config.yaml  --start 2019-06-27T06:00:03
其中config.yaml里面存放着rules的目录。仔细看好目录名称不要写错。

调试elasticsearch 的 查询 语句是否有问题，可以加上参数 es_debug、es_debug_trace trace后面接文件名，里面会输出查询语句。

trace文件打印出查询语句：

curl -XGET 'http://localhost:9200/?pretty' -d ''
  "query": {
    "bool": {
      "filter": {
        "bool": {
          "must": [
                  "gt": "2019-07-03T09:30:38.664486Z",
                  "lte": "2019-07-03T09:32:03.508890Z"
                }
              }
            },
            {
              "query_string": {
                "query": "NOT AggregatedHealthState.keyword: Ok"
              }
            }
          ]
        }
      }
    }
  },
  "sort": [
    {
      "@timestamp": {
        "order": "asc"
      }
    }
  ]
}


给Elasticsearch添加测试数据：

$ curl -X POST "http://127.0.0.1:9200/logstash-2017.08.28/test"  -d '{
"@timestamp": "2017-08-28T10:54:41.000Z",
"field": "value"
}'

注意：上述的@timestamp是UTC时间，也就是说这条数据是在2017-08-28 18:54:41产生的。

Alert

可在邮箱中看到如下图所示的邮件：

收到邮件中会包含当前符合告警规则的日志数据。

参考资料

ElastAlert - Easy & Flexible Alerting With Elasticsearch
Rule Types and Configuration Options
Writing Filters For Rules
https://www.cnblogs.com/zhaishaomin/p/7417306.html

https://blog.csdn.net/pujiaolin/article/details/52252950?locationNum=3

https://blog.csdn.net/gamer_gyt/article/details/52917116

https://blog.csdn.net/davinciyxw/article/details/79104498

https://blog.csdn.net/vbaspdelphi/article/details/54311764

http://www.freebuf.com/sectool/164591.html

alert还有kibana插件,以及elasticalert还可以容器方式启动

参考:
https://github.com/bitsensor/elastalert-kibana-plugin
插件下载慢:

https://git.bitsensor.io/front-end/elastalert-kibana-plugin/-/jobs/10874/artifacts/raw/artifact/elastalert-5.6.4-latest.zip
然后
./bin/kibana-plugin install file:///usr/local/src/elastalert-5.6.4-latest.zip

安装后效果:

微信报警：https://github.com/anjia0532/elastalert-wechat-plugin
钉钉报警：https://github.com/xuyaoqiang/elastalert-dingtalk-plugin

告警规则
告警模板
告警时间
告警方式
-----
参考:https://xizhibei.github.io/2017/11/19/alerting-with-elastalert/

https://github.com/chenryn/ELKstack-guide-cn/blob/master/elasticsearch/other/elastalert.md

安装elastalert（按照官网的操作走）

先切换到自己的工作目录

我这里是 /usr/local/dev/

git 克隆源码

git clone https://github.com/Yelp/elastalert.git

cd elastalert

安装

sudo python setup.py install

出错了

Running blist-1.3.6/setup.py -q bdist_egg –dist-dir /tmp/easy_install-Gc6gbe/blist-1.3.6/egg-dist-tmp-Ik7LL2

The required version of setuptools (>=1.1.6) is not available,

and can’t be installed while this script is running. Please

install a more recent version first, using

‘easy_install -U setuptools’.

(Currently using setuptools 0.9.8 (/usr/lib/python2.7/site-packages))

ok按照它提示的，，，执行：

sudo easy_install -U setuptools

再执行安装（sudo python setup.py install），又出错了

warning: no files found matching 'blist.rst'

blist/_blist.c:38:20: 致命错误：Python.h：没有那个文件或目录

#include<Python.h>

这其实是缺少开发包，执行命令：

sudo yum install python-devel

再执行安装（sudo python setup.py install）

到这里基本能安装成功。

有时候会报一些包找不到的错误，可能是网络问题，重复执行安装命令即可（我在阿里云上遇到了）。

继续执行安装命令

pip install -r requirements.txt

找不到pip ，先安装pip并更新

sudo yum -y install python-pip

sudo pip install --upgrade pip

安装完继续

sudo pip install -r requirements.txt

在elasticsearch中创建elastalert的日志索引

sudo elastalert-create-index

根据自己的情况，填入elasticsearch的相关信息，关于

elastalert_status部分直接回车默认的即可。

如下所示：

Enter elasticsearch host: log.example.com

Enter elasticsearch port: 9200

Use SSL? t/f: t

Enter optional basic-auth username (or leave blank): es_admin

Enter optional basic-auth password (or leave blank):

Enter optional Elasticsearch URL prefix (prepends a string to the URL of every request):

New index name? (Default elastalert_status)

Name of existing index to copy? (Default None)

/usr/lib/python2.7/site-packages/requests/packages/urllib3/connectionpool.py:734: InsecureRequestWarning: Unverified HTTPS request is being made. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.org/en/latest/security.html

InsecureRequestWarning)

New index elastalert_status created

Done!

3.创建配置文件

sudo cp config.yaml.example config.yaml

sudo vi config.yaml

根据自己的具体情况进行修改

比如我这里的elasticsearch是ssl 并需要用户登陆的，配置大致如下：

run_every:

minutes: 1

buffer_time:

minutes: 15

es_host: log.example.com

es_port: 9200

use_ssl: True

es_send_get_body_as: GET

es_username: es_admin

es_password: es_password

writeback_index: elastalert_status

alert_time_limit:

days: 2

以同样的方式配置规则

cd example_rules/

sudo cp example_frequency.yaml my_rule.yaml

sudo vi my_rule.yaml

这里就是根据具体的elasticsearch的信息进行配置

es_host: log.example.com

es_port: 9200

use_ssl: True

es_username: es_admin

es_password: es_password

#name属性要求唯一，这里最好能标示自己的产品

name: My-Product Exception Alert

#类型，我选择任何匹配的条件都发送邮件警告

type: any

#需要监控的索引，支持通配

index: logstash-*

#下面两个随意配置

num_events: 50

timeframe:

hours: 4

#根据条件进行过滤查询（这里我只要出现异常的日志，并且排除业务异常（自定义异常））

filter:

- query:

query_string:

query: "message: *exception* AND message: (!*BusinessException*) AND message: (!*ServiceException*)"

#email的警告方式

alert:

- "email"

#增加邮件内容，这里我附加一个日志访问路径

alert_text: "Ref Log https://log.example.com:5601/app/kibana"

#SMTP协议的邮件服务器相关配置（我这里是腾讯企业邮箱）

smtp_host: smtp.exmail.qq.com

smtp_port: 25

#用户认证文件，需要user和password两个属性

smtp_auth_file: smtp_auth_file.yaml

email_reply_to: no-reply@example.com

from_addr: no-reply@example.com

#需要接受邮件的邮箱地址列表

email:

- "user1@example.com"

接下来创建smtp_auth_file.yaml

sudo touch smtp_auth_file.yaml

sudo vi smtp_auth_file.yaml

##配置文件内容

user: "no-reply@example.com"

password: "password"

现在可以简单测试一下配置文件是否正确

sudo elastalert-test-rule ./my_rule.yaml

如果有问题，就检查配置文件。

然后启动运行一下试试：

sudo python -m elastalert.elastalert --verbose --rule my_rule.yaml

然后故意产生一下异常试试，比如

if(true){

throw new NullPointerException("测试日志异常监控服务");

}

执行一下，一般没啥问题，邮件应该就能收到了，有问题就检查配置文件。

4.我是要监控服务，所以要让elastalert以服务的形式运行

首先创建elastalert的配置目录和相关文件

sudo mkdir /etc/elastalert

cd /etc/elastalert

-- 复制配置文件

sudo cp /usr/local/dev/elastalert/config.yaml config.yaml

sudo mkdir rules

cd rules

-- 复制规则文件

sudo cp /usr/local/dev/elastalert/example_rules/my_rule.yaml my_rule.yaml

-- 复制邮件用户认证文件

sudo cp /usr/local/dev/elastalert/example_rules/smtp_auth_file.yaml smtp_auth_file.yaml

接下来修改配置文件

修改 config.yaml 中

rules_folder: /etc/elastalert/rules

修改 my_rule.yaml中

smtp_auth_file: /etc/elastalert/rules/umu_smtp_auth_file.yaml

接下来就是创建systemd服务了

cd /etc/systemd/system

sudo touch elastalert.service

sudo vi elastalert.service

elastalert.service 内容

[Unit]

Description=elastalert

After=elasticsearch.service

[Service]

Type=simple

User=root

Group=root

Restart=on-failure

WorkingDirectory=/usr/local/dev/elastalert

ExecStart=/usr/bin/elastalert --config /etc/elastalert/config.yaml --rule /etc/elastalert/rules/my_rule.yaml

[Install]

WantedBy=multi-user.target

保存退出。

启动服务

sudo systemctl start elastalert

查看状态

sudo systemctl status elastalert

不出意外的话，你将看到绿色的Active: active (running)

至此，日志的异常监控服务就搭建好了，好的，收工！

---------------------

作者：梦得溪

来源：CSDN

原文：https://blog.csdn.net/pujiaolin/article/details/52252950

安装elastalert（按照官网的操作走）
先切换到自己的工作目录我这里是 /usr/local/dev/ git 克隆源码
git clone https://github.com/Yelp/elastalert.git
cd elastalert123安装
sudo python setup.py install1出错了 Running blist-1.3.6/setup.py -q bdist_egg –dist-dir /tmp/easy_install-Gc6gbe/blist-1.3.6/egg-dist-tmp-Ik7LL2 The required version of setuptools (>=1.1.6) is not available, and can’t be installed while this script is running. Please install a more recent version first, using ‘easy_install -U setuptools’.
(Currently using setuptools 0.9.8 (/usr/lib/python2.7/site-packages))
ok按照它提示的，，，执行：
sudo easy_install -U setuptools1再执行安装（sudo python setup.py install），又出错了
warning: no files found matching 'blist.rst'blist/_blist.c:38:20: 致命错误：Python.h：没有那个文件或目录 #include<Python.h>123这其实是缺少开发包，执行命令：
sudo yum install python-devel1再执行安装（sudo python setup.py install）到这里基本能安装成功。有时候会报一些包找不到的错误，可能是网络问题，重复执行安装命令即可（我在阿里云上遇到了）。
继续执行安装命令
pip install -r requirements.txt1找不到pip ，先安装pip并更新
sudo yum -y install python-pipsudo pip install --upgrade pip12安装完继续
sudo pip install -r requirements.txt1在elasticsearch中创建elastalert的日志索引
sudo elastalert-create-index1根据自己的情况，填入elasticsearch的相关信息，关于 elastalert_status部分直接回车默认的即可。如下所示：
Enter elasticsearch host: log.example.comEnter elasticsearch port: 9200Use SSL? t/f: tEnter optional basic-auth username (or leave blank): es_adminEnter optional basic-auth password (or leave blank):Enter optional Elasticsearch URL prefix (prepends a string to the URL of every request):New index name? (Default elastalert_status)Name of existing index to copy? (Default None)/usr/lib/python2.7/site-packages/requests/packages/urllib3/connectionpool.py:734: InsecureRequestWarning: Unverified HTTPS request is being made. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.org/en/latest/security.html InsecureRequestWarning)/usr/lib/python2.7/site-packages/requests/packages/urllib3/connectionpool.py:734: InsecureRequestWarning: Unverified HTTPS request is being made. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.org/en/latest/security.html InsecureRequestWarning)/usr/lib/python2.7/site-packages/requests/packages/urllib3/connectionpool.py:734: InsecureRequestWarning: Unverified HTTPS request is being made. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.org/en/latest/security.html InsecureRequestWarning)/usr/lib/python2.7/site-packages/requests/packages/urllib3/connectionpool.py:734: InsecureRequestWarning: Unverified HTTPS request is being made. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.org/en/latest/security.html InsecureRequestWarning)/usr/lib/python2.7/site-packages/requests/packages/urllib3/connectionpool.py:734: InsecureRequestWarning: Unverified HTTPS request is being made. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.org/en/latest/security.html InsecureRequestWarning)/usr/lib/python2.7/site-packages/requests/packages/urllib3/connectionpool.py:734: InsecureRequestWarning: Unverified HTTPS request is being made. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.org/en/latest/security.html InsecureRequestWarning)/usr/lib/python2.7/site-packages/requests/packages/urllib3/connectionpool.py:734: InsecureRequestWarning: Unverified HTTPS request is being made. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.org/en/latest/security.html InsecureRequestWarning)New index elastalert_status createdDone!1234567891011121314151617181920212223243.创建配置文件
sudo cp config.yaml.example config.yaml
sudo vi config.yaml123根据自己的具体情况进行修改
比如我这里的elasticsearch是ssl 并需要用户登陆的，配置大致如下：
run_every: minutes: 1
buffer_time: minutes: 15
es_host: log.example.com
es_port: 9200
use_ssl: True
es_send_get_body_as: GET
es_username: es_admin
es_password: es_password
writeback_index: elastalert_status
alert_time_limit: days: 212345678910111213141516171819202122以同样的方式配置规则
cd example_rules/
sudo cp example_frequency.yaml my_rule.yaml
sudo vi my_rule.yaml12345这里就是根据具体的elasticsearch的信息进行配置
es_host: log.example.comes_port: 9200use_ssl: Truees_username: es_admines_password: es_password#name属性要求唯一，这里最好能标示自己的产品name: My-Product Exception Alert#类型，我选择任何匹配的条件都发送邮件警告type: any#需要监控的索引，支持通配index: logstash-*#下面两个随意配置num_events: 50timeframe: hours: 4#根据条件进行过滤查询（这里我只要出现异常的日志，并且排除业务异常（自定义异常））filter:- query: query_string: query: "message: *exception* AND message: (!*BusinessException*) AND message: (!*ServiceException*)"#email的警告方式alert:- "email"
#增加邮件内容，这里我附加一个日志访问路径alert_text: "Ref Log https://log.example.com:5601/app/kibana"#SMTP协议的邮件服务器相关配置（我这里是腾讯企业邮箱）smtp_host: smtp.exmail.qq.comsmtp_port: 25#用户认证文件，需要user和password两个属性smtp_auth_file: smtp_auth_file.yamlemail_reply_to: no-reply@example.comfrom_addr: no-reply@example.com
#需要接受邮件的邮箱地址列表email:- "user1@example.com"- "user1@example.com"1234567891011121314151617181920212223242526272829303132333435363738接下来创建smtp_auth_file.yaml
sudo touch smtp_auth_file.yamlsudo vi smtp_auth_file.yaml
##配置文件内容user: "no-reply@example.com"password: "password"123456现在可以简单测试一下配置文件是否正确
sudo elastalert-test-rule ./my_rule.yaml1如果有问题，就检查配置文件。然后启动运行一下试试：
sudo python -m elastalert.elastalert --verbose --rule my_rule.yaml1然后故意产生一下异常试试，比如
if(true){ throw new NullPointerException("测试日志异常监控服务");}123执行一下，一般没啥问题，邮件应该就能收到了，有问题就检查配置文件。
4.我是要监控服务，所以要让elastalert以服务的形式运行首先创建elastalert的配置目录和相关文件
sudo mkdir /etc/elastalertcd /etc/elastalert-- 复制配置文件
sudo cp /usr/local/dev/elastalert/config.yaml config.yamlsudo mkdir rulescd rules
-- 复制规则文件sudo cp /usr/local/dev/elastalert/example_rules/my_rule.yaml my_rule.yaml
-- 复制邮件用户认证文件sudo cp /usr/local/dev/elastalert/example_rules/smtp_auth_file.yaml smtp_auth_file.yaml12345678910111213接下来修改配置文件修改 config.yaml 中
rules_folder: /etc/elastalert/rules1修改 my_rule.yaml中
smtp_auth_file: /etc/elastalert/rules/umu_smtp_auth_file.yaml1接下来就是创建systemd服务了
cd /etc/systemd/systemsudo touch elastalert.service
sudo vi elastalert.service1234elastalert.service 内容
[Unit]Description=elastalertAfter=elasticsearch.service
[Service]Type=simpleUser=rootGroup=rootRestart=on-failureWorkingDirectory=/usr/local/dev/elastalertExecStart=/usr/bin/elastalert --config /etc/elastalert/config.yaml --rule /etc/elastalert/rules/my_rule.yaml
[Install]WantedBy=multi-user.target1234567891011121314保存退出。启动服务
sudo systemctl start elastalert1查看状态
sudo systemctl status elastalert1不出意外的话，你将看到绿色的Active: active (running)
至此，日志的异常监控服务就搭建好了，好的，收工！--------------------- 作者：梦得溪来源：CSDN 原文：https://blog.csdn.net/pujiaolin/article/details/52252950 版权声明：本文为博主原创文章，转载请附上博文链接！

posted @ 2018-11-30 11:21 Bigben 阅读(2021) 评论(0) 编辑收藏举报

刷新页面返回顶部