kestrel对接elasticsearch踩坑记

写在前面：kestrel当前发展很快，官方文档更新也不及时，比如这个：https://opencybersecurityalliance.org/posts/kestrel-2021-07-26/，巨坑！用最新版本的kestrel，里面的语法都出现解析错误，你说尴尬不。。。没办法，自己看源码搞吧，源码测试里仅仅有单元测试的，没有端到端的测试，只能去看底层源码揣摩使用方法。。。总之，我最终修改了elastic_ecs模块下stix_transmission/api_client.py的源码才搞定。

本文目标：kestrel对接elasticsearch，最终能够使用kestrel查询出ES的数据来。最终效果图：

kestrel查询脚本：

1 2	`browsers = GET process FROM stixshifter://host110 WHERE [process:name IN ('firefox.exe', 'chrome.exe')] START t'2021-01-01T00:00:00Z' STOP t'2021-12-31T00:00:00Z'` `DISP browsers ATTR name, pid`

输出：

       name         pid
 chrome.exe 12132112121
firefox.exe      121321
firefox.exe     9121321

ES数据源，主要写入了几条：

curl -k -uelastic:changeme -H "Content-Type: application/json" -XPUT https://YOUR_IP:9200/host110/_doc/1 -d '{
    "process": {
        "name": "firefox.exe",
        "content": "I hava a friend who loves smile, gymbo is his name",
        "pid": "121321"},
    "@timestamp": "2021-11-02T14:44:23.453+0000"
}'
curl -k -uelastic:changeme -H "Content-Type: application/json" -XPUT https://YOUR_IP:9200/host110/_doc/2 -d '{
    "process": {
        "name": "chrome.exe",
        "content": "I hava a friend who loves smile, gymbo is his name",
        "pid": "12132112121"},
    "@timestamp": "2021-11-02T14:44:23.453+0000"
}'
curl -k -uelastic:changeme -H "Content-Type: application/json" -XPUT https://YOUR_IP:9200/host110/_doc/3 -d '{
    "process": {
        "name": "twitter.exe",
        "content": "I hava a friend who loves smile, gymbo is his name",
        "pid": "1213211242123"},
    "@timestamp": "2021-11-02T14:44:23.453+0000"
}'
curl -k -uelastic:changeme -H "Content-Type: application/json" -XPUT https://YOUR_IP:9200/host110/_doc/4 -d '{
    "process": {
        "name": "firefox.exe",
        "content": "I hava a friend who loves smile, gymbo is his name",
        "pid": "9121321"},
    "@timestamp": "2021-11-02T14:44:23.453+0000"
}'

在已经安装好了kestrel的前提下，操作如下：

第一步，设置ES的api_key，可以参考：https://blog.csdn.net/UbuntuTouch/article/details/107181440，我的ES配置：

node.name: node-1
network.host: 0.0.0.0
http.port: 9200
cluster.initial_master_nodes: ["node-1"]
xpack.security.enabled: true
xpack.security.authc.api_key.enabled: true
xpack.security.http.ssl.enabled: true
xpack.security.http.ssl.keystore.path: /home/es_user/elasticsearch-7.15.1/config/elastic-stack-ca.p12
xpack.security.http.ssl.truststore.path: /home/es_user/elasticsearch-7.15.1/config/elastic-stack-ca.p12
xpack.security.transport.ssl.enabled: true
xpack.security.transport.ssl.verification_mode: certificate
xpack.security.transport.ssl.keystore.path: /home/es_user/elasticsearch-7.15.1/config/elastic-stack-ca.p12
xpack.security.transport.ssl.truststore.path: /home/es_user/elasticsearch-7.15.1/config/elastic-stack-ca.p12
http.cors.enabled: true
http.cors.allow-origin: "*"

然后就是生成api_key了。

第二步：安装stix_shifter好以后，设置connector环境变量。

1	`pip install stix-shifter-modules-elastic_ecs`

export STIXSHIFTER_HOST110_CONFIG='{"auth":{"id":"ZT3QznwBhSK3ri59dnDv", "api_key":"oY8lmKTpTOOXxNqcwJuiqA"}}'
export STIXSHIFTER_HOST110_CONNECTION='{"host":"localhost", "port":9200, "indices":"host110"}'
export STIXSHIFTER_HOST110_CONNECTOR=elastic_ecs

补充说下stix_shifter的用途，本质上是将kestrel lang对ES数据的查询语句转换为ES的查询语法。

例如，我的脚本中：

1	`GET process FROM stixshifter://host110 WHERE [process:name IN ('firefox.exe', 'chrome.exe')] START t'2021-01-01T00:00:00Z' STOP t'2021-12-31T00:00:00Z'`

这条语句会被stix_shifter转换为如下ES查询：

{'Content-Type': 'application/json'} search data==> {'_source': {'includes': ['@timestamp', 'source.*', 'destination.*', 'event.*', 'client.*', 'server.*', 'host.*', 'network.*', 'process.*', 'user.*', 'file.*', 'url.*', 'registry.*', 'dns.*']}, 'query': {'query_string': {'query':

'(process.pid : ("12132112121" OR "9121321" OR "121321") OR process.ppid : ("12132112121" OR "9121321" OR "121321") OR process.parent.pid : ("12132112121" OR "9121321" OR "121321") OR process.parent.ppid : ("12132112121" OR "9121321" OR "121321")) AND (@timestamp:["2021-01-01T00:00:00.000Z" TO "2021-12-31T00:00:00.000Z"])'

}}}

当然要通过追踪kestrel源码追踪分析才知道。

第三步：kestrel在对接ES自签名证书的时候，有bug会一致报这样错误，真是蛋疼啊，我是通过源码分析才发现的。

错误如下：

  File "/root/bone/huntingspace/lib/python3.8/site-packages/kestrel_datasource_stixshifter/interface.py", line 151, in query
    raise DataSourceError(
kestrel.exceptions.DataSourceError: [ERROR] DataSourceError: data source internal error: STIX-shifter transmission.results() failed. please test data source manually.

然后你深入源码分析才知道是这个错误：

"Wrong certificate: HTTPSConnectionPool(host='localhost', port=9200): Max retries exceeded with url: /_cluster/health?pretty (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self signed certificate (_ssl.c:1131)')))"

, 'code': 'authentication_fail'}

就是因为自签名证书的问题，解决方式如下：

修改源码：site-packages/stix_shifter_modules/elastic_ecs/stix_transmission/api_client.py

42         self.client = RestApiClient(connection.get('host'),
43                                     connection.get('port'),
44                                     headers,
45                                     url_modifier_function=url_modifier_function,
46                                     cert_verify= connection.get('selfSignedCert', True),
47                                     sni=connection.get('sni', None)
48                                     )

将46行的True修改为False就好了！

第四步，ES写入mapping和数据，一个bash脚本：

curl -k -uelastic:changeme -H "Content-Type: application/json" -XPUT https://YOUR_IP:9200/host110?pretty
 
curl -k -uelastic:changeme -H "Content-Type: application/json" -XPUT https://YOUR_IP:9200/host110?pretty -d '{"mappings": {
"properties": {
     "process": {
        "type": "object",
        "properties": {
          "@timestamp": {"type": "date"},
          "name": {"type": "text"},
          "pid": {"type": "text"},
          "content": {"type": "text"}
        }
}}}}'
 
 
curl -k -uelastic:changeme -H "Content-Type: application/json" -XPUT https://YOUR_IP:9200/host110/_doc/1 -d '{
    "process": {
        "name": "firefox.exe",
        "content": "I hava a friend who loves smile, gymbo is his name",
        "pid": "121321"},
    "@timestamp": "2021-11-02T14:44:23.453+0000"
}'
curl -k -uelastic:changeme -H "Content-Type: application/json" -XPUT https://YOUR_IP:9200/host110/_doc/2 -d '{
    "process": {
        "name": "chrome.exe",
        "content": "I hava a friend who loves smile, gymbo is his name",
        "pid": "12132112121"},
    "@timestamp": "2021-11-02T14:44:23.453+0000"
}'
curl -k -uelastic:changeme -H "Content-Type: application/json" -XPUT https://YOUR_IP:9200/host110/_doc/3 -d '{
    "process": {
        "name": "twitter.exe",
        "content": "I hava a friend who loves smile, gymbo is his name",
        "pid": "1213211242123"},
    "@timestamp": "2021-11-02T14:44:23.453+0000"
}'
curl -k -uelastic:changeme -H "Content-Type: application/json" -XPUT https://YOUR_IP:9200/host110/_doc/4 -d '{
    "process": {
        "name": "firefox.exe",
        "content": "I hava a friend who loves smile, gymbo is his name",
        "pid": "9121321"},
    "@timestamp": "2021-11-02T14:44:23.453+0000"
}'