graylog 索引模版处理
graylog 默认分词只支持对应几个固定的字段，如果需要自定义索引信息，就可以使用模版能力，默认包含了一个graylog-internal，order 为-1 但是我们可以扩展
默认索引信息
- 查询信息
GET <endpoint>/_template/graylog-internal?pretty'
效果
{
"graylog-internal": {
"order": -1,
"index_patterns": [
"graylog_*"
],
"settings": {
"index": {
"analysis": {
"analyzer": {
"analyzer_keyword": {
"filter": "lowercase",
"tokenizer": "keyword"
}
}
}
}
},
"mappings": {
"_source": {
"enabled": true
},
"dynamic_templates": [
{
"internal_fields": {
"mapping": {
"type": "keyword"
},
"match_mapping_type": "string",
"match": "gl2_*"
}
},
{
"store_generic": {
"mapping": {
"type": "keyword"
},
"match_mapping_type": "string"
}
}
],
"properties": {
"gl2_processing_timestamp": {
"format": "uuuu-MM-dd HH:mm:ss.SSS",
"type": "date"
},
"gl2_accounted_message_size": {
"type": "long"
},
"gl2_receive_timestamp": {
"format": "uuuu-MM-dd HH:mm:ss.SSS",
"type": "date"
},
"full_message": {
"fielddata": false,
"analyzer": "standard",
"type": "text"
},
"streams": {
"type": "keyword"
},
"source": {
"fielddata": true,
"analyzer": "analyzer_keyword",
"type": "text"
},
"message": {
"fielddata": false,
"analyzer": "standard",
"type": "text"
},
"timestamp": {
"format": "uuuu-MM-dd HH:mm:ss.SSS",
"type": "date"
}
}
},
"aliases": {}
}
}
调整
- 模版内容
{
"template": "graylog_*",
"index_patterns": ["*"],
"mappings": {
"properties": {
"http_method": {
"type": "keyword"
},
"http_response_code": {
"type": "long"
},
"ingest_time": {
"type": "date",
"format": "strict_date_time"
},
"took_ms": {
"type": "long"
},
"response_body": {
"type": "text"
},
"request_body": {
"type": "text"
},
"request": {
"type": "text"
},
"http_user_agent": {
"type": "text"
}
}
}
}
- 配置
PUT /_template/graylog-custom-mapping?pretty
- 查看效果
GET /_template/graylog-custom-mapping?pretty
内容
{
"graylog-custom-mapping": {
"order": 0,
"index_patterns": [
"*"
],
"settings": {},
"mappings": {
"properties": {
"request": {
"type": "text"
},
"http_method": {
"type": "keyword"
},
"ingest_time": {
"format": "strict_date_time",
"type": "date"
},
"request_body": {
"type": "text"
},
"took_ms": {
"type": "long"
},
"response_body": {
"type": "text"
},
"http_response_code": {
"type": "long"
},
"http_user_agent": {
"type": "text"
}
}
},
"aliases": {}
}
}
代码处理
graylog2-server/src/main/java/org/graylog2/indexer/indices/Indices.java
- Indices.java
public void ensureIndexTemplate(IndexSet indexSet) {
final IndexSetConfig indexSetConfig = indexSet.getConfig();
final String templateName = indexSetConfig.indexTemplateName();
try {
final Map<String, Object> template = buildTemplate(indexSet, indexSetConfig);
if (indicesAdapter.ensureIndexTemplate(templateName, template)) {
LOG.info("Successfully ensured index template {}", templateName);
} else {
LOG.warn("Failed to create index template {}", templateName);
}
} catch (IgnoreIndexTemplate e) {
LOG.warn(e.getMessage());
if (e.isFailOnMissingTemplate() && !indicesAdapter.indexTemplateExists(templateName)) {
throw new IndexTemplateNotFoundException(f("No index template with name '%s' (type - '%s') found in Elasticsearch",
templateName, indexSetConfig.indexTemplateType().orElse(null)));
}
}
}
- 不同es 适配
比如es7 IndicesAdapterES7.java
@Override
public boolean ensureIndexTemplate(String templateName, Map<String, Object> template) {
final PutIndexTemplateRequest request = new PutIndexTemplateRequest(templateName)
.source(template);
final AcknowledgedResponse result = client.execute((c, requestOptions) -> c.indices().putTemplate(request, requestOptions),
"Unable to create index template " + templateName);
return result.isAcknowledged();
}
具体内部处理实际上是基于了sysjob，相关job 如下 
es 索引模型
- 写路径
- 读路径
说明
graylog 对于es 索引的管理还是比较方便的，充分利用了es 的能力，实现了比较强大的日志检索
参考资料
https://docs.graylog.org/docs/elasticsearch
https://docs.graylog.org/docs/index-model
https://docs.graylog.org/docs/query-language
https://www.elastic.co/guide/en/elasticsearch/reference/6.8/indices-templates.html
https://github.com/Graylog2/graylog2-server/blob/626be1f0d80506705b5ba41fbea33c2ec0164bc0/graylog2-server/src/main/java/org/graylog2/indexer/indices/Indices.java
https://github.com/Graylog2/graylog2-server/blob/626be1f0d80506705b5ba41fbea33c2ec0164bc0/graylog2-server/src/main/java/org/graylog2/indexer/indices/IndicesAdapter.java
