ES7.5.2索引生命周期管理(附操作示例)
一、前言
es可以用来存储日志，一般日志存储只是短期保存，超过一定时间日志要是能自动删除最好，这样保证索引文档不会过多，查询时效性也能得到保证。本文参考的官网地址是：https://www.elastic.co/guide/en/elasticsearch/reference/7.5/getting-started-index-lifecycle-management.html 。索引的生命周期分为四个阶段：HOT->WARM->COLD->DELETE。
上面除了HOT为必须的阶段外，其他为非必须阶段，可以任意选择配置。因为日志索引只要满足自己删除功能，所以下文只配置了HOT与DELETE阶段。
二、配置
配置自己删除功能，必须配置策略(policy)、索引模版(template)、索引(index)。上述配置缺一不可。
1）配置策略
PUT _ilm/policy/datastream_policy { "policy": { "phases": { "hot": { "actions": { "rollover":{ "max_docx":1 } } }, "delete": { "min_age": "30s", "actions": { "delete": {} } } } } }
采用PUT方式向es创建一个名为“datastream_policy“的策略,"_ilm/policy"为固定写法。如果是使用postman，选择PUT请求方式，地址为：http://ip:port/_ilm/policy/datastream_policy ，HOT阶段：当日志索引超过1条时就会发生rollover，此处设置1条主要是为了测试方便。DELETE阶段：超过30秒时执行删除。
hot->actions->rollover还支持其他维度的控制，比如：
"max_size": "50GB" "max_age": "30d"
2）配置索引模版
PUT _template/datastream_template { "index_patterns": ["datastream-*"], "settings": { "number_of_shards": 1, "number_of_replicas": 1, "index.lifecycle.name": "datastream_policy", "index.lifecycle.rollover_alias": "datastream" } }
创建一个索引模版，索引以datastream-开头的自动采用settings的配置。"index.lifecycle.name"意为采用”datastream_policy“的策略，"index.lifecycle.rollover_alias"意为创建使用该模版创建的索引统一用”datastream“的别名进行管理。
3）创建索引
PUT datastream-000001 { "aliases": { "datastream": { "is_write_index": true } } }
创建一个开始的索引，并设置索引可写入。
4）配置lifecycle检测时间
PUT /_cluster/settings
{ "transient": { "indices.lifecycle.poll_interval": "10s" } }
默认为十分钟，为了测试看效果，改为10秒钟。
5）验证功能
向别名为"datastream"的索引中进行写入多条数据
PUT /datastream/_doc
{
"message":"hello-01"
}
索引采用别名索引。
写入后，es根据配置的lifecycle时间进行rollover。用以下命令查看索引rollover进度
GET datastream-*/_ilm/explain
等待30秒后，返回的数据如下：
{ "indices": { "datastream-000002": { "index": "datastream-000002", "managed": true, "policy": "datastream_policy", "lifecycle_date_millis": 1587608411208, "age": "1.08m", "phase": "hot", "phase_time_millis": 1587608411441, "action": "rollover", "action_time_millis": 1587608471383, "step": "check-rollover-ready", "step_time_millis": 1587608471383, "phase_execution": { "policy": "datastream_policy", "phase_definition": { "min_age": "0ms", "actions": { "rollover": { "max_docs": 1 } } }, "version": 1, "modified_date_in_millis": 1587608224989 } }, "datastream-000001": { "index": "datastream-000001", "managed": true, "policy": "datastream_policy", "lifecycle_date_millis": 1587608411268, "age": "1.08m", "phase": "delete", "phase_time_millis": 1587608471345, "action": "delete", "action_time_millis": 1587608471345, "step": "wait-for-shard-history-leases", "step_time_millis": 1587608471345, "phase_execution": { "policy": "datastream_policy", "phase_definition": { "min_age": "60s", "actions": { "delete": {} } }, "version": 1, "modified_date_in_millis": 1587608224989 } } } }
索引自动创建了一个新的"datastream-000002"，旧的索引”datastream-000001“ action变成DELETE。
再过一段时间再次查看索引
GET datastream-*/_ilm/explain
查看结果：
{ "indices": { "datastream-000002": { "index": "datastream-000002", "managed": true, "policy": "datastream_policy", "lifecycle_date_millis": 1587608411208, "age": "2.8m", "phase": "hot", "phase_time_millis": 1587608411441, "action": "rollover", "action_time_millis": 1587608471383, "step": "check-rollover-ready", "step_time_millis": 1587608471383, "phase_execution": { "policy": "datastream_policy", "phase_definition": { "min_age": "0ms", "actions": { "rollover": { "max_docs": 1 } } }, "version": 1, "modified_date_in_millis": 1587608224989 } } } }
旧的索引进行了删除。在做日志查询时，我们只需要使用别名”datastream“进行日志查询即可，es便可无感知完成日志存储删除动作。
