ES索引瘦身 压缩——_source _all 均disable filed store为no，引入第三方DB存储原始数据，去掉pos倒排和doc_values，强制定期merge segments，将所有fileds合并为一个field big string

原始数据：835MB

 

ES 设置了_source _all disabled 且设置了仅仅存docs倒排
Wed Feb 22 11:58:27 CST 2017
Before size:
1 /home/bone/elasticsearch-2.4.1/data/
OK, waiting...
0
After size:
435 /home/bone/elasticsearch-2.4.1/data/
Wed Feb 22 11:59:43 CST 2017
done

不做任何压缩：
Wed Feb 22 14:56:04 CST 2017
Before size:
1 /home/bone/elasticsearch-2.4.1/data/
OK, waiting...
0
After size:
1656 /home/bone/elasticsearch-2.4.1/data/
Wed Feb 22 14:57:54 CST 2017
done
curl -XPOST 'localhost:9200/hec_test3/_forcemerge'
bone@PEK1000074003:~/test_data/splunk_es_cmp$ du -sm ~/elasticsearch-2.4.1/data/
969 /home/bone/elasticsearch-2.4.1/data/

_all disabled：
Wed Feb 22 15:22:09 CST 2017
Before size:
1 /home/bone/elasticsearch-2.4.1/data/
OK, waiting...
0
After size:
1576 /home/bone/elasticsearch-2.4.1/data/
Wed Feb 22 15:23:25 CST 2017
done
merge后：
787

_source disabled: field store为false 搜索不到东西了 如果不存DB的话
Wed Feb 22 15:26:06 CST 2017
Before size:
1 /home/bone/elasticsearch-2.4.1/data/
OK, waiting...
0
After size:
1288 /home/bone/elasticsearch-2.4.1/data/
Wed Feb 22 15:27:45 CST 2017
done
merge后：
512

_source disabled: field store为yes
Wed Feb 22 15:33:28 CST 2017
Before size:
1 /home/bone/elasticsearch-2.4.1/data/
OK, waiting...
0
After size:
1583 /home/bone/elasticsearch-2.4.1/data/
Wed Feb 22 15:34:45 CST 2017
done
merge后：
780

最终设置：
curl -XPUT 'http://localhost:9200/hec_test3' -d '
{
"mappings": {
"hec_type3": {
"_source": {
"enabled": false
},
"_all": {
"enabled": false
},
"properties": {
"event": {
"type": "string",
"include_in_all": false,
"doc_values": false,
"index_options": "docs"
}
}
}
}
}
'
压缩配置：
index.codec: best_compression

bash test_es24.sh
Wed Feb 22 15:39:39 CST 2017
Before size:
1 /home/bone/elasticsearch-2.4.1/data/
OK, waiting...
0
After size:
892 /home/bone/elasticsearch-2.4.1/data/
Wed Feb 22 15:40:36 CST 2017
done
merge后：
202 MB

50个字段拆分后：
用上述最终设置的效果：
Wed Feb 22 16:02:59 CST 2017
Before size:
1 /home/bone/elasticsearch-2.4.1/data/
OK, waiting...
0
After size:
1423 /home/bone/elasticsearch-2.4.1/data/
Wed Feb 22 16:05:04 CST 2017
done
merge后：686

不做任何配置：
Wed Feb 22 16:08:30 CST 2017
Before size:
1 /home/bone/elasticsearch-2.4.1/data/
OK, waiting...
0
After size:
2641 /home/bone/elasticsearch-2.4.1/data/
Wed Feb 22 16:11:54 CST 2017
done
merge后：
1650