logstash同步es到es从库

 

docker run -dit --name logstash153_60 -v /home/docker/logstash/:/etc/logstash/pipeline logstash:5.6.13

 

logstash -f /etc/logstash/pipeline/es153_60.conf --path.data=/root/ &

 

如果执行上述命令有报错如下日志需要再output的es中加入配置

logstash retrying failed action with response code: 429 - kuroniko - 博客园 (cnblogs.com)

 

 

使用logstash进行同步的时候报如下错误：

09:08:22.284 [[main]>worker4] INFO  logstash.outputs.elasticsearch - retrying failed action with response code: 429 ({"type"=>"es_rejected_execution_exception", "reason"=>"rejected execution of org.elasticsearch.transport.TransportService$7@6d1e3b7a on EsThreadPoolExecutor[bulk, queue capacity = 200, org.elasticsearch.common.util.concurrent.EsThreadPoolExecutor@3cd6876e[Running, pool size = 8, active threads = 8, queued tasks = 200, completed tasks = 651632]]"})

 

判断应该是es的写入性能不行，修改es的配置添加如下配置，然后重启es生效

 

thread_pool.bulk.queue_size: 10000 # 写入队列大小
indices.memory.index_buffer_size: 30%

 

 

 

 

 

 

input{
elasticsearch {
hosts => "15.99.72.153:9200"
index => "eaddcomponentindex_*"
schedule => "* */6 * * *"
docinfo => true
size => 1000        scroll => "5m"      }}

 

output {
elasticsearch {
hosts => ["15.31.212.60:9200"]
index => "%{[@metadata][_index]}"
document_type => "%{[@metadata][_type]}"
document_id => "%{[@metadata][_id]}"}}