flink写入elasticsearch报错！OOM内存溢出！连接异常关闭！

最近公司有个项目，需要flink实时地对elasticsearch进行频繁的插入。但是在写入elasticsearch的时候出现了OOM内存溢出的异常，以及连接异常中断的错误。

报错如下：1.Caused by: java.lang.IllegalStateException: I/O reactor has been shut down  连接异常关闭。
         2.java.lang.OutOfMemoryError: Direct buffer memory    OOM内存溢出。

首先解决第一个异常，连接中断。网上很多人说是因为es的client调用了close方法，client请求在还没有完毕时就已经被关闭掉，导致后面的连接不可用，从而报出来这个异常。

但是我的代码一开始用的原生elasticsearch7.12来执行插入请求，没用调用close方法，所以异常可能是别的原因造成的。后面改为了flink封装的方法，需要手动关闭。

当然了，在解决这个问题之前，一定要保证代码本身执行没有问题，否则可能是其他的异常导致连接的关闭。

为了解决这个异常我们做了如下努力：

用flink封装的ElasticsearchSink代替es原生的client来执行插入的请求。（可能原生的也可以，但是我们在测试过程中发现，flink封装的效果更好，更不容易出错）

然后设置参数：

1.设置超时时间： requestBuilder.setConnectTimeout(60000); requestBuilder.setSocketTimeout(60000);这里两个超时时间都设置的一分钟。

2.设置最大连接数和刷新周期： esSinkBuilder.setBulkFlushMaxActions(1); esSinkBuilder.setBulkFlushMaxSizeMb(1); esSinkBuilder.setBulkFlushInterval(1);//刷新周期设置的1毫秒。

3.设置线程数量：

IOReactorConfig.custom().setIoThreadCount(5).build());

esSinkBuilder.setFailureHandler(new RetryRequestFailureHandler());//处理失败的Elasticsearch请求

这里sink每执行一次就要建立一次请求，所以要进行关闭。if（build！=null）build.close();

elasticsearch7.12版本使用了登录验证

完整代码如下：

  `//operator为flink数据流
    SingleOutputStreamOperator<JSONObject> operator;
    //elasticsearch 地址
    List<HttpHost> esAddresses = ESSinkUtil.getEsAddresses("locolhost1:9200,locolhost2:9200,locolhost3:9200,locolhost4:9200,locolhost5:9200");
    //getEsAddresses实体类
    public static List<HttpHost> getEsAddresses(String hosts) throws MalformedURLException {
    String[] hostList = hosts.split(",");
    List<HttpHost> addresses = new ArrayList<>();
    for (String host : hostList) {
        if (host.startsWith("http")) {
            URL url = new URL(host);
            addresses.add(new HttpHost(url.getHost(), url.getPort()));
        } else {
            String[] parts = host.split(":", 2);
            if (parts.length > 1) {
                addresses.add(new HttpHost(parts[0], Integer.parseInt(parts[1])));
            } else {
                throw new MalformedURLException("invalid elasticsearch hosts format");
            }
        }
    }
    return addresses;
}

    //elasticsearch插入请求
    ESSinkUtil.addSink(esAddresses, 1, 8, operator, new ElasticsearchSinkFunction<JSONObject>() {
            @Override
            public void process(JSONObject metric, RuntimeContext runtimeContext, RequestIndexer requestIndexer) {
                requestIndexer.add(Requests.indexRequest()
                        .index(INDEX)
                        .id(metric.get("id"))
                        .source(JSON.toJSONString(metric), XContentType.JSON));
            }
        });

    //flink封装的elasticsearch连接sink
    public static <T> void addSink(List<HttpHost> hosts, int bulkFlushMaxActions, int parallelism,
                               SingleOutputStreamOperator<T> data, ElasticsearchSinkFunction<T> func) {
        try {
            ElasticsearchSink.Builder<T> esSinkBuilder = new ElasticsearchSink.Builder<>(hosts, func);
            esSinkBuilder.setBulkFlushMaxActions(bulkFlushMaxActions);//每次最大插入数量
            esSinkBuilder.setBulkFlushMaxSizeMb(1);//最大插入内存
            esSinkBuilder.setBulkFlushInterval(1);//插入刷新周期
            esSinkBuilder.setFailureHandler(new RetryRequestFailureHandler());//处理失败的Elasticsearch请求

            //设置自定义http客户端配置
            esSinkBuilder.setRestClientFactory(new RestClientFactory() {
                @Override
                public void configureRestClientBuilder(RestClientBuilder restClientBuilder) {
                    final CredentialsProvider credentialsProvider =new BasicCredentialsProvider();
                    credentialsProvider.setCredentials(AuthScope.ANY,new UsernamePasswordCredentials(USER, PASSWORD));
                    restClientBuilder.setHttpClientConfigCallback(new RestClientBuilder.HttpClientConfigCallback() {

                        public HttpAsyncClientBuilder customizeHttpClient(HttpAsyncClientBuilder httpClientBuilder) {
                            //httpClientBuilder.disableAuthCaching();
                            httpClientBuilder.setDefaultCredentialsProvider(credentialsProvider);
                            return httpClientBuilder.setDefaultIOReactorConfig(
                                    IOReactorConfig.custom()
                                            .setIoThreadCount(5)//设置线程数量为5
                                            .build());
                        }
                    }).setRequestConfigCallback(new RestClientBuilder.RequestConfigCallback(){
                        @Override
                        public RequestConfig.Builder customizeRequestConfig(RequestConfig.Builder requestBuilder) {
                            requestBuilder.setConnectTimeout(5000);//连接超时时间
                            requestBuilder.setSocketTimeout(60000);
                            requestBuilder.setConnectionRequestTimeout(10000);
                            return requestBuilder;
                        }
                    });
                }
            });
            //todo:xpack security
            ElasticsearchSink<T> build = esSinkBuilder.build();
            data.addSink(build).setParallelism(parallelism);
            if (build!=null) build.close();//build用完后一定要关闭
        } catch (Exception e) {
            e.printStackTrace();
        }
  }

  //处理失败的Elasticsearch请求
  public class RetryRequestFailureHandler implements ActionRequestFailureHandler {

      public RetryRequestFailureHandler() {
      }

      @Override
      public void onFailure(ActionRequest actionRequest, Throwable throwable, int i, RequestIndexer requestIndexer) throws Throwable {
          if (ExceptionUtils.findThrowable(throwable, EsRejectedExecutionException.class).isPresent()) {
              requestIndexer.add(new ActionRequest[]{actionRequest});
          } else {
              if (ExceptionUtils.findThrowable(throwable, SocketTimeoutException.class).isPresent()) {
                  return;
              } else {
                  Optional<IOException> exp = ExceptionUtils.findThrowable(throwable, IOException.class);
                  if (exp.isPresent()) {
                      IOException ioExp = exp.get();
                      if (ioExp != null && ioExp.getMessage() != null && ioExp.getMessage().contains("max retry timeout")) {
                           return;
                      }
                  }
              }
              throw throwable;
          }
      }
  }

  //下面原生elasticsearch建立client连接
  public static RestHighLevelClient client(){
    final CredentialsProvider credentialsProvider = new BasicCredentialsProvider();
    credentialsProvider.setCredentials(AuthScope.ANY,
            new UsernamePasswordCredentials(USER, PASSWORD));  //es账号密码（默认用户名为elastic）
    //创建带用户名密码的ES客户端对象
    try {
        if (null == client){
            client = new RestHighLevelClient(RestClient.builder(new HttpHost(PRODUCE_HOST,PORT,SCHEMA)
                    ,new HttpHost(PRODUCE_HOST2,PORT,SCHEMA),new HttpHost(PRODUCE_HOST3,PORT,SCHEMA),new HttpHost(PRODUCE_HOST4,PORT,SCHEMA)
                    ,new HttpHost(PRODUCE_HOST5,PORT,SCHEMA))
                    //异步HTTPclient连接数配置
                    .setHttpClientConfigCallback(new RestClientBuilder.HttpClientConfigCallback() {
                        @Override
                        public HttpAsyncClientBuilder customizeHttpClient(HttpAsyncClientBuilder httpClientBuilder) {
                            //httpClientBuilder.disableAuthCaching();
                            httpClientBuilder.setDefaultCredentialsProvider(credentialsProvider);
                            return httpClientBuilder.setDefaultIOReactorConfig(
                                    IOReactorConfig.custom()
                                            .setIoThreadCount(5)
                                            .build());

                        }
                    })
                    .setRequestConfigCallback(new RestClientBuilder.RequestConfigCallback(){
                        @Override
                        public RequestConfig.Builder customizeRequestConfig(RequestConfig.Builder requestBuilder) {
                            requestBuilder.setConnectTimeout(5000);
                            requestBuilder.setSocketTimeout(60000);
                            requestBuilder.setConnectionRequestTimeout(10000);
                            return requestBuilder;
                        }
                    })
            );
        }
    } catch (Exception e) {
        e.printStackTrace();
    }
    return client;
}`

然后解决OOM内存溢出的问题，我们出发点是代码里调整并行度，保证有足够的slot可用，暂定为8。然后环境配置里调大内存。一般来说内存溢出就是存在内存泄漏

还有可能是代码本身异常太多，导致程序异常。通过修改代码，找到可能出现异常的地方，进行修改。

接着就是给flink设置重启策略

上述操作弄好之后，flink的报错就消失了，之前任务一直跑不上去，放到ui上面马上就报红失败。

码字不易，如果问题解决了别忘了留言点赞噢