flink 与 es 的一些问题

写入es maping字段类型冲突

error1:
org.apache.flink.streaming.connectors.elasticsearch.util.NoOpFailureHandler.onFailure(NoOpFailureHandler.java:40) - Failed Elasticsearch item request: Elasticsearch exception [type=mapper_parsing_exception, reason=failed to parse field [alarm_id] of type [long] in document with id 'jQTyeYcBKEN7eJDWu4vB'. Preview of field's value: '30001601a']
org.elasticsearch.ElasticsearchException: Elasticsearch exception [type=mapper_parsing_exception, reason=failed to parse field [alarm_id] of type [long] in document with id 'jQTyeYcBKEN7eJDWu4vB'. Preview of field's value: '30001601a']

error2:
Caused by: ElasticsearchException[Elasticsearch exception [type=mapper_parsing_exception, reason=object mapping for [action_object] tried to parse field [action_object] as object, but found a concrete value]]
    at org.elasticsearch.ElasticsearchException.innerFromXContent(ElasticsearchException.java:496)

java.lang.NoClassDefFoundError 通常由于maven依赖和服务器的依赖冲突

Caused by: java.lang.NoClassDefFoundError: org/elasticsearch/common/unit/TimeValue

心跳超时，服务器taskmanager挂掉

java.util.concurrent.TimeoutException: Heartbeat of TaskManager with id 172.16.55.125:40811-057c3d timed out.

数据进入es sink算子没有数据写出的场景
添加配置
setBulkFlushMaxActions
setBulkFlushInterval

由于泛型擦除，需要添加类型推断

org.apache.flink.api.common.functions.InvalidTypesException: Could not determine TypeInformation for the OutputTag type. The most common reason is forgetting to make the OutputTag an anonymous inner class. It is also not possible to use generic type variables with OutputTags, such as 'Tuple2<A, B>'. 

内存溢出：大于es配置内存

Suppressed: org.elasticsearch.client.ResponseException: method [POST], host [http://xxxx:xxx], URI [/xxxx/_search?typed_keys=true&max_concurrent_shard_requests=5&ignore_unavailable=false&expand_wildcards=open&allow_no_indices=true&ignore_throttled=true&search_type=query_then_fetch&batched_reduce_size=512&ccs_minimize_roundtrips=true], status line [HTTP/1.1 429 Too Many Requests]
{"error":{"root_cause":[{"type":"circuit_breaking_exception","reason":"[parent] Data too large, data for [<http_request>] would be [2071980026/1.9gb], which is larger than the limit of [2040109465/1.8gb], real usage: [2071979472/1.9gb], new bytes reserved: [554/554b], usages [request=0/0b, fielddata=16381/15.9kb, in_flight_requests=46624318/44.4mb, model_inference=0/0b, eql_sequence=0/0b, accounting=8707688/8.3mb]","bytes_wanted":2071980026,"bytes_limit":2040109465,"durability":"TRANSIENT"}],"type":"circuit_breaking_exception","reason":"[parent] Data too large, data for [<http_request>] would be [2071980026/1.9gb], which is larger than the limit of [2040109465/1.8gb], real usage: [2071979472/1.9gb], new bytes reserved: [554/554b], usages [request=0/0b, fielddata=16381/15.9kb, in_flight_requests=46624318/44.4mb, model_inference=0/0b, eql_sequence=0/0b, accounting=8707688/8.3mb]","bytes_wanted":2071980026,"bytes_limit":2040109465,"durability":"TRANSIENT"},"status":429}
        at org.elasticsearch.client.RestClient.convertResponse(RestClient.java:326)
        at org.elasticsearch.client.RestClient.performRequest(RestClient.java:296)
        at org.elasticsearch.client.RestClient.performRequest(RestClient.java:270)
        at org.elasticsearch.client.RestHighLevelClient.internalPerformRequest(RestHighLevelClient.java:1632)
        ... 24 more

es写入文档超过限制

处理思路：减少对象数量，增加es中文档对应的对象限制

PUT /index/_settings
{
"index.mapping.nested_objects.limit": 20000
}

Caused by: ElasticsearchException[Elasticsearch exception [type=mapper_parsing_exception, reason=The number of nested documents has exceeded the allowed limit of [10000]. This limit can be set by changing the [index.mapping.nested_objects.limit] index level setting.]]
    at org.elasticsearch.ElasticsearchException.innerFromXContent(ElasticsearchException.java:496)
    at org.elasticsearch.ElasticsearchException.fromXContent(ElasticsearchException.java:407)
    at org.elasticsearch.action.bulk.BulkItemResponse.fromXContent(BulkItemResponse.java:139)
    at org.elasticsearch.action.bulk.BulkResponse.fromXContent(BulkResponse.java:188)
    at org.elasticsearch.client.RestHighLevelClient.parseEntity(RestHighLevelClient.java:1911)
    at org.elasticsearch.client.RestHighLevelClient.lambda$performRequestAsyncAndParseEntity$10(RestHighLevelClient.java:1699)
    at org.elasticsearch.client.RestHighLevelClient$1.onSuccess(RestHighLevelClient.java:1781)
    at org.elasticsearch.client.RestClient$FailureTrackingResponseListener.onSuccess(RestClient.java:636)

flink单机版-分布式部署存在的问题

flink任务重启时会存在找不到检查点数据的情况

Caused by: java.io.FileNotFoundException: /home/data/flink/flink-bucket/flink-checkpoints/532e6feb43cfd4981e7041b7bc5f6207/shared/dcf5a996-8b18-421d-8ced-47ed89e53dc0 (没有那个文件或目录)
    at java.io.FileInputStream.open0(Native Method)
    at java.io.FileInputStream.open(FileInputStream.java:195)
    at java.io.FileInputStream.<init>(FileInputStream.java:138)
    at org.apache.flink.core.fs.local.LocalDataInputStream.<init>(LocalDataInputStream.java:50)
    at org.apache.flink.core.fs.local.LocalFileSystem.open(LocalFileSystem.java:134)
    at org.apache.flink.runtime.state.filesystem.FileStateHandle.openInputStream(FileStateHandle.java:69)
    at org.apache.flink.contrib.streaming.state.RocksDBStateDownloader.downloadDataForStateHandle(RocksDBStateDownloader.java:127)
    at org.apache.flink.contrib.streaming.state.RocksDBStateDownloader.lambda$createDownloadRunnables$0(RocksDBStateDownloader.java:110)
    at org.apache.flink.util.function.ThrowingRunnable.lambda$unchecked$0(ThrowingRunnable.java:49)
    at java.util.concurrent.CompletableFuture$AsyncRun.run(CompletableFuture.java:1626)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    ... 1 more

测输出流标签，未指定类型

java.lang.IllegalArgumentException: The requested state does not exist. Check for typos in your state descriptor, or specify the state descriptor in the datastream.broadcast(...) call if you forgot to register it.

org.apache.flink.api.common.functions.InvalidTypesException: Could not determine TypeInformation for the OutputTag type. The most common reason is forgetting to make the OutputTag an anonymous inner class. It is also not possible to use generic type variables with OutputTags, such as 'Tuple2<A, B>'.

__EOF__

本文作者：Choice
本文链接：https://www.cnblogs.com/choice7/p/17746289.html
关于博主：评论和私信会在第一时间回复。或者直接私信我。
版权声明：本博客所有文章除特别声明外，均采用 BY-NC-SA 许可协议。转载请注明出处！
声援博主：如果您觉得文章对您有帮助，可以点击文章右下角【推荐】一下。您的鼓励是博主的最大动力！