Spark笔记(一):错误总结

1.转义字符:

　　常见的replaceAll,split,mkstring中涉及到特殊字符的都要加上转义字符,比如str.split("\\|"),str.replaceAll("\\[", "")

     报错信息:

      java.util.regex.PatternSyntaxException: Unclosed character class near index 0

      java.util.regex.PatternSyntaxException: Unexpected internal error near index 1

 

2.kafka中数据还没来得及消费,数据就已经丢失或者过期了:

　　[https://blog.csdn.net/yxgxy270187133/article/details/53666760]

　　报错信息:

　　org.apache.kafka.clients.consumer.OffsetOutOfRangeException: Offsets out of range with no configured reset policy for partitions:{...}

3.spark submit 内存参数太小 --executor-memory 8G \  --driver-memory 8G \

　　报错信息:

　　 Application application_1547156777102_0243 failed 2 times due to AM Container for appattempt_1547156777102_0243_000002 exited with exitCode: -104
For more detailed output, check the application tracking page:https://host-10-31-4-246:26001/cluster/app/application_1547156777102_0243 Then click on links to logs of each attempt.
Diagnostics: Container [pid=5064,containerID=container_e62_1547156777102_0243_02_000001] is running beyond physical memory limits. Current usage: 4.6 GB of 4.5 GB physical memory used; 6.3 GB of 22.5 GB virtual memory used. Killing container.