flink web ui提交任务时出现Server Respoonse Message-Internal server error
环境说明
这段时间在梳理 flink,今天刚好测试之前的一个应用:在web ui 上提交任务报错了:
如图:这里指定了运行的主程序类名和并行度,然后点击submit 时,出错了
单纯从页面报错信息无法准确定位错误原因,此时可以通过 日志文件查看错误原因‘
[hui@hadoop103 ~]$ cd /opt/module/flink-local/log/ [hui@hadoop103 log]$ ls -ltr 总用量 440 -rw-r--r-- 1 hui wd 43688 6月 10 06:15 flink-hui-taskexecutor-0-hadoop103.log.1 -rw-r--r-- 1 hui wd 37268 6月 10 06:15 flink-hui-standalonesession-0-hadoop103.log.1 -rw-r--r-- 1 hui wd 750 6月 10 06:16 flink-hui-standalonesession-0-hadoop103.out -rw-r--r-- 1 hui wd 750 6月 10 06:16 flink-hui-taskexecutor-0-hadoop103.out -rw-r--r-- 1 hui wd 36562 6月 10 06:16 flink-hui-taskexecutor-0-hadoop103.log -rw-r--r-- 1 hui wd 28430 6月 10 06:22 flink-hui-standalonesession-1-hadoop103.log.1 -rw-r--r-- 1 hui wd 32769 6月 10 06:22 flink-hui-standalonesession-1-hadoop103.log.2 -rw-r--r-- 1 hui wd 36750 6月 10 06:26 flink-hui-standalonesession-0-hadoop103.log -rw-r--r-- 1 hui wd 31715 6月 10 08:28 flink-hui-client-hadoop103.log -rw-r--r-- 1 hui wd 128992 6月 10 09:42 flink-hui-standalonesession-1-hadoop103.log.3 -rw-r--r-- 1 hui wd 750 6月 10 16:39 flink-hui-standalonesession-1-hadoop103.out -rw-r--r-- 1 hui wd 47418 6月 10 16:50 flink-hui-standalonesession-1-hadoop103.log [hui@hadoop103 log]$ tail -100 flink-hui-standalonesession-1-hadoop103.log at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_144] at java.lang.Thread.run(Thread.java:748) [?:1.8.0_144] Caused by: java.lang.RuntimeException: No data for required key 'port' at org.apache.flink.api.java.utils.AbstractParameterTool.getRequired(AbstractParameterTool.java:79) ~[flink-dist_2.12-1.13.0.jar:1.13.0] at org.apache.flink.api.java.utils.AbstractParameterTool.getInt(AbstractParameterTool.java:106) ~[flink-dist_2.12-1.13.0.jar:1.13.0] at org.wdh01.wc.StreamWordCount.main(StreamWordCount.java:20) ~[?:?] at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_144] at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_144] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_144] at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_144] at org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:355) ~[flink-dist_2.12-1.13.0.jar:1.13.0] ... 13 more 2022-06-10 16:49:21,738 ERROR org.apache.flink.runtime.webmonitor.handlers.JarRunHandler [] - Exception occurred in REST handler: Could not execute application. 2022-06-10 16:49:30,964 WARN org.apache.flink.runtime.webmonitor.handlers.JarRunHandler [] - Configuring the job submission via query parameters is deprecated. Please migrate to submitting a JSON request instead. 2022-06-10 16:49:31,006 INFO org.apache.flink.client.ClientUtils [] - Starting program (detached: true) 2022-06-10 16:49:31,007 WARN org.apache.flink.client.deployment.application.DetachedApplicationRunner [] - Could not execute application: org.apache.flink.client.program.ProgramInvocationException: The main method caused an error: No data for required key 'port' at org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:372) ~[flink-dist_2.12-1.13.0.jar:1.13.0] at org.apache.flink.client.program.PackagedProgram.invokeInteractiveModeForExecution(PackagedProgram.java:222) ~[flink-dist_2.12-1.13.0.jar:1.13.0] at org.apache.flink.client.ClientUtils.executeProgram(ClientUtils.java:114) ~[flink-dist_2.12-1.13.0.jar:1.13.0] at org.apache.flink.client.deployment.application.DetachedApplicationRunner.tryExecuteJobs(DetachedApplicationRunner.java:84) ~[flink-dist_2.12-1.13.0.jar:1.13.0] at org.apache.flink.client.deployment.application.DetachedApplicationRunner.run(DetachedApplicationRunner.java:70) ~[flink-dist_2.12-1.13.0.jar:1.13.0] at org.apache.flink.runtime.webmonitor.handlers.JarRunHandler.lambda$handleRequest$0(JarRunHandler.java:102) ~[flink-dist_2.12-1.13.0.jar:1.13.0] at java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1590) [?:1.8.0_144] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_144] at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_144] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) [?:1.8.0_144] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) [?:1.8.0_144] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_144] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_144] at java.lang.Thread.run(Thread.java:748) [?:1.8.0_144] Caused by: java.lang.RuntimeException: No data for required key 'port' at org.apache.flink.api.java.utils.AbstractParameterTool.getRequired(AbstractParameterTool.java:79) ~[flink-dist_2.12-1.13.0.jar:1.13.0] at org.apache.flink.api.java.utils.AbstractParameterTool.getInt(AbstractParameterTool.java:106) ~[flink-dist_2.12-1.13.0.jar:1.13.0] at org.wdh01.wc.StreamWordCount.main(StreamWordCount.java:20) ~[?:?] at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_144] at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_144] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_144] at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_144] at org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:355) ~[flink-dist_2.12-1.13.0.jar:1.13.0] ... 13 more 2022-06-10 16:49:31,024 ERROR org.apache.flink.runtime.webmonitor.handlers.JarRunHandler [] - Exception occurred in REST handler: Could not execute application. 2022-06-10 16:49:39,493 WARN org.apache.flink.runtime.webmonitor.handlers.JarRunHandler [] - Configuring the job submission via query parameters is deprecated. Please migrate to submitting a JSON request instead. 2022-06-10 16:49:39,501 INFO org.apache.flink.client.ClientUtils [] - Starting program (detached: true) 2022-06-10 16:49:39,501 WARN org.apache.flink.client.deployment.application.DetachedApplicationRunner [] - Could not execute application: org.apache.flink.client.program.ProgramInvocationException: The main method caused an error: No data for required key 'port' at org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:372) ~[flink-dist_2.12-1.13.0.jar:1.13.0] at org.apache.flink.client.program.PackagedProgram.invokeInteractiveModeForExecution(PackagedProgram.java:222) ~[flink-dist_2.12-1.13.0.jar:1.13.0] at org.apache.flink.client.ClientUtils.executeProgram(ClientUtils.java:114) ~[flink-dist_2.12-1.13.0.jar:1.13.0] at org.apache.flink.client.deployment.application.DetachedApplicationRunner.tryExecuteJobs(DetachedApplicationRunner.java:84) ~[flink-dist_2.12-1.13.0.jar:1.13.0] at org.apache.flink.client.deployment.application.DetachedApplicationRunner.run(DetachedApplicationRunner.java:70) ~[flink-dist_2.12-1.13.0.jar:1.13.0] at org.apache.flink.runtime.webmonitor.handlers.JarRunHandler.lambda$handleRequest$0(JarRunHandler.java:102) ~[flink-dist_2.12-1.13.0.jar:1.13.0] at java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1590) [?:1.8.0_144] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_144] at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_144] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) [?:1.8.0_144] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) [?:1.8.0_144] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_144] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_144] at java.lang.Thread.run(Thread.java:748) [?:1.8.0_144] Caused by: java.lang.RuntimeException: No data for required key 'port' at org.apache.flink.api.java.utils.AbstractParameterTool.getRequired(AbstractParameterTool.java:79) ~[flink-dist_2.12-1.13.0.jar:1.13.0] at org.apache.flink.api.java.utils.AbstractParameterTool.getInt(AbstractParameterTool.java:106) ~[flink-dist_2.12-1.13.0.jar:1.13.0] at org.wdh01.wc.StreamWordCount.main(StreamWordCount.java:20) ~[?:?] at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_144] at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_144] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_144] at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_144] at org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:355) ~[flink-dist_2.12-1.13.0.jar:1.13.0] ... 13 more 2022-06-10 16:49:39,507 ERROR org.apache.flink.runtime.webmonitor.handlers.JarRunHandler [] - Exception occurred in REST handler: Could not execute application. 2022-06-10 16:50:18,223 WARN org.apache.flink.runtime.webmonitor.handlers.JarRunHandler [] - Configuring the job submission via query parameters is deprecated. Please migrate to submitting a JSON request instead. 2022-06-10 16:50:18,232 INFO org.apache.flink.client.ClientUtils [] - Starting program (detached: true) 2022-06-10 16:50:18,232 WARN org.apache.flink.client.deployment.application.DetachedApplicationRunner [] - Could not execute application: org.apache.flink.client.program.ProgramInvocationException: The main method caused an error: No data for required key 'port' at org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:372) ~[flink-dist_2.12-1.13.0.jar:1.13.0] at org.apache.flink.client.program.PackagedProgram.invokeInteractiveModeForExecution(PackagedProgram.java:222) ~[flink-dist_2.12-1.13.0.jar:1.13.0] at org.apache.flink.client.ClientUtils.executeProgram(ClientUtils.java:114) ~[flink-dist_2.12-1.13.0.jar:1.13.0] at org.apache.flink.client.deployment.application.DetachedApplicationRunner.tryExecuteJobs(DetachedApplicationRunner.java:84) ~[flink-dist_2.12-1.13.0.jar:1.13.0] at org.apache.flink.client.deployment.application.DetachedApplicationRunner.run(DetachedApplicationRunner.java:70) ~[flink-dist_2.12-1.13.0.jar:1.13.0] at org.apache.flink.runtime.webmonitor.handlers.JarRunHandler.lambda$handleRequest$0(JarRunHandler.java:102) ~[flink-dist_2.12-1.13.0.jar:1.13.0] at java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1590) [?:1.8.0_144] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_144] at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_144] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) [?:1.8.0_144] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) [?:1.8.0_144] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_144] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_144] at java.lang.Thread.run(Thread.java:748) [?:1.8.0_144] Caused by: java.lang.RuntimeException: No data for required key 'port' at org.apache.flink.api.java.utils.AbstractParameterTool.getRequired(AbstractParameterTool.java:79) ~[flink-dist_2.12-1.13.0.jar:1.13.0] at org.apache.flink.api.java.utils.AbstractParameterTool.getInt(AbstractParameterTool.java:106) ~[flink-dist_2.12-1.13.0.jar:1.13.0] at org.wdh01.wc.StreamWordCount.main(StreamWordCount.java:20) ~[?:?] at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_144] at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_144] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_144] at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_144] at org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:355) ~[flink-dist_2.12-1.13.0.jar:1.13.0] ... 13 more 2022-06-10 16:50:18,247 ERROR org.apache.flink.runtime.webmonitor.handlers.JarRunHandler [] - Exception occurred in REST handler: Could not execute application.
通过日志查看原来是,运行的这个java程序需要两个参数,也就是说这个程序需要运行时动态传入参数:它的java程序是这样
public static void main(String[] args) throws Exception { //从参数读取 host & port ParameterTool parameterTool = ParameterTool.fromArgs(args); String host = parameterTool.get("host"); int port = parameterTool.getInt("port"); StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment(); DataStreamSource<String> socketTextStream = env.socketTextStream(host, port); socketTextStream.flatMap((String line, Collector<Tuple2<String, Long>> out) -> { String[] s = line.split(" "); for (String s1 : s) { out.collect(Tuple2.of(s1, 1L)); } }).returns(Types.TUPLE(Types.STRING, Types.LONG)) .keyBy(data -> data.f0) .sum(1) .print(); env.execute(); }
所以在提交作业的时候需要传入两个参数’
这样就可以提交任务了。如果在运行程序的时候除了问题,不用着急,一般情况下是可以通过日志来判断问题所在,这样就可以急需处理了。