spark submit 提交任务报错

/opt/module/spark-2.1.1-bin-hadoop2.7/bin/spark-submit --class org.apache.spark.examples.SparkPi --master spark://s1:7077 --executor-memory 1G --total-executor-cores 2 /opt/module/spark-2.1.1-bin-hadoop2.7/examples/jars/spark-examples_2.11-2.1.1.jar 100
19/06/04 17:34:44 INFO SparkContext: Running Spark version 2.1.1
19/06/04 17:34:44 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
19/06/04 17:34:44 INFO SecurityManager: Changing view acls to: s
19/06/04 17:34:44 INFO SecurityManager: Changing modify acls to: s
19/06/04 17:34:44 INFO SecurityManager: Changing view acls groups to:
19/06/04 17:34:44 INFO SecurityManager: Changing modify acls groups to:
19/06/04 17:34:44 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(s); groups with view permissions: Set(); users with modify permissions: Set(s); groups with modify permissions: Set()
19/06/04 17:34:44 INFO Utils: Successfully started service 'sparkDriver' on port 45658.
19/06/04 17:34:44 INFO SparkEnv: Registering MapOutputTracker
19/06/04 17:34:44 INFO SparkEnv: Registering BlockManagerMaster
19/06/04 17:34:44 INFO BlockManagerMasterEndpoint: Using org.apache.spark.storage.DefaultTopologyMapper for getting topology information
19/06/04 17:34:44 INFO BlockManagerMasterEndpoint: BlockManagerMasterEndpoint up
19/06/04 17:34:45 INFO DiskBlockManager: Created local directory at /tmp/blockmgr-8340dbad-afd5-416a-8b23-937d014b066e
19/06/04 17:34:45 INFO MemoryStore: MemoryStore started with capacity 366.3 MB
19/06/04 17:34:45 INFO SparkEnv: Registering OutputCommitCoordinator
19/06/04 17:34:45 INFO Utils: Successfully started service 'SparkUI' on port 4040.
19/06/04 17:34:45 INFO SparkUI: Bound SparkUI to 0.0.0.0, and started at http://192.168.100.5:4040
19/06/04 17:34:45 INFO SparkContext: Added JAR file:/opt/module/spark-2.1.1-bin-hadoop2.7/examples/jars/spark-examples_2.11-2.1.1.jar at spark://192.168.100.5:45658/jars/spark-examples_2.11-2.1.1.jar with timestamp 1559640885341
19/06/04 17:34:45 INFO StandaloneAppClient$ClientEndpoint: Connecting to master spark://s1:7077...
19/06/04 17:34:45 INFO TransportClientFactory: Successfully created connection to s1/192.168.100.5:7077 after 23 ms (0 ms spent in bootstraps)
19/06/04 17:34:45 INFO StandaloneSchedulerBackend: Connected to Spark cluster with app ID app-20190604173445-0001
19/06/04 17:34:45 INFO StandaloneAppClient$ClientEndpoint: Executor added: app-20190604173445-0001/0 on worker-20190604171227-192.168.100.6-42147 (192.168.100.6:42147) with 1 cores
19/06/04 17:34:45 INFO Utils: Successfully started service 'org.apache.spark.network.netty.NettyBlockTransferService' on port 41557.
19/06/04 17:34:45 INFO NettyBlockTransferService: Server created on 192.168.100.5:41557
19/06/04 17:34:45 INFO StandaloneSchedulerBackend: Granted executor ID app-20190604173445-0001/0 on hostPort 192.168.100.6:42147 with 1 cores, 1024.0 MB RAM
19/06/04 17:34:45 INFO StandaloneAppClient$ClientEndpoint: Executor added: app-20190604173445-0001/1 on worker-20190604171226-192.168.100.5-44172 (192.168.100.5:44172) with 1 cores
19/06/04 17:34:45 INFO StandaloneSchedulerBackend: Granted executor ID app-20190604173445-0001/1 on hostPort 192.168.100.5:44172 with 1 cores, 1024.0 MB RAM
19/06/04 17:34:45 INFO BlockManager: Using org.apache.spark.storage.RandomBlockReplicationPolicy for block replication policy
19/06/04 17:34:45 INFO BlockManagerMaster: Registering BlockManager BlockManagerId(driver, 192.168.100.5, 41557, None)
19/06/04 17:34:45 INFO BlockManagerMasterEndpoint: Registering block manager 192.168.100.5:41557 with 366.3 MB RAM, BlockManagerId(driver, 192.168.100.5, 41557, None)
19/06/04 17:34:45 INFO BlockManagerMaster: Registered BlockManager BlockManagerId(driver, 192.168.100.5, 41557, None)
19/06/04 17:34:45 INFO StandaloneAppClient$ClientEndpoint: Executor updated: app-20190604173445-0001/0 is now RUNNING
19/06/04 17:34:45 INFO BlockManager: Initialized BlockManager: BlockManagerId(driver, 192.168.100.5, 41557, None)
19/06/04 17:34:45 INFO StandaloneAppClient$ClientEndpoint: Executor updated: app-20190604173445-0001/1 is now RUNNING
19/06/04 17:34:47 ERROR SparkContext: Error initializing SparkContext.
java.net.ConnectException: Call From s1/192.168.100.5 to s1:8021 failed on connection exception: java.net.ConnectException: 拒绝连接; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at org.apache.hadoop.net.NetUtils.wrapWithMessage(NetUtils.java:792)
at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:732)
at org.apache.hadoop.ipc.Client.call(Client.java:1479)
at org.apache.hadoop.ipc.Client.call(Client.java:1412)
at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:229)
at com.sun.proxy.$Proxy13.getFileInfo(Unknown Source)
at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getFileInfo(ClientNamenodeProtocolTranslatorPB.java:771)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:191)
at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102)
at com.sun.proxy.$Proxy14.getFileInfo(Unknown Source)
at org.apache.hadoop.hdfs.DFSClient.getFileInfo(DFSClient.java:2108)
at org.apache.hadoop.hdfs.DistributedFileSystem$22.doCall(DistributedFileSystem.java:1305)
at org.apache.hadoop.hdfs.DistributedFileSystem$22.doCall(DistributedFileSystem.java:1301)
at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1317)
at org.apache.spark.scheduler.EventLoggingListener.start(EventLoggingListener.scala:93)
at org.apache.spark.SparkContext.<init>(SparkContext.scala:531)
at org.apache.spark.SparkContext$.getOrCreate(SparkContext.scala:2320)
at org.apache.spark.sql.SparkSession$Builder$$anonfun$6.apply(SparkSession.scala:868)
at org.apache.spark.sql.SparkSession$Builder$$anonfun$6.apply(SparkSession.scala:860)
at scala.Option.getOrElse(Option.scala:121)
at org.apache.spark.sql.SparkSession$Builder.getOrCreate(SparkSession.scala:860)
at org.apache.spark.examples.SparkPi$.main(SparkPi.scala:31)
at org.apache.spark.examples.SparkPi.main(SparkPi.scala)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:743)
at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:187)
at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:212)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:126)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
Caused by: java.net.ConnectException: 拒绝连接
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
at org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206)
at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:531)
at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:495)
at org.apache.hadoop.ipc.Client$Connection.setupConnection(Client.java:614)
at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:712)
at org.apache.hadoop.ipc.Client$Connection.access$2900(Client.java:375)
at org.apache.hadoop.ipc.Client.getConnection(Client.java:1528)
at org.apache.hadoop.ipc.Client.call(Client.java:1451)
... 34 more
19/06/04 17:34:47 INFO SparkUI: Stopped Spark web UI at http://192.168.100.5:4040
19/06/04 17:34:47 INFO StandaloneSchedulerBackend: Shutting down all executors
19/06/04 17:34:47 INFO CoarseGrainedSchedulerBackend$DriverEndpoint: Asking each executor to shut down
19/06/04 17:34:47 INFO MapOutputTrackerMasterEndpoint: MapOutputTrackerMasterEndpoint stopped!
19/06/04 17:34:47 INFO MemoryStore: MemoryStore cleared
19/06/04 17:34:47 INFO BlockManager: BlockManager stopped
19/06/04 17:34:47 INFO BlockManagerMaster: BlockManagerMaster stopped
19/06/04 17:34:47 INFO OutputCommitCoordinator$OutputCommitCoordinatorEndpoint: OutputCommitCoordinator stopped!

 

 

 

 

 

 

 

原因是：spark-default文件配置错误