18/03/18 04:53:44 WARN TaskSchedulerImpl: Initial job has not accepted any resources; check your cluster UI to ensure that workers are registered and have sufficient resources
1:遇到这个问题是在启动bin/spark-shell以后,然后呢,执行spark实现wordcount的例子的时候出现错误了,如:
scala> sc.textFile("hdfs://slaver1:9000/hello.txt").flatMap(_.split(" ")).map((_,1)).reduceByKey(_ + _).collect
执行上面操作之前我是这样启动我的spark-shell的,如下所示:
[root@slaver1 spark-1.6.1-bin-hadoop2.6]# bin/spark-shell
问题就出现在这里,首先你要知道自己笔记本的内存大小,如果你租的服务器内存应该可以满足你的需求,这里就说我的本本8G内存,三个虚拟机分别分配了1G内存,然后呢,spark部署的时候,在spark-env.sh配置的时候,特别写了spark可以使用800M的内存,毕竟这货是靠吃内存闻名的。
然后呢,感觉一切都挺正常的,但是呢,启动这个spark-shell的时候指明内存大小,就可以解决这个问题,但是指定的内存大小必须比800M小,我指定了500M,如下所示:
[root@slaver1 spark-1.6.1-bin-hadoop2.6]# bin/spark-shell --master spark://slaver1:7077 --executor-memory 512M --total-executor-cores 2 18/03/18 05:03:18 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 18/03/18 05:03:18 INFO SecurityManager: Changing view acls to: root 18/03/18 05:03:18 INFO SecurityManager: Changing modify acls to: root 18/03/18 05:03:18 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(root); users with modify permissions: Set(root) 18/03/18 05:03:19 INFO HttpServer: Starting HTTP Server 18/03/18 05:03:19 INFO Utils: Successfully started service 'HTTP class server' on port 35095. Welcome to ____ __ / __/__ ___ _____/ /__ _\ \/ _ \/ _ `/ __/ '_/ /___/ .__/\_,_/_/ /_/\_\ version 1.6.1 /_/ Using Scala version 2.10.5 (Java HotSpot(TM) Client VM, Java 1.7.0_65) Type in expressions to have them evaluated. Type :help for more information. 18/03/18 05:03:24 INFO SparkContext: Running Spark version 1.6.1 18/03/18 05:03:24 WARN SparkConf: SPARK_WORKER_INSTANCES was detected (set to '1'). This is deprecated in Spark 1.0+. Please instead use: - ./spark-submit with --num-executors to specify the number of executors - Or set SPARK_EXECUTOR_INSTANCES - spark.executor.instances to configure the number of instances in the spark config. 18/03/18 05:03:24 INFO SecurityManager: Changing view acls to: root 18/03/18 05:03:24 INFO SecurityManager: Changing modify acls to: root 18/03/18 05:03:24 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(root); users with modify permissions: Set(root) 18/03/18 05:03:25 INFO Utils: Successfully started service 'sparkDriver' on port 56128. 18/03/18 05:03:26 INFO Slf4jLogger: Slf4jLogger started 18/03/18 05:03:26 INFO Remoting: Starting remoting 18/03/18 05:03:26 INFO Remoting: Remoting started; listening on addresses :[akka.tcp://sparkDriverActorSystem@192.168.19.128:35173] 18/03/18 05:03:26 INFO Utils: Successfully started service 'sparkDriverActorSystem' on port 35173. 18/03/18 05:03:26 INFO SparkEnv: Registering MapOutputTracker 18/03/18 05:03:26 INFO SparkEnv: Registering BlockManagerMaster 18/03/18 05:03:26 INFO DiskBlockManager: Created local directory at /tmp/blockmgr-4a581245-fbf6-4741-81b1-bb295995d95a 18/03/18 05:03:26 INFO MemoryStore: MemoryStore started with capacity 517.4 MB 18/03/18 05:03:26 INFO SparkEnv: Registering OutputCommitCoordinator 18/03/18 05:03:27 INFO Utils: Successfully started service 'SparkUI' on port 4040. 18/03/18 05:03:27 INFO SparkUI: Started SparkUI at http://192.168.19.128:4040 18/03/18 05:03:27 INFO AppClient$ClientEndpoint: Connecting to master spark://slaver1:7077... 18/03/18 05:03:28 INFO SparkDeploySchedulerBackend: Connected to Spark cluster with app ID app-20180318050328-0006 18/03/18 05:03:28 INFO Utils: Successfully started service 'org.apache.spark.network.netty.NettyBlockTransferService' on port 40416. 18/03/18 05:03:28 INFO NettyBlockTransferService: Server created on 40416 18/03/18 05:03:28 INFO BlockManagerMaster: Trying to register BlockManager 18/03/18 05:03:28 INFO AppClient$ClientEndpoint: Executor added: app-20180318050328-0006/0 on worker-20180318032642-192.168.19.130-39494 (192.168.19.130:39494) with 1 cores 18/03/18 05:03:28 INFO SparkDeploySchedulerBackend: Granted executor ID app-20180318050328-0006/0 on hostPort 192.168.19.130:39494 with 1 cores, 512.0 MB RAM 18/03/18 05:03:28 INFO BlockManagerMasterEndpoint: Registering block manager 192.168.19.128:40416 with 517.4 MB RAM, BlockManagerId(driver, 192.168.19.128, 40416) 18/03/18 05:03:28 INFO BlockManagerMaster: Registered BlockManager 18/03/18 05:03:28 INFO AppClient$ClientEndpoint: Executor added: app-20180318050328-0006/1 on worker-20180318032636-192.168.19.129-32847 (192.168.19.129:32847) with 1 cores 18/03/18 05:03:28 INFO SparkDeploySchedulerBackend: Granted executor ID app-20180318050328-0006/1 on hostPort 192.168.19.129:32847 with 1 cores, 512.0 MB RAM 18/03/18 05:03:28 INFO AppClient$ClientEndpoint: Executor updated: app-20180318050328-0006/0 is now RUNNING 18/03/18 05:03:28 INFO AppClient$ClientEndpoint: Executor updated: app-20180318050328-0006/1 is now RUNNING 18/03/18 05:03:29 INFO SparkDeploySchedulerBackend: SchedulerBackend is ready for scheduling beginning after reached minRegisteredResourcesRatio: 0.0 18/03/18 05:03:29 INFO SparkILoop: Created spark context.. Spark context available as sc. 18/03/18 05:03:31 INFO HiveContext: Initializing execution hive, version 1.2.1 18/03/18 05:03:31 INFO ClientWrapper: Inspected Hadoop version: 2.6.0 18/03/18 05:03:31 INFO ClientWrapper: Loaded org.apache.hadoop.hive.shims.Hadoop23Shims for Hadoop version 2.6.0 18/03/18 05:03:32 INFO HiveMetaStore: 0: Opening raw store with implemenation class:org.apache.hadoop.hive.metastore.ObjectStore 18/03/18 05:03:32 INFO ObjectStore: ObjectStore, initialize called 18/03/18 05:03:33 INFO SparkDeploySchedulerBackend: Registered executor NettyRpcEndpointRef(null) (slaver3:58953) with ID 0 18/03/18 05:03:33 INFO BlockManagerMasterEndpoint: Registering block manager slaver3:44986 with 146.2 MB RAM, BlockManagerId(0, slaver3, 44986) 18/03/18 05:03:33 INFO Persistence: Property datanucleus.cache.level2 unknown - will be ignored 18/03/18 05:03:33 INFO Persistence: Property hive.metastore.integral.jdo.pushdown unknown - will be ignored 18/03/18 05:03:33 INFO SparkDeploySchedulerBackend: Registered executor NettyRpcEndpointRef(null) (slaver2:55466) with ID 1 18/03/18 05:03:33 INFO BlockManagerMasterEndpoint: Registering block manager slaver2:57065 with 146.2 MB RAM, BlockManagerId(1, slaver2, 57065) 18/03/18 05:03:33 WARN Connection: BoneCP specified but not present in CLASSPATH (or one of dependencies) 18/03/18 05:03:34 WARN Connection: BoneCP specified but not present in CLASSPATH (or one of dependencies) 18/03/18 05:03:36 INFO ObjectStore: Setting MetaStore object pin classes with hive.metastore.cache.pinobjtypes="Table,StorageDescriptor,SerDeInfo,Partition,Database,Type,FieldSchema,Order" 18/03/18 05:03:38 INFO Datastore: The class "org.apache.hadoop.hive.metastore.model.MFieldSchema" is tagged as "embedded-only" so does not have its own datastore table. 18/03/18 05:03:38 INFO Datastore: The class "org.apache.hadoop.hive.metastore.model.MOrder" is tagged as "embedded-only" so does not have its own datastore table. 18/03/18 05:03:39 INFO Datastore: The class "org.apache.hadoop.hive.metastore.model.MFieldSchema" is tagged as "embedded-only" so does not have its own datastore table. 18/03/18 05:03:39 INFO Datastore: The class "org.apache.hadoop.hive.metastore.model.MOrder" is tagged as "embedded-only" so does not have its own datastore table. 18/03/18 05:03:39 INFO MetaStoreDirectSql: Using direct SQL, underlying DB is DERBY 18/03/18 05:03:39 INFO ObjectStore: Initialized ObjectStore 18/03/18 05:03:40 WARN ObjectStore: Version information not found in metastore. hive.metastore.schema.verification is not enabled so recording the schema version 1.2.0 18/03/18 05:03:40 WARN ObjectStore: Failed to get database default, returning NoSuchObjectException Java HotSpot(TM) Client VM warning: You have loaded library /tmp/libnetty-transport-native-epoll5989457308944360401.so which might have disabled stack guard. The VM will try to fix the stack guard now. It's highly recommended that you fix the library with 'execstack -c <libfile>', or link it with '-z noexecstack'. 18/03/18 05:03:41 INFO HiveMetaStore: Added admin role in metastore 18/03/18 05:03:41 INFO HiveMetaStore: Added public role in metastore 18/03/18 05:03:41 INFO HiveMetaStore: No user is added in admin role, since config is empty 18/03/18 05:03:41 INFO HiveMetaStore: 0: get_all_databases 18/03/18 05:03:41 INFO audit: ugi=root ip=unknown-ip-addr cmd=get_all_databases 18/03/18 05:03:41 INFO HiveMetaStore: 0: get_functions: db=default pat=* 18/03/18 05:03:41 INFO audit: ugi=root ip=unknown-ip-addr cmd=get_functions: db=default pat=* 18/03/18 05:03:41 INFO Datastore: The class "org.apache.hadoop.hive.metastore.model.MResourceUri" is tagged as "embedded-only" so does not have its own datastore table. 18/03/18 05:03:41 INFO SessionState: Created local directory: /tmp/c6bce663-21e6-4d05-a30a-0114773b6959_resources 18/03/18 05:03:41 INFO SessionState: Created HDFS directory: /tmp/hive/root/c6bce663-21e6-4d05-a30a-0114773b6959 18/03/18 05:03:41 INFO SessionState: Created local directory: /tmp/root/c6bce663-21e6-4d05-a30a-0114773b6959 18/03/18 05:03:41 INFO SessionState: Created HDFS directory: /tmp/hive/root/c6bce663-21e6-4d05-a30a-0114773b6959/_tmp_space.db 18/03/18 05:03:42 INFO HiveContext: default warehouse location is /user/hive/warehouse 18/03/18 05:03:42 INFO HiveContext: Initializing HiveMetastoreConnection version 1.2.1 using Spark classes. 18/03/18 05:03:42 INFO ClientWrapper: Inspected Hadoop version: 2.6.0 18/03/18 05:03:42 INFO ClientWrapper: Loaded org.apache.hadoop.hive.shims.Hadoop23Shims for Hadoop version 2.6.0 18/03/18 05:03:43 INFO HiveMetaStore: 0: Opening raw store with implemenation class:org.apache.hadoop.hive.metastore.ObjectStore 18/03/18 05:03:43 INFO ObjectStore: ObjectStore, initialize called 18/03/18 05:03:44 INFO Persistence: Property datanucleus.cache.level2 unknown - will be ignored 18/03/18 05:03:44 INFO Persistence: Property hive.metastore.integral.jdo.pushdown unknown - will be ignored 18/03/18 05:03:44 WARN Connection: BoneCP specified but not present in CLASSPATH (or one of dependencies) 18/03/18 05:03:44 WARN Connection: BoneCP specified but not present in CLASSPATH (or one of dependencies) 18/03/18 05:03:46 INFO ObjectStore: Setting MetaStore object pin classes with hive.metastore.cache.pinobjtypes="Table,StorageDescriptor,SerDeInfo,Partition,Database,Type,FieldSchema,Order" 18/03/18 05:03:47 INFO Datastore: The class "org.apache.hadoop.hive.metastore.model.MFieldSchema" is tagged as "embedded-only" so does not have its own datastore table. 18/03/18 05:03:47 INFO Datastore: The class "org.apache.hadoop.hive.metastore.model.MOrder" is tagged as "embedded-only" so does not have its own datastore table. 18/03/18 05:03:47 INFO Datastore: The class "org.apache.hadoop.hive.metastore.model.MFieldSchema" is tagged as "embedded-only" so does not have its own datastore table. 18/03/18 05:03:47 INFO Datastore: The class "org.apache.hadoop.hive.metastore.model.MOrder" is tagged as "embedded-only" so does not have its own datastore table. 18/03/18 05:03:48 INFO Query: Reading in results for query "org.datanucleus.store.rdbms.query.SQLQuery@0" since the connection used is closing 18/03/18 05:03:48 INFO MetaStoreDirectSql: Using direct SQL, underlying DB is DERBY 18/03/18 05:03:48 INFO ObjectStore: Initialized ObjectStore 18/03/18 05:03:48 INFO HiveMetaStore: Added admin role in metastore 18/03/18 05:03:48 INFO HiveMetaStore: Added public role in metastore 18/03/18 05:03:48 INFO HiveMetaStore: No user is added in admin role, since config is empty 18/03/18 05:03:48 INFO HiveMetaStore: 0: get_all_databases 18/03/18 05:03:48 INFO audit: ugi=root ip=unknown-ip-addr cmd=get_all_databases 18/03/18 05:03:48 INFO HiveMetaStore: 0: get_functions: db=default pat=* 18/03/18 05:03:48 INFO audit: ugi=root ip=unknown-ip-addr cmd=get_functions: db=default pat=* 18/03/18 05:03:48 INFO Datastore: The class "org.apache.hadoop.hive.metastore.model.MResourceUri" is tagged as "embedded-only" so does not have its own datastore table. 18/03/18 05:03:49 INFO SessionState: Created local directory: /tmp/bcc435a7-3ad6-4cd4-9672-b11e5b6c4adc_resources 18/03/18 05:03:49 INFO SessionState: Created HDFS directory: /tmp/hive/root/bcc435a7-3ad6-4cd4-9672-b11e5b6c4adc 18/03/18 05:03:49 INFO SessionState: Created local directory: /tmp/root/bcc435a7-3ad6-4cd4-9672-b11e5b6c4adc 18/03/18 05:03:49 INFO SessionState: Created HDFS directory: /tmp/hive/root/bcc435a7-3ad6-4cd4-9672-b11e5b6c4adc/_tmp_space.db 18/03/18 05:03:49 INFO SparkILoop: Created sql context (with Hive support).. SQL context available as sqlContext. scala>
然后再执行wordcount的例子,如下所示,就可以解决上面出现的问题:
scala> sc.textFile("hdfs://slaver1:9000/hello.txt").flatMap(_.split(" ")).map((_,1)).reduceByKey(_ + _).collect 18/03/18 05:05:16 WARN SizeEstimator: Failed to check whether UseCompressedOops is set; assuming yes 18/03/18 05:05:16 INFO MemoryStore: Block broadcast_0 stored as values in memory (estimated size 82.6 KB, free 82.6 KB) 18/03/18 05:05:17 INFO MemoryStore: Block broadcast_0_piece0 stored as bytes in memory (estimated size 19.3 KB, free 102.0 KB) 18/03/18 05:05:17 INFO BlockManagerInfo: Added broadcast_0_piece0 in memory on 192.168.19.128:40416 (size: 19.3 KB, free: 517.4 MB) 18/03/18 05:05:17 INFO SparkContext: Created broadcast 0 from textFile at <console>:28 18/03/18 05:05:19 INFO FileInputFormat: Total input paths to process : 1 18/03/18 05:05:20 INFO SparkContext: Starting job: collect at <console>:28 18/03/18 05:05:20 INFO DAGScheduler: Registering RDD 3 (map at <console>:28) 18/03/18 05:05:20 INFO DAGScheduler: Got job 0 (collect at <console>:28) with 2 output partitions 18/03/18 05:05:20 INFO DAGScheduler: Final stage: ResultStage 1 (collect at <console>:28) 18/03/18 05:05:20 INFO DAGScheduler: Parents of final stage: List(ShuffleMapStage 0) 18/03/18 05:05:20 INFO DAGScheduler: Missing parents: List(ShuffleMapStage 0) 18/03/18 05:05:20 INFO DAGScheduler: Submitting ShuffleMapStage 0 (MapPartitionsRDD[3] at map at <console>:28), which has no missing parents 18/03/18 05:05:21 INFO MemoryStore: Block broadcast_1 stored as values in memory (estimated size 4.2 KB, free 106.1 KB) 18/03/18 05:05:21 INFO MemoryStore: Block broadcast_1_piece0 stored as bytes in memory (estimated size 2.3 KB, free 108.4 KB) 18/03/18 05:05:21 INFO BlockManagerInfo: Added broadcast_1_piece0 in memory on 192.168.19.128:40416 (size: 2.3 KB, free: 517.4 MB) 18/03/18 05:05:21 INFO SparkContext: Created broadcast 1 from broadcast at DAGScheduler.scala:1006 18/03/18 05:05:21 INFO DAGScheduler: Submitting 2 missing tasks from ShuffleMapStage 0 (MapPartitionsRDD[3] at map at <console>:28) 18/03/18 05:05:21 INFO TaskSchedulerImpl: Adding task set 0.0 with 2 tasks 18/03/18 05:05:21 INFO TaskSetManager: Starting task 0.0 in stage 0.0 (TID 0, slaver3, partition 0,NODE_LOCAL, 2120 bytes) 18/03/18 05:05:22 INFO BlockManagerInfo: Added broadcast_1_piece0 in memory on slaver3:44986 (size: 2.3 KB, free: 146.2 MB) 18/03/18 05:05:22 INFO BlockManagerInfo: Added broadcast_0_piece0 in memory on slaver3:44986 (size: 19.3 KB, free: 146.2 MB) 18/03/18 05:05:24 INFO TaskSetManager: Starting task 1.0 in stage 0.0 (TID 1, slaver3, partition 1,NODE_LOCAL, 2120 bytes) 18/03/18 05:05:24 INFO TaskSetManager: Finished task 0.0 in stage 0.0 (TID 0) in 3056 ms on slaver3 (1/2) 18/03/18 05:05:24 INFO TaskSetManager: Finished task 1.0 in stage 0.0 (TID 1) in 199 ms on slaver3 (2/2) 18/03/18 05:05:24 INFO DAGScheduler: ShuffleMapStage 0 (map at <console>:28) finished in 3.242 s 18/03/18 05:05:24 INFO DAGScheduler: looking for newly runnable stages 18/03/18 05:05:24 INFO DAGScheduler: running: Set() 18/03/18 05:05:24 INFO DAGScheduler: waiting: Set(ResultStage 1) 18/03/18 05:05:24 INFO TaskSchedulerImpl: Removed TaskSet 0.0, whose tasks have all completed, from pool 18/03/18 05:05:24 INFO DAGScheduler: failed: Set() 18/03/18 05:05:24 INFO DAGScheduler: Submitting ResultStage 1 (ShuffledRDD[4] at reduceByKey at <console>:28), which has no missing parents 18/03/18 05:05:24 INFO MemoryStore: Block broadcast_2 stored as values in memory (estimated size 2.6 KB, free 111.0 KB) 18/03/18 05:05:24 INFO MemoryStore: Block broadcast_2_piece0 stored as bytes in memory (estimated size 1589.0 B, free 112.6 KB) 18/03/18 05:05:24 INFO BlockManagerInfo: Added broadcast_2_piece0 in memory on 192.168.19.128:40416 (size: 1589.0 B, free: 517.4 MB) 18/03/18 05:05:24 INFO SparkContext: Created broadcast 2 from broadcast at DAGScheduler.scala:1006 18/03/18 05:05:24 INFO DAGScheduler: Submitting 2 missing tasks from ResultStage 1 (ShuffledRDD[4] at reduceByKey at <console>:28) 18/03/18 05:05:24 INFO TaskSchedulerImpl: Adding task set 1.0 with 2 tasks 18/03/18 05:05:24 INFO TaskSetManager: Starting task 0.0 in stage 1.0 (TID 2, slaver3, partition 0,NODE_LOCAL, 1894 bytes) 18/03/18 05:05:24 INFO BlockManagerInfo: Added broadcast_2_piece0 in memory on slaver3:44986 (size: 1589.0 B, free: 146.2 MB) 18/03/18 05:05:24 INFO MapOutputTrackerMasterEndpoint: Asked to send map output locations for shuffle 0 to slaver3:58953 18/03/18 05:05:24 INFO MapOutputTrackerMaster: Size of output statuses for shuffle 0 is 147 bytes 18/03/18 05:05:24 INFO TaskSetManager: Starting task 1.0 in stage 1.0 (TID 3, slaver3, partition 1,NODE_LOCAL, 1894 bytes) 18/03/18 05:05:24 INFO TaskSetManager: Finished task 0.0 in stage 1.0 (TID 2) in 131 ms on slaver3 (1/2) 18/03/18 05:05:24 INFO DAGScheduler: ResultStage 1 (collect at <console>:28) finished in 0.153 s 18/03/18 05:05:24 INFO TaskSetManager: Finished task 1.0 in stage 1.0 (TID 3) in 30 ms on slaver3 (2/2) 18/03/18 05:05:24 INFO TaskSchedulerImpl: Removed TaskSet 1.0, whose tasks have all completed, from pool 18/03/18 05:05:24 INFO DAGScheduler: Job 0 finished: collect at <console>:28, took 3.883545 s res0: Array[(String, Int)] = Array((hive,1), (hello,7), (sqoop,1), (spark,1), (hadoop,1), (storm,1), (hbase,1), (biexiansheng,1))
完结.......