Spark的Local运行模式部署实战案例
Spark的Local运行模式部署实战案例
作者：尹正杰
版权声明：原创作品，谢绝转载！否则将追究法律责任。
一.部署spark
1>.下载spark二进制安装包
下载Spark地址: http://spark.apache.org/downloads.html
2>.解压spark到指定路径
[root@hadoop101.yinzhengjie.org.cn ~]# ll total 227752 -rw-r--r-- 1 root root 233215067 Jun 27 23:33 spark-2.4.6-bin-hadoop2.7.tgz [root@hadoop101.yinzhengjie.org.cn ~]# [root@hadoop101.yinzhengjie.org.cn ~]# tar -zxf spark-2.4.6-bin-hadoop2.7.tgz -C /yinzhengjie/softwares/ [root@hadoop101.yinzhengjie.org.cn ~]# [root@hadoop101.yinzhengjie.org.cn ~]# ll /yinzhengjie/softwares/spark-2.4.6-bin-hadoop2.7/ total 104 drwxr-xr-x 2 yinzhengjie yinzhengjie 4096 May 30 08:02 bin drwxr-xr-x 2 yinzhengjie yinzhengjie 230 May 30 08:02 conf drwxr-xr-x 5 yinzhengjie yinzhengjie 50 May 30 08:02 data drwxr-xr-x 4 yinzhengjie yinzhengjie 29 May 30 08:02 examples drwxr-xr-x 2 yinzhengjie yinzhengjie 12288 May 30 08:02 jars drwxr-xr-x 4 yinzhengjie yinzhengjie 38 May 30 08:02 kubernetes -rw-r--r-- 1 yinzhengjie yinzhengjie 21371 May 30 08:02 LICENSE drwxr-xr-x 2 yinzhengjie yinzhengjie 4096 May 30 08:02 licenses -rw-r--r-- 1 yinzhengjie yinzhengjie 42919 May 30 08:02 NOTICE drwxr-xr-x 9 yinzhengjie yinzhengjie 311 May 30 08:02 python drwxr-xr-x 3 yinzhengjie yinzhengjie 17 May 30 08:02 R -rw-r--r-- 1 yinzhengjie yinzhengjie 3756 May 30 08:02 README.md -rw-r--r-- 1 yinzhengjie yinzhengjie 187 May 30 08:02 RELEASE drwxr-xr-x 2 yinzhengjie yinzhengjie 4096 May 30 08:02 sbin drwxr-xr-x 2 yinzhengjie yinzhengjie 42 May 30 08:02 yarn [root@hadoop101.yinzhengjie.org.cn ~]#
3>.创建符号链接
[root@hadoop101.yinzhengjie.org.cn ~]# ll /yinzhengjie/softwares/spark-2.4.6-bin-hadoop2.7/ total 104 drwxr-xr-x 2 yinzhengjie yinzhengjie 4096 May 30 08:02 bin drwxr-xr-x 2 yinzhengjie yinzhengjie 230 May 30 08:02 conf drwxr-xr-x 5 yinzhengjie yinzhengjie 50 May 30 08:02 data drwxr-xr-x 4 yinzhengjie yinzhengjie 29 May 30 08:02 examples drwxr-xr-x 2 yinzhengjie yinzhengjie 12288 May 30 08:02 jars drwxr-xr-x 4 yinzhengjie yinzhengjie 38 May 30 08:02 kubernetes -rw-r--r-- 1 yinzhengjie yinzhengjie 21371 May 30 08:02 LICENSE drwxr-xr-x 2 yinzhengjie yinzhengjie 4096 May 30 08:02 licenses -rw-r--r-- 1 yinzhengjie yinzhengjie 42919 May 30 08:02 NOTICE drwxr-xr-x 9 yinzhengjie yinzhengjie 311 May 30 08:02 python drwxr-xr-x 3 yinzhengjie yinzhengjie 17 May 30 08:02 R -rw-r--r-- 1 yinzhengjie yinzhengjie 3756 May 30 08:02 README.md -rw-r--r-- 1 yinzhengjie yinzhengjie 187 May 30 08:02 RELEASE drwxr-xr-x 2 yinzhengjie yinzhengjie 4096 May 30 08:02 sbin drwxr-xr-x 2 yinzhengjie yinzhengjie 42 May 30 08:02 yarn [root@hadoop101.yinzhengjie.org.cn ~]# [root@hadoop101.yinzhengjie.org.cn ~]# ln -sv /yinzhengjie/softwares/spark-2.4.6-bin-hadoop2.7/ /yinzhengjie/softwares/spark ‘/yinzhengjie/softwares/spark’ -> ‘/yinzhengjie/softwares/spark-2.4.6-bin-hadoop2.7/’ [root@hadoop101.yinzhengjie.org.cn ~]# [root@hadoop101.yinzhengjie.org.cn ~]# ll /yinzhengjie/softwares/spark lrwxrwxrwx 1 root root 49 Jun 28 02:24 /yinzhengjie/softwares/spark -> /yinzhengjie/softwares/spark-2.4.6-bin-hadoop2.7/ [root@hadoop101.yinzhengjie.org.cn ~]# [root@hadoop101.yinzhengjie.org.cn ~]# [root@hadoop101.yinzhengjie.org.cn ~]# ll /yinzhengjie/softwares/spark/ total 104 drwxr-xr-x 2 yinzhengjie yinzhengjie 4096 May 30 08:02 bin drwxr-xr-x 2 yinzhengjie yinzhengjie 230 May 30 08:02 conf drwxr-xr-x 5 yinzhengjie yinzhengjie 50 May 30 08:02 data drwxr-xr-x 4 yinzhengjie yinzhengjie 29 May 30 08:02 examples drwxr-xr-x 2 yinzhengjie yinzhengjie 12288 May 30 08:02 jars drwxr-xr-x 4 yinzhengjie yinzhengjie 38 May 30 08:02 kubernetes -rw-r--r-- 1 yinzhengjie yinzhengjie 21371 May 30 08:02 LICENSE drwxr-xr-x 2 yinzhengjie yinzhengjie 4096 May 30 08:02 licenses -rw-r--r-- 1 yinzhengjie yinzhengjie 42919 May 30 08:02 NOTICE drwxr-xr-x 9 yinzhengjie yinzhengjie 311 May 30 08:02 python drwxr-xr-x 3 yinzhengjie yinzhengjie 17 May 30 08:02 R -rw-r--r-- 1 yinzhengjie yinzhengjie 3756 May 30 08:02 README.md -rw-r--r-- 1 yinzhengjie yinzhengjie 187 May 30 08:02 RELEASE drwxr-xr-x 2 yinzhengjie yinzhengjie 4096 May 30 08:02 sbin drwxr-xr-x 2 yinzhengjie yinzhengjie 42 May 30 08:02 yarn [root@hadoop101.yinzhengjie.org.cn ~]# [root@hadoop101.yinzhengjie.org.cn ~]#
4>.配置spark环境变量
[root@hadoop101.yinzhengjie.org.cn ~]# vim /etc/profile.d/spark.sh [root@hadoop101.yinzhengjie.org.cn ~]# [root@hadoop101.yinzhengjie.org.cn ~]# cat /etc/profile.d/spark.sh #Add ${SPARK_HOME} by yinzhengjie SPARK_HOME=/yinzhengjie/softwares/spark PATH=$PATH:${SPARK_HOME}/bin:${SPARK_HOME}/sbin [root@hadoop101.yinzhengjie.org.cn ~]#
5>.使环境变量生效
[root@hadoop101.yinzhengjie.org.cn ~]# source /etc/profile.d/spark.sh #使用"source"命令使咱们自定义的环境变量生效 [root@hadoop101.yinzhengjie.org.cn ~]# [root@hadoop101.yinzhengjie.org.cn ~]# spark #输入spark后连续按两下"tab"键如果有自动补齐功能说明环境变量配置生效啦~ spark-class spark-config.sh spark-daemon.sh spark-daemons.sh sparkR spark-shell spark-sql spark-submit [root@hadoop101.yinzhengjie.org.cn ~]# spark
6>.启动spark-shell
[root@hadoop101.yinzhengjie.org.cn ~]# spark-shell 20/06/28 02:50:47 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties Setting default log level to "WARN". To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel). Spark context Web UI available at http://hadoop101.yinzhengjie.org.cn:4040 Spark context available as 'sc' (master = local[*], app id = local-1593283854727). Spark session available as 'spark'. Welcome to ____ __ / __/__ ___ _____/ /__ _\ \/ _ \/ _ `/ __/ '_/ /___/ .__/\_,_/_/ /_/\_\ version 2.4.6 /_/ Using Scala version 2.11.12 (Java HotSpot(TM) 64-Bit Server VM, Java 1.8.0_201) Type in expressions to have them evaluated. Type :help for more information. scala>
二.计算圆周率案例
1>.基本语法
语法格式如下: [root@hadoop101.yinzhengjie.org.cn ~]# spark-submit \ > --class <main-class> \ > --master <master-url> \ > --deploy-mode <deploy-mode> \ > --conf <key>=<value> \ > ...(other options) > <application-jar> \ > [application-arguments] 相关参数说明： --class: 指定你的应用的启动类，如:"org.apache.spark.examples.SparkPi" --master: 指定Master的地址，默认为Local --deploy-mode: 指定是否发布你的驱动到worker节点(cluster),或者作为一个本地客户端(默认值: client) --conf: 指定任意的Spark配置属性， 格式key=value. 如果值包含空格，可以加引号“key=value” application-jar: 指定打包好的应用jar,包含依赖. 这个URL在集群中全局可见。 比如hdfs://yinzhengjie/hbase，如果是file://path， 那么所有的节点的path都包含同样的jar application-arguments: 传给main()方法的参数 --executor-memory 1G: 指定每个executor可用内存为1G --total-executor-cores 2: 指定每个executor使用的cup核数为2个
2>.官方求PI案例
[root@hadoop101.yinzhengjie.org.cn ~]# spark-submit \ > --class org.apache.spark.examples.SparkPi \ > --executor-memory 1G \ > --total-executor-cores 2 \ > /yinzhengjie/softwares/spark/examples/jars/spark-examples_2.11-2.4.6.jar \ > 100 20/06/28 02:36:34 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties 20/06/28 02:36:34 INFO SparkContext: Running Spark version 2.4.6 20/06/28 02:36:34 INFO SparkContext: Submitted application: Spark Pi 20/06/28 02:36:34 INFO SecurityManager: Changing view acls to: root 20/06/28 02:36:34 INFO SecurityManager: Changing modify acls to: root 20/06/28 02:36:34 INFO SecurityManager: Changing view acls groups to: 20/06/28 02:36:34 INFO SecurityManager: Changing modify acls groups to: 20/06/28 02:36:34 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(root); groups with view permissions: Set(); users with modify permissions: Set(root); groups with modify permissions: Set() 20/06/28 02:36:35 INFO Utils: Successfully started service 'sparkDriver' on port 26502. 20/06/28 02:36:35 INFO SparkEnv: Registering MapOutputTracker 20/06/28 02:36:35 INFO SparkEnv: Registering BlockManagerMaster 20/06/28 02:36:35 INFO BlockManagerMasterEndpoint: Using org.apache.spark.storage.DefaultTopologyMapper for getting topology information 20/06/28 02:36:35 INFO BlockManagerMasterEndpoint: BlockManagerMasterEndpoint up 20/06/28 02:36:35 INFO DiskBlockManager: Created local directory at /tmp/blockmgr-a8219297-dd0e-42fe-b3f9-1c5652a7807e 20/06/28 02:36:35 INFO MemoryStore: MemoryStore started with capacity 366.3 MB 20/06/28 02:36:35 INFO SparkEnv: Registering OutputCommitCoordinator 20/06/28 02:36:35 INFO Utils: Successfully started service 'SparkUI' on port 4040. 20/06/28 02:36:35 INFO SparkUI: Bound SparkUI to 0.0.0.0, and started at http://hadoop101.yinzhengjie.org.cn:4040 20/06/28 02:36:35 INFO SparkContext: Added JAR file:/yinzhengjie/softwares/spark/examples/jars/spark-examples_2.11-2.4.6.jar at spark://hadoop101.yinzhengjie.org.cn:26502/jars/spark-examples_2.11-2.4.6.jar with timestamp 1593282995928 20/06/28 02:36:35 INFO Executor: Starting executor ID driver on host localhost 20/06/28 02:36:36 INFO Utils: Successfully started service 'org.apache.spark.network.netty.NettyBlockTransferService' on port 29014. 20/06/28 02:36:36 INFO NettyBlockTransferService: Server created on hadoop101.yinzhengjie.org.cn:29014 20/06/28 02:36:36 INFO BlockManager: Using org.apache.spark.storage.RandomBlockReplicationPolicy for block replication policy 20/06/28 02:36:36 INFO BlockManagerMaster: Registering BlockManager BlockManagerId(driver, hadoop101.yinzhengjie.org.cn, 29014, None) 20/06/28 02:36:36 INFO BlockManagerMasterEndpoint: Registering block manager hadoop101.yinzhengjie.org.cn:29014 with 366.3 MB RAM, BlockManagerId(driver, hadoop101.yinzhengjie.org.cn, 29014, None) 20/06/28 02:36:36 INFO BlockManagerMaster: Registered BlockManager BlockManagerId(driver, hadoop101.yinzhengjie.org.cn, 29014, None) 20/06/28 02:36:36 INFO BlockManager: Initialized BlockManager: BlockManagerId(driver, hadoop101.yinzhengjie.org.cn, 29014, None) 20/06/28 02:36:37 INFO SparkContext: Starting job: reduce at SparkPi.scala:38 20/06/28 02:36:37 INFO DAGScheduler: Got job 0 (reduce at SparkPi.scala:38) with 100 output partitions 20/06/28 02:36:37 INFO DAGScheduler: Final stage: ResultStage 0 (reduce at SparkPi.scala:38) 20/06/28 02:36:37 INFO DAGScheduler: Parents of final stage: List() 20/06/28 02:36:37 INFO DAGScheduler: Missing parents: List() 20/06/28 02:36:37 INFO DAGScheduler: Submitting ResultStage 0 (MapPartitionsRDD[1] at map at SparkPi.scala:34), which has no missing parents 20/06/28 02:36:37 INFO MemoryStore: Block broadcast_0 stored as values in memory (estimated size 2.0 KB, free 366.3 MB) 20/06/28 02:36:38 INFO MemoryStore: Block broadcast_0_piece0 stored as bytes in memory (estimated size 1381.0 B, free 366.3 MB) 20/06/28 02:36:38 INFO BlockManagerInfo: Added broadcast_0_piece0 in memory on hadoop101.yinzhengjie.org.cn:29014 (size: 1381.0 B, free: 366.3 MB) 20/06/28 02:36:38 INFO SparkContext: Created broadcast 0 from broadcast at DAGScheduler.scala:1163 20/06/28 02:36:38 INFO DAGScheduler: Submitting 100 missing tasks from ResultStage 0 (MapPartitionsRDD[1] at map at SparkPi.scala:34) (first 15 tasks are for partitions Vector(0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14)) 20/06/28 02:36:38 INFO TaskSchedulerImpl: Adding task set 0.0 with 100 tasks 20/06/28 02:36:38 INFO TaskSetManager: Starting task 0.0 in stage 0.0 (TID 0, localhost, executor driver, partition 0, PROCESS_LOCAL, 7866 bytes) 20/06/28 02:36:38 INFO TaskSetManager: Starting task 1.0 in stage 0.0 (TID 1, localhost, executor driver, partition 1, PROCESS_LOCAL, 7866 bytes) 20/06/28 02:36:38 INFO Executor: Running task 0.0 in stage 0.0 (TID 0) 20/06/28 02:36:38 INFO Executor: Fetching spark://hadoop101.yinzhengjie.org.cn:26502/jars/spark-examples_2.11-2.4.6.jar with timestamp 1593282995928 20/06/28 02:36:38 INFO Executor: Running task 1.0 in stage 0.0 (TID 1) 20/06/28 02:36:38 INFO TransportClientFactory: Successfully created connection to hadoop101.yinzhengjie.org.cn/172.200.4.101:26502 after 33 ms (0 ms spent in bootstraps) 20/06/28 02:36:38 INFO Utils: Fetching spark://hadoop101.yinzhengjie.org.cn:26502/jars/spark-examples_2.11-2.4.6.jar to /tmp/spark-7e4020d1-9318-4c3e-9740-95e1a389bbdf/userFiles-6717e654-ec92-4cd8-9ebc-e4a9313d5e12/fetchFileTemp8903238475767909399.tmp 20/06/28 02:36:38 INFO Executor: Adding file:/tmp/spark-7e4020d1-9318-4c3e-9740-95e1a389bbdf/userFiles-6717e654-ec92-4cd8-9ebc-e4a9313d5e12/spark-examples_2.11-2.4.6.jar to class loader 20/06/28 02:36:38 INFO Executor: Finished task 0.0 in stage 0.0 (TID 0). 824 bytes result sent to driver 20/06/28 02:36:38 INFO TaskSetManager: Starting task 2.0 in stage 0.0 (TID 2, localhost, executor driver, partition 2, PROCESS_LOCAL, 7866 bytes) 20/06/28 02:36:38 INFO Executor: Running task 2.0 in stage 0.0 (TID 2) 20/06/28 02:36:38 INFO Executor: Finished task 1.0 in stage 0.0 (TID 1). 824 bytes result sent to driver 20/06/28 02:36:38 INFO TaskSetManager: Starting task 3.0 in stage 0.0 (TID 3, localhost, executor driver, partition 3, PROCESS_LOCAL, 7866 bytes) 20/06/28 02:36:38 INFO Executor: Finished task 2.0 in stage 0.0 (TID 2). 824 bytes result sent to driver 20/06/28 02:36:38 INFO TaskSetManager: Starting task 4.0 in stage 0.0 (TID 4, localhost, executor driver, partition 4, PROCESS_LOCAL, 7866 bytes) 20/06/28 02:36:38 INFO Executor: Running task 4.0 in stage 0.0 (TID 4) 20/06/28 02:36:38 INFO Executor: Running task 3.0 in stage 0.0 (TID 3) 20/06/28 02:36:38 INFO TaskSetManager: Finished task 1.0 in stage 0.0 (TID 1) in 317 ms on localhost (executor driver) (1/100) 20/06/28 02:36:38 INFO Executor: Finished task 4.0 in stage 0.0 (TID 4). 824 bytes result sent to driver 20/06/28 02:36:38 INFO Executor: Finished task 3.0 in stage 0.0 (TID 3). 824 bytes result sent to driver 20/06/28 02:36:38 INFO TaskSetManager: Starting task 5.0 in stage 0.0 (TID 5, localhost, executor driver, partition 5, PROCESS_LOCAL, 7866 bytes) 20/06/28 02:36:38 INFO TaskSetManager: Starting task 6.0 in stage 0.0 (TID 6, localhost, executor driver, partition 6, PROCESS_LOCAL, 7866 bytes) 20/06/28 02:36:38 INFO TaskSetManager: Finished task 0.0 in stage 0.0 (TID 0) in 360 ms on localhost (executor driver) (2/100) 20/06/28 02:36:38 INFO TaskSetManager: Finished task 3.0 in stage 0.0 (TID 3) in 41 ms on localhost (executor driver) (3/100) 20/06/28 02:36:38 INFO Executor: Running task 5.0 in stage 0.0 (TID 5) 20/06/28 02:36:38 INFO TaskSetManager: Finished task 2.0 in stage 0.0 (TID 2) in 58 ms on localhost (executor driver) (4/100) 20/06/28 02:36:38 INFO Executor: Finished task 5.0 in stage 0.0 (TID 5). 824 bytes result sent to driver 20/06/28 02:36:38 INFO Executor: Running task 6.0 in stage 0.0 (TID 6) 20/06/28 02:36:38 INFO Executor: Finished task 6.0 in stage 0.0 (TID 6). 824 bytes result sent to driver 20/06/28 02:36:38 INFO TaskSetManager: Starting task 7.0 in stage 0.0 (TID 7, localhost, executor driver, partition 7, PROCESS_LOCAL, 7866 bytes) 20/06/28 02:36:38 INFO TaskSetManager: Starting task 8.0 in stage 0.0 (TID 8, localhost, executor driver, partition 8, PROCESS_LOCAL, 7866 bytes) 20/06/28 02:36:38 INFO TaskSetManager: Finished task 4.0 in stage 0.0 (TID 4) in 54 ms on localhost (executor driver) (5/100) 20/06/28 02:36:38 INFO TaskSetManager: Finished task 5.0 in stage 0.0 (TID 5) in 24 ms on localhost (executor driver) (6/100) 20/06/28 02:36:38 INFO Executor: Running task 7.0 in stage 0.0 (TID 7) 20/06/28 02:36:38 INFO Executor: Finished task 7.0 in stage 0.0 (TID 7). 824 bytes result sent to driver 20/06/28 02:36:38 INFO TaskSetManager: Starting task 9.0 in stage 0.0 (TID 9, localhost, executor driver, partition 9, PROCESS_LOCAL, 7866 bytes) 20/06/28 02:36:38 INFO TaskSetManager: Finished task 7.0 in stage 0.0 (TID 7) in 18 ms on localhost (executor driver) (7/100) 20/06/28 02:36:38 INFO Executor: Running task 8.0 in stage 0.0 (TID 8) 20/06/28 02:36:38 INFO TaskSetManager: Finished task 6.0 in stage 0.0 (TID 6) in 45 ms on localhost (executor driver) (8/100) 20/06/28 02:36:38 INFO Executor: Finished task 8.0 in stage 0.0 (TID 8). 824 bytes result sent to driver 20/06/28 02:36:38 INFO TaskSetManager: Starting task 10.0 in stage 0.0 (TID 10, localhost, executor driver, partition 10, PROCESS_LOCAL, 7866 bytes) 20/06/28 02:36:38 INFO TaskSetManager: Finished task 8.0 in stage 0.0 (TID 8) in 28 ms on localhost (executor driver) (9/100) 20/06/28 02:36:38 INFO Executor: Running task 10.0 in stage 0.0 (TID 10) 20/06/28 02:36:38 INFO Executor: Running task 9.0 in stage 0.0 (TID 9) 20/06/28 02:36:38 INFO Executor: Finished task 10.0 in stage 0.0 (TID 10). 824 bytes result sent to driver 20/06/28 02:36:38 INFO TaskSetManager: Starting task 11.0 in stage 0.0 (TID 11, localhost, executor driver, partition 11, PROCESS_LOCAL, 7866 bytes) 20/06/28 02:36:38 INFO TaskSetManager: Finished task 10.0 in stage 0.0 (TID 10) in 18 ms on localhost (executor driver) (10/100) 20/06/28 02:36:38 INFO Executor: Running task 11.0 in stage 0.0 (TID 11) 20/06/28 02:36:38 INFO Executor: Finished task 9.0 in stage 0.0 (TID 9). 824 bytes result sent to driver 20/06/28 02:36:38 INFO Executor: Finished task 11.0 in stage 0.0 (TID 11). 781 bytes result sent to driver 20/06/28 02:36:38 INFO TaskSetManager: Starting task 12.0 in stage 0.0 (TID 12, localhost, executor driver, partition 12, PROCESS_LOCAL, 7866 bytes) 20/06/28 02:36:38 INFO TaskSetManager: Finished task 9.0 in stage 0.0 (TID 9) in 41 ms on localhost (executor driver) (11/100) 20/06/28 02:36:38 INFO TaskSetManager: Starting task 13.0 in stage 0.0 (TID 13, localhost, executor driver, partition 13, PROCESS_LOCAL, 7866 bytes) 20/06/28 02:36:38 INFO TaskSetManager: Finished task 11.0 in stage 0.0 (TID 11) in 19 ms on localhost (executor driver) (12/100) 20/06/28 02:36:38 INFO Executor: Running task 13.0 in stage 0.0 (TID 13) 20/06/28 02:36:38 INFO Executor: Finished task 13.0 in stage 0.0 (TID 13). 824 bytes result sent to driver 20/06/28 02:36:38 INFO TaskSetManager: Starting task 14.0 in stage 0.0 (TID 14, localhost, executor driver, partition 14, PROCESS_LOCAL, 7866 bytes) 20/06/28 02:36:38 INFO TaskSetManager: Finished task 13.0 in stage 0.0 (TID 13) in 12 ms on localhost (executor driver) (13/100) 20/06/28 02:36:38 INFO Executor: Running task 12.0 in stage 0.0 (TID 12) 20/06/28 02:36:38 INFO Executor: Running task 14.0 in stage 0.0 (TID 14) 20/06/28 02:36:38 INFO Executor: Finished task 12.0 in stage 0.0 (TID 12). 824 bytes result sent to driver 20/06/28 02:36:38 INFO TaskSetManager: Starting task 15.0 in stage 0.0 (TID 15, localhost, executor driver, partition 15, PROCESS_LOCAL, 7866 bytes) 20/06/28 02:36:38 INFO Executor: Running task 15.0 in stage 0.0 (TID 15) 20/06/28 02:36:38 INFO TaskSetManager: Finished task 12.0 in stage 0.0 (TID 12) in 39 ms on localhost (executor driver) (14/100) 20/06/28 02:36:38 INFO Executor: Finished task 14.0 in stage 0.0 (TID 14). 781 bytes result sent to driver 20/06/28 02:36:38 INFO TaskSetManager: Starting task 16.0 in stage 0.0 (TID 16, localhost, executor driver, partition 16, PROCESS_LOCAL, 7866 bytes) 20/06/28 02:36:38 INFO TaskSetManager: Finished task 14.0 in stage 0.0 (TID 14) in 32 ms on localhost (executor driver) (15/100) 20/06/28 02:36:38 INFO Executor: Running task 16.0 in stage 0.0 (TID 16) 20/06/28 02:36:38 INFO Executor: Finished task 16.0 in stage 0.0 (TID 16). 781 bytes result sent to driver 20/06/28 02:36:38 INFO TaskSetManager: Starting task 17.0 in stage 0.0 (TID 17, localhost, executor driver, partition 17, PROCESS_LOCAL, 7866 bytes) 20/06/28 02:36:38 INFO Executor: Running task 17.0 in stage 0.0 (TID 17) 20/06/28 02:36:38 INFO Executor: Finished task 17.0 in stage 0.0 (TID 17). 781 bytes result sent to driver 20/06/28 02:36:38 INFO TaskSetManager: Starting task 18.0 in stage 0.0 (TID 18, localhost, executor driver, partition 18, PROCESS_LOCAL, 7866 bytes) 20/06/28 02:36:38 INFO Executor: Running task 18.0 in stage 0.0 (TID 18) 20/06/28 02:36:38 INFO TaskSetManager: Finished task 16.0 in stage 0.0 (TID 16) in 89 ms on localhost (executor driver) (16/100) 20/06/28 02:36:38 INFO TaskSetManager: Finished task 17.0 in stage 0.0 (TID 17) in 10 ms on localhost (executor driver) (17/100) 20/06/28 02:36:38 INFO Executor: Finished task 15.0 in stage 0.0 (TID 15). 867 bytes result sent to driver 20/06/28 02:36:38 INFO TaskSetManager: Starting task 19.0 in stage 0.0 (TID 19, localhost, executor driver, partition 19, PROCESS_LOCAL, 7866 bytes) 20/06/28 02:36:38 INFO TaskSetManager: Finished task 15.0 in stage 0.0 (TID 15) in 113 ms on localhost (executor driver) (18/100) 20/06/28 02:36:38 INFO Executor: Running task 19.0 in stage 0.0 (TID 19) 20/06/28 02:36:38 INFO Executor: Finished task 18.0 in stage 0.0 (TID 18). 824 bytes result sent to driver 20/06/28 02:36:38 INFO TaskSetManager: Starting task 20.0 in stage 0.0 (TID 20, localhost, executor driver, partition 20, PROCESS_LOCAL, 7866 bytes) 20/06/28 02:36:38 INFO TaskSetManager: Finished task 18.0 in stage 0.0 (TID 18) in 53 ms on localhost (executor driver) (19/100) 20/06/28 02:36:38 INFO Executor: Running task 20.0 in stage 0.0 (TID 20) 20/06/28 02:36:38 INFO Executor: Finished task 19.0 in stage 0.0 (TID 19). 824 bytes result sent to driver 20/06/28 02:36:38 INFO TaskSetManager: Starting task 21.0 in stage 0.0 (TID 21, localhost, executor driver, partition 21, PROCESS_LOCAL, 7866 bytes) 20/06/28 02:36:38 INFO Executor: Running task 21.0 in stage 0.0 (TID 21) 20/06/28 02:36:38 INFO TaskSetManager: Finished task 19.0 in stage 0.0 (TID 19) in 118 ms on localhost (executor driver) (20/100) 20/06/28 02:36:38 INFO Executor: Finished task 20.0 in stage 0.0 (TID 20). 781 bytes result sent to driver 20/06/28 02:36:38 INFO TaskSetManager: Starting task 22.0 in stage 0.0 (TID 22, localhost, executor driver, partition 22, PROCESS_LOCAL, 7866 bytes) 20/06/28 02:36:38 INFO Executor: Running task 22.0 in stage 0.0 (TID 22) 20/06/28 02:36:38 INFO TaskSetManager: Finished task 20.0 in stage 0.0 (TID 20) in 126 ms on localhost (executor driver) (21/100) 20/06/28 02:36:38 INFO Executor: Finished task 21.0 in stage 0.0 (TID 21). 824 bytes result sent to driver 20/06/28 02:36:38 INFO TaskSetManager: Starting task 23.0 in stage 0.0 (TID 23, localhost, executor driver, partition 23, PROCESS_LOCAL, 7866 bytes) 20/06/28 02:36:38 INFO TaskSetManager: Finished task 21.0 in stage 0.0 (TID 21) in 62 ms on localhost (executor driver) (22/100) 20/06/28 02:36:38 INFO Executor: Running task 23.0 in stage 0.0 (TID 23) 20/06/28 02:36:38 INFO Executor: Finished task 22.0 in stage 0.0 (TID 22). 781 bytes result sent to driver 20/06/28 02:36:38 INFO TaskSetManager: Starting task 24.0 in stage 0.0 (TID 24, localhost, executor driver, partition 24, PROCESS_LOCAL, 7866 bytes) 20/06/28 02:36:38 INFO TaskSetManager: Finished task 22.0 in stage 0.0 (TID 22) in 43 ms on localhost (executor driver) (23/100) 20/06/28 02:36:38 INFO Executor: Running task 24.0 in stage 0.0 (TID 24) 20/06/28 02:36:38 INFO Executor: Finished task 23.0 in stage 0.0 (TID 23). 867 bytes result sent to driver 20/06/28 02:36:38 INFO TaskSetManager: Starting task 25.0 in stage 0.0 (TID 25, localhost, executor driver, partition 25, PROCESS_LOCAL, 7866 bytes) 20/06/28 02:36:38 INFO TaskSetManager: Finished task 23.0 in stage 0.0 (TID 23) in 82 ms on localhost (executor driver) (24/100) 20/06/28 02:36:38 INFO Executor: Running task 25.0 in stage 0.0 (TID 25) 20/06/28 02:36:38 INFO Executor: Finished task 25.0 in stage 0.0 (TID 25). 824 bytes result sent to driver 20/06/28 02:36:38 INFO TaskSetManager: Starting task 26.0 in stage 0.0 (TID 26, localhost, executor driver, partition 26, PROCESS_LOCAL, 7866 bytes) 20/06/28 02:36:38 INFO TaskSetManager: Finished task 25.0 in stage 0.0 (TID 25) in 49 ms on localhost (executor driver) (25/100) 20/06/28 02:36:38 INFO Executor: Finished task 24.0 in stage 0.0 (TID 24). 824 bytes result sent to driver 20/06/28 02:36:38 INFO TaskSetManager: Starting task 27.0 in stage 0.0 (TID 27, localhost, executor driver, partition 27, PROCESS_LOCAL, 7866 bytes) 20/06/28 02:36:38 INFO Executor: Running task 27.0 in stage 0.0 (TID 27) 20/06/28 02:36:38 INFO TaskSetManager: Finished task 24.0 in stage 0.0 (TID 24) in 97 ms on localhost (executor driver) (26/100) 20/06/28 02:36:38 INFO Executor: Running task 26.0 in stage 0.0 (TID 26) 20/06/28 02:36:39 INFO Executor: Finished task 27.0 in stage 0.0 (TID 27). 781 bytes result sent to driver 20/06/28 02:36:39 INFO TaskSetManager: Starting task 28.0 in stage 0.0 (TID 28, localhost, executor driver, partition 28, PROCESS_LOCAL, 7866 bytes) 20/06/28 02:36:39 INFO Executor: Finished task 26.0 in stage 0.0 (TID 26). 824 bytes result sent to driver 20/06/28 02:36:39 INFO TaskSetManager: Starting task 29.0 in stage 0.0 (TID 29, localhost, executor driver, partition 29, PROCESS_LOCAL, 7866 bytes) 20/06/28 02:36:39 INFO Executor: Running task 29.0 in stage 0.0 (TID 29) 20/06/28 02:36:39 INFO TaskSetManager: Finished task 26.0 in stage 0.0 (TID 26) in 113 ms on localhost (executor driver) (27/100) 20/06/28 02:36:39 INFO Executor: Running task 28.0 in stage 0.0 (TID 28) 20/06/28 02:36:39 INFO Executor: Finished task 29.0 in stage 0.0 (TID 29). 781 bytes result sent to driver 20/06/28 02:36:39 INFO TaskSetManager: Starting task 30.0 in stage 0.0 (TID 30, localhost, executor driver, partition 30, PROCESS_LOCAL, 7866 bytes) 20/06/28 02:36:39 INFO Executor: Running task 30.0 in stage 0.0 (TID 30) 20/06/28 02:36:39 INFO TaskSetManager: Finished task 27.0 in stage 0.0 (TID 27) in 137 ms on localhost (executor driver) (28/100) 20/06/28 02:36:39 INFO TaskSetManager: Finished task 29.0 in stage 0.0 (TID 29) in 44 ms on localhost (executor driver) (29/100) 20/06/28 02:36:39 INFO Executor: Finished task 28.0 in stage 0.0 (TID 28). 824 bytes result sent to driver 20/06/28 02:36:39 INFO TaskSetManager: Starting task 31.0 in stage 0.0 (TID 31, localhost, executor driver, partition 31, PROCESS_LOCAL, 7866 bytes) 20/06/28 02:36:39 INFO TaskSetManager: Finished task 28.0 in stage 0.0 (TID 28) in 81 ms on localhost (executor driver) (30/100) 20/06/28 02:36:39 INFO Executor: Running task 31.0 in stage 0.0 (TID 31) 20/06/28 02:36:39 INFO Executor: Finished task 30.0 in stage 0.0 (TID 30). 781 bytes result sent to driver 20/06/28 02:36:39 INFO TaskSetManager: Starting task 32.0 in stage 0.0 (TID 32, localhost, executor driver, partition 32, PROCESS_LOCAL, 7866 bytes) 20/06/28 02:36:39 INFO TaskSetManager: Finished task 30.0 in stage 0.0 (TID 30) in 99 ms on localhost (executor driver) (31/100) 20/06/28 02:36:39 INFO Executor: Running task 32.0 in stage 0.0 (TID 32) 20/06/28 02:36:39 INFO Executor: Finished task 31.0 in stage 0.0 (TID 31). 867 bytes result sent to driver 20/06/28 02:36:39 INFO TaskSetManager: Starting task 33.0 in stage 0.0 (TID 33, localhost, executor driver, partition 33, PROCESS_LOCAL, 7866 bytes) 20/06/28 02:36:39 INFO TaskSetManager: Finished task 31.0 in stage 0.0 (TID 31) in 97 ms on localhost (executor driver) (32/100) 20/06/28 02:36:39 INFO Executor: Running task 33.0 in stage 0.0 (TID 33) 20/06/28 02:36:39 INFO Executor: Finished task 32.0 in stage 0.0 (TID 32). 781 bytes result sent to driver 20/06/28 02:36:39 INFO Executor: Finished task 33.0 in stage 0.0 (TID 33). 781 bytes result sent to driver 20/06/28 02:36:39 INFO TaskSetManager: Starting task 34.0 in stage 0.0 (TID 34, localhost, executor driver, partition 34, PROCESS_LOCAL, 7866 bytes) 20/06/28 02:36:39 INFO TaskSetManager: Starting task 35.0 in stage 0.0 (TID 35, localhost, executor driver, partition 35, PROCESS_LOCAL, 7866 bytes) 20/06/28 02:36:39 INFO TaskSetManager: Finished task 32.0 in stage 0.0 (TID 32) in 106 ms on localhost (executor driver) (33/100) 20/06/28 02:36:39 INFO TaskSetManager: Finished task 33.0 in stage 0.0 (TID 33) in 42 ms on localhost (executor driver) (34/100) 20/06/28 02:36:39 INFO Executor: Running task 34.0 in stage 0.0 (TID 34) 20/06/28 02:36:39 INFO Executor: Running task 35.0 in stage 0.0 (TID 35) 20/06/28 02:36:39 INFO Executor: Finished task 34.0 in stage 0.0 (TID 34). 824 bytes result sent to driver 20/06/28 02:36:39 INFO TaskSetManager: Starting task 36.0 in stage 0.0 (TID 36, localhost, executor driver, partition 36, PROCESS_LOCAL, 7866 bytes) 20/06/28 02:36:39 INFO TaskSetManager: Finished task 34.0 in stage 0.0 (TID 34) in 47 ms on localhost (executor driver) (35/100) 20/06/28 02:36:39 INFO Executor: Running task 36.0 in stage 0.0 (TID 36) 20/06/28 02:36:39 INFO Executor: Finished task 36.0 in stage 0.0 (TID 36). 867 bytes result sent to driver 20/06/28 02:36:39 INFO TaskSetManager: Starting task 37.0 in stage 0.0 (TID 37, localhost, executor driver, partition 37, PROCESS_LOCAL, 7866 bytes) 20/06/28 02:36:39 INFO TaskSetManager: Finished task 36.0 in stage 0.0 (TID 36) in 40 ms on localhost (executor driver) (36/100) 20/06/28 02:36:39 INFO Executor: Running task 37.0 in stage 0.0 (TID 37) 20/06/28 02:36:39 INFO Executor: Finished task 35.0 in stage 0.0 (TID 35). 781 bytes result sent to driver 20/06/28 02:36:39 INFO TaskSetManager: Starting task 38.0 in stage 0.0 (TID 38, localhost, executor driver, partition 38, PROCESS_LOCAL, 7866 bytes) 20/06/28 02:36:39 INFO Executor: Running task 38.0 in stage 0.0 (TID 38) 20/06/28 02:36:39 INFO TaskSetManager: Finished task 35.0 in stage 0.0 (TID 35) in 109 ms on localhost (executor driver) (37/100) 20/06/28 02:36:39 INFO Executor: Finished task 37.0 in stage 0.0 (TID 37). 781 bytes result sent to driver 20/06/28 02:36:39 INFO TaskSetManager: Starting task 39.0 in stage 0.0 (TID 39, localhost, executor driver, partition 39, PROCESS_LOCAL, 7866 bytes) 20/06/28 02:36:39 INFO TaskSetManager: Finished task 37.0 in stage 0.0 (TID 37) in 109 ms on localhost (executor driver) (38/100) 20/06/28 02:36:39 INFO Executor: Running task 39.0 in stage 0.0 (TID 39) 20/06/28 02:36:39 INFO Executor: Finished task 38.0 in stage 0.0 (TID 38). 824 bytes result sent to driver 20/06/28 02:36:39 INFO TaskSetManager: Starting task 40.0 in stage 0.0 (TID 40, localhost, executor driver, partition 40, PROCESS_LOCAL, 7866 bytes) 20/06/28 02:36:39 INFO Executor: Running task 40.0 in stage 0.0 (TID 40) 20/06/28 02:36:39 INFO TaskSetManager: Finished task 38.0 in stage 0.0 (TID 38) in 125 ms on localhost (executor driver) (39/100) 20/06/28 02:36:39 INFO Executor: Finished task 39.0 in stage 0.0 (TID 39). 781 bytes result sent to driver 20/06/28 02:36:39 INFO Executor: Finished task 40.0 in stage 0.0 (TID 40). 781 bytes result sent to driver 20/06/28 02:36:39 INFO TaskSetManager: Starting task 41.0 in stage 0.0 (TID 41, localhost, executor driver, partition 41, PROCESS_LOCAL, 7866 bytes) 20/06/28 02:36:39 INFO TaskSetManager: Starting task 42.0 in stage 0.0 (TID 42, localhost, executor driver, partition 42, PROCESS_LOCAL, 7866 bytes) 20/06/28 02:36:39 INFO TaskSetManager: Finished task 39.0 in stage 0.0 (TID 39) in 92 ms on localhost (executor driver) (40/100) 20/06/28 02:36:39 INFO TaskSetManager: Finished task 40.0 in stage 0.0 (TID 40) in 69 ms on localhost (executor driver) (41/100) 20/06/28 02:36:39 INFO Executor: Running task 41.0 in stage 0.0 (TID 41) 20/06/28 02:36:39 INFO Executor: Running task 42.0 in stage 0.0 (TID 42) 20/06/28 02:36:39 INFO Executor: Finished task 41.0 in stage 0.0 (TID 41). 824 bytes result sent to driver 20/06/28 02:36:39 INFO TaskSetManager: Starting task 43.0 in stage 0.0 (TID 43, localhost, executor driver, partition 43, PROCESS_LOCAL, 7866 bytes) 20/06/28 02:36:39 INFO TaskSetManager: Finished task 41.0 in stage 0.0 (TID 41) in 107 ms on localhost (executor driver) (42/100) 20/06/28 02:36:39 INFO Executor: Running task 43.0 in stage 0.0 (TID 43) 20/06/28 02:36:39 INFO Executor: Finished task 42.0 in stage 0.0 (TID 42). 781 bytes result sent to driver 20/06/28 02:36:40 INFO TaskSetManager: Starting task 44.0 in stage 0.0 (TID 44, localhost, executor driver, partition 44, PROCESS_LOCAL, 7866 bytes) 20/06/28 02:36:40 INFO Executor: Finished task 43.0 in stage 0.0 (TID 43). 867 bytes result sent to driver 20/06/28 02:36:40 INFO Executor: Running task 44.0 in stage 0.0 (TID 44) 20/06/28 02:36:40 INFO TaskSetManager: Finished task 42.0 in stage 0.0 (TID 42) in 563 ms on localhost (executor driver) (43/100) 20/06/28 02:36:40 INFO TaskSetManager: Starting task 45.0 in stage 0.0 (TID 45, localhost, executor driver, partition 45, PROCESS_LOCAL, 7866 bytes) 20/06/28 02:36:40 INFO TaskSetManager: Finished task 43.0 in stage 0.0 (TID 43) in 461 ms on localhost (executor driver) (44/100) 20/06/28 02:36:40 INFO Executor: Running task 45.0 in stage 0.0 (TID 45) 20/06/28 02:36:40 INFO Executor: Finished task 44.0 in stage 0.0 (TID 44). 824 bytes result sent to driver 20/06/28 02:36:40 INFO TaskSetManager: Starting task 46.0 in stage 0.0 (TID 46, localhost, executor driver, partition 46, PROCESS_LOCAL, 7866 bytes) 20/06/28 02:36:40 INFO TaskSetManager: Finished task 44.0 in stage 0.0 (TID 44) in 11 ms on localhost (executor driver) (45/100) 20/06/28 02:36:40 INFO Executor: Running task 46.0 in stage 0.0 (TID 46) 20/06/28 02:36:40 INFO Executor: Finished task 46.0 in stage 0.0 (TID 46). 824 bytes result sent to driver 20/06/28 02:36:40 INFO TaskSetManager: Starting task 47.0 in stage 0.0 (TID 47, localhost, executor driver, partition 47, PROCESS_LOCAL, 7866 bytes) 20/06/28 02:36:40 INFO TaskSetManager: Finished task 46.0 in stage 0.0 (TID 46) in 12 ms on localhost (executor driver) (46/100) 20/06/28 02:36:40 INFO Executor: Running task 47.0 in stage 0.0 (TID 47) 20/06/28 02:36:40 INFO Executor: Finished task 47.0 in stage 0.0 (TID 47). 824 bytes result sent to driver 20/06/28 02:36:40 INFO TaskSetManager: Starting task 48.0 in stage 0.0 (TID 48, localhost, executor driver, partition 48, PROCESS_LOCAL, 7866 bytes) 20/06/28 02:36:40 INFO TaskSetManager: Finished task 47.0 in stage 0.0 (TID 47) in 15 ms on localhost (executor driver) (47/100) 20/06/28 02:36:40 INFO Executor: Running task 48.0 in stage 0.0 (TID 48) 20/06/28 02:36:40 INFO Executor: Finished task 45.0 in stage 0.0 (TID 45). 867 bytes result sent to driver 20/06/28 02:36:40 INFO Executor: Finished task 48.0 in stage 0.0 (TID 48). 781 bytes result sent to driver 20/06/28 02:36:40 INFO TaskSetManager: Starting task 49.0 in stage 0.0 (TID 49, localhost, executor driver, partition 49, PROCESS_LOCAL, 7866 bytes) 20/06/28 02:36:40 INFO TaskSetManager: Starting task 50.0 in stage 0.0 (TID 50, localhost, executor driver, partition 50, PROCESS_LOCAL, 7866 bytes) 20/06/28 02:36:40 INFO TaskSetManager: Finished task 45.0 in stage 0.0 (TID 45) in 44 ms on localhost (executor driver) (48/100) 20/06/28 02:36:40 INFO TaskSetManager: Finished task 48.0 in stage 0.0 (TID 48) in 15 ms on localhost (executor driver) (49/100) 20/06/28 02:36:40 INFO Executor: Running task 50.0 in stage 0.0 (TID 50) 20/06/28 02:36:40 INFO Executor: Running task 49.0 in stage 0.0 (TID 49) 20/06/28 02:36:40 INFO Executor: Finished task 49.0 in stage 0.0 (TID 49). 824 bytes result sent to driver 20/06/28 02:36:40 INFO TaskSetManager: Starting task 51.0 in stage 0.0 (TID 51, localhost, executor driver, partition 51, PROCESS_LOCAL, 7866 bytes) 20/06/28 02:36:40 INFO TaskSetManager: Finished task 49.0 in stage 0.0 (TID 49) in 17 ms on localhost (executor driver) (50/100) 20/06/28 02:36:40 INFO Executor: Finished task 50.0 in stage 0.0 (TID 50). 824 bytes result sent to driver 20/06/28 02:36:40 INFO TaskSetManager: Starting task 52.0 in stage 0.0 (TID 52, localhost, executor driver, partition 52, PROCESS_LOCAL, 7866 bytes) 20/06/28 02:36:40 INFO TaskSetManager: Finished task 50.0 in stage 0.0 (TID 50) in 21 ms on localhost (executor driver) (51/100) 20/06/28 02:36:40 INFO Executor: Running task 51.0 in stage 0.0 (TID 51) 20/06/28 02:36:40 INFO Executor: Running task 52.0 in stage 0.0 (TID 52) 20/06/28 02:36:40 INFO Executor: Finished task 51.0 in stage 0.0 (TID 51). 781 bytes result sent to driver 20/06/28 02:36:40 INFO TaskSetManager: Starting task 53.0 in stage 0.0 (TID 53, localhost, executor driver, partition 53, PROCESS_LOCAL, 7866 bytes) 20/06/28 02:36:40 INFO TaskSetManager: Finished task 51.0 in stage 0.0 (TID 51) in 24 ms on localhost (executor driver) (52/100) 20/06/28 02:36:40 INFO Executor: Finished task 52.0 in stage 0.0 (TID 52). 781 bytes result sent to driver 20/06/28 02:36:40 INFO TaskSetManager: Starting task 54.0 in stage 0.0 (TID 54, localhost, executor driver, partition 54, PROCESS_LOCAL, 7866 bytes) 20/06/28 02:36:40 INFO TaskSetManager: Finished task 52.0 in stage 0.0 (TID 52) in 24 ms on localhost (executor driver) (53/100) 20/06/28 02:36:40 INFO Executor: Running task 53.0 in stage 0.0 (TID 53) 20/06/28 02:36:40 INFO Executor: Finished task 53.0 in stage 0.0 (TID 53). 824 bytes result sent to driver 20/06/28 02:36:40 INFO TaskSetManager: Starting task 55.0 in stage 0.0 (TID 55, localhost, executor driver, partition 55, PROCESS_LOCAL, 7866 bytes) 20/06/28 02:36:40 INFO TaskSetManager: Finished task 53.0 in stage 0.0 (TID 53) in 16 ms on localhost (executor driver) (54/100) 20/06/28 02:36:40 INFO Executor: Running task 54.0 in stage 0.0 (TID 54) 20/06/28 02:36:40 INFO Executor: Running task 55.0 in stage 0.0 (TID 55) 20/06/28 02:36:40 INFO Executor: Finished task 55.0 in stage 0.0 (TID 55). 781 bytes result sent to driver 20/06/28 02:36:40 INFO TaskSetManager: Starting task 56.0 in stage 0.0 (TID 56, localhost, executor driver, partition 56, PROCESS_LOCAL, 7866 bytes) 20/06/28 02:36:40 INFO TaskSetManager: Finished task 55.0 in stage 0.0 (TID 55) in 24 ms on localhost (executor driver) (55/100) 20/06/28 02:36:40 INFO Executor: Running task 56.0 in stage 0.0 (TID 56) 20/06/28 02:36:40 INFO Executor: Finished task 56.0 in stage 0.0 (TID 56). 824 bytes result sent to driver 20/06/28 02:36:40 INFO TaskSetManager: Starting task 57.0 in stage 0.0 (TID 57, localhost, executor driver, partition 57, PROCESS_LOCAL, 7866 bytes) 20/06/28 02:36:40 INFO TaskSetManager: Finished task 56.0 in stage 0.0 (TID 56) in 22 ms on localhost (executor driver) (56/100) 20/06/28 02:36:40 INFO Executor: Running task 57.0 in stage 0.0 (TID 57) 20/06/28 02:36:40 INFO Executor: Finished task 54.0 in stage 0.0 (TID 54). 781 bytes result sent to driver 20/06/28 02:36:40 INFO TaskSetManager: Starting task 58.0 in stage 0.0 (TID 58, localhost, executor driver, partition 58, PROCESS_LOCAL, 7866 bytes) 20/06/28 02:36:40 INFO TaskSetManager: Finished task 54.0 in stage 0.0 (TID 54) in 76 ms on localhost (executor driver) (57/100) 20/06/28 02:36:40 INFO Executor: Finished task 57.0 in stage 0.0 (TID 57). 781 bytes result sent to driver 20/06/28 02:36:40 INFO TaskSetManager: Starting task 59.0 in stage 0.0 (TID 59, localhost, executor driver, partition 59, PROCESS_LOCAL, 7866 bytes) 20/06/28 02:36:40 INFO TaskSetManager: Finished task 57.0 in stage 0.0 (TID 57) in 24 ms on localhost (executor driver) (58/100) 20/06/28 02:36:40 INFO Executor: Running task 59.0 in stage 0.0 (TID 59) 20/06/28 02:36:40 INFO Executor: Running task 58.0 in stage 0.0 (TID 58) 20/06/28 02:36:40 INFO Executor: Finished task 59.0 in stage 0.0 (TID 59). 824 bytes result sent to driver 20/06/28 02:36:40 INFO TaskSetManager: Starting task 60.0 in stage 0.0 (TID 60, localhost, executor driver, partition 60, PROCESS_LOCAL, 7866 bytes) 20/06/28 02:36:40 INFO TaskSetManager: Finished task 59.0 in stage 0.0 (TID 59) in 12 ms on localhost (executor driver) (59/100) 20/06/28 02:36:40 INFO Executor: Running task 60.0 in stage 0.0 (TID 60) 20/06/28 02:36:40 INFO Executor: Finished task 60.0 in stage 0.0 (TID 60). 824 bytes result sent to driver 20/06/28 02:36:40 INFO TaskSetManager: Starting task 61.0 in stage 0.0 (TID 61, localhost, executor driver, partition 61, PROCESS_LOCAL, 7866 bytes) 20/06/28 02:36:40 INFO TaskSetManager: Finished task 60.0 in stage 0.0 (TID 60) in 25 ms on localhost (executor driver) (60/100) 20/06/28 02:36:40 INFO Executor: Running task 61.0 in stage 0.0 (TID 61) 20/06/28 02:36:40 INFO Executor: Finished task 58.0 in stage 0.0 (TID 58). 824 bytes result sent to driver 20/06/28 02:36:40 INFO TaskSetManager: Starting task 62.0 in stage 0.0 (TID 62, localhost, executor driver, partition 62, PROCESS_LOCAL, 7866 bytes) 20/06/28 02:36:40 INFO Executor: Running task 62.0 in stage 0.0 (TID 62) 20/06/28 02:36:40 INFO TaskSetManager: Finished task 58.0 in stage 0.0 (TID 58) in 50 ms on localhost (executor driver) (61/100) 20/06/28 02:36:40 INFO Executor: Finished task 61.0 in stage 0.0 (TID 61). 781 bytes result sent to driver 20/06/28 02:36:40 INFO TaskSetManager: Starting task 63.0 in stage 0.0 (TID 63, localhost, executor driver, partition 63, PROCESS_LOCAL, 7866 bytes) 20/06/28 02:36:40 INFO TaskSetManager: Finished task 61.0 in stage 0.0 (TID 61) in 17 ms on localhost (executor driver) (62/100) 20/06/28 02:36:40 INFO Executor: Running task 63.0 in stage 0.0 (TID 63) 20/06/28 02:36:40 INFO Executor: Finished task 63.0 in stage 0.0 (TID 63). 824 bytes result sent to driver 20/06/28 02:36:40 INFO TaskSetManager: Starting task 64.0 in stage 0.0 (TID 64, localhost, executor driver, partition 64, PROCESS_LOCAL, 7866 bytes) 20/06/28 02:36:40 INFO TaskSetManager: Finished task 63.0 in stage 0.0 (TID 63) in 11 ms on localhost (executor driver) (63/100) 20/06/28 02:36:40 INFO Executor: Running task 64.0 in stage 0.0 (TID 64) 20/06/28 02:36:40 INFO Executor: Finished task 64.0 in stage 0.0 (TID 64). 781 bytes result sent to driver 20/06/28 02:36:40 INFO TaskSetManager: Starting task 65.0 in stage 0.0 (TID 65, localhost, executor driver, partition 65, PROCESS_LOCAL, 7866 bytes) 20/06/28 02:36:40 INFO TaskSetManager: Finished task 64.0 in stage 0.0 (TID 64) in 11 ms on localhost (executor driver) (64/100) 20/06/28 02:36:40 INFO Executor: Running task 65.0 in stage 0.0 (TID 65) 20/06/28 02:36:40 INFO Executor: Finished task 65.0 in stage 0.0 (TID 65). 781 bytes result sent to driver 20/06/28 02:36:40 INFO TaskSetManager: Starting task 66.0 in stage 0.0 (TID 66, localhost, executor driver, partition 66, PROCESS_LOCAL, 7866 bytes) 20/06/28 02:36:40 INFO TaskSetManager: Finished task 65.0 in stage 0.0 (TID 65) in 19 ms on localhost (executor driver) (65/100) 20/06/28 02:36:40 INFO Executor: Running task 66.0 in stage 0.0 (TID 66) 20/06/28 02:36:40 INFO Executor: Finished task 66.0 in stage 0.0 (TID 66). 824 bytes result sent to driver 20/06/28 02:36:40 INFO TaskSetManager: Starting task 67.0 in stage 0.0 (TID 67, localhost, executor driver, partition 67, PROCESS_LOCAL, 7866 bytes) 20/06/28 02:36:40 INFO TaskSetManager: Finished task 66.0 in stage 0.0 (TID 66) in 10 ms on localhost (executor driver) (66/100) 20/06/28 02:36:40 INFO Executor: Running task 67.0 in stage 0.0 (TID 67) 20/06/28 02:36:40 INFO Executor: Finished task 67.0 in stage 0.0 (TID 67). 824 bytes result sent to driver 20/06/28 02:36:40 INFO TaskSetManager: Starting task 68.0 in stage 0.0 (TID 68, localhost, executor driver, partition 68, PROCESS_LOCAL, 7866 bytes) 20/06/28 02:36:40 INFO TaskSetManager: Finished task 67.0 in stage 0.0 (TID 67) in 17 ms on localhost (executor driver) (67/100) 20/06/28 02:36:40 INFO Executor: Finished task 62.0 in stage 0.0 (TID 62). 824 bytes result sent to driver 20/06/28 02:36:40 INFO TaskSetManager: Starting task 69.0 in stage 0.0 (TID 69, localhost, executor driver, partition 69, PROCESS_LOCAL, 7866 bytes) 20/06/28 02:36:40 INFO TaskSetManager: Finished task 62.0 in stage 0.0 (TID 62) in 80 ms on localhost (executor driver) (68/100) 20/06/28 02:36:40 INFO Executor: Running task 68.0 in stage 0.0 (TID 68) 20/06/28 02:36:40 INFO Executor: Finished task 68.0 in stage 0.0 (TID 68). 824 bytes result sent to driver 20/06/28 02:36:40 INFO TaskSetManager: Starting task 70.0 in stage 0.0 (TID 70, localhost, executor driver, partition 70, PROCESS_LOCAL, 7866 bytes) 20/06/28 02:36:40 INFO TaskSetManager: Finished task 68.0 in stage 0.0 (TID 68) in 12 ms on localhost (executor driver) (69/100) 20/06/28 02:36:40 INFO Executor: Running task 69.0 in stage 0.0 (TID 69) 20/06/28 02:36:40 INFO Executor: Running task 70.0 in stage 0.0 (TID 70) 20/06/28 02:36:40 INFO Executor: Finished task 70.0 in stage 0.0 (TID 70). 781 bytes result sent to driver 20/06/28 02:36:40 INFO TaskSetManager: Starting task 71.0 in stage 0.0 (TID 71, localhost, executor driver, partition 71, PROCESS_LOCAL, 7866 bytes) 20/06/28 02:36:40 INFO TaskSetManager: Finished task 70.0 in stage 0.0 (TID 70) in 27 ms on localhost (executor driver) (70/100) 20/06/28 02:36:40 INFO Executor: Running task 71.0 in stage 0.0 (TID 71) 20/06/28 02:36:40 INFO Executor: Finished task 69.0 in stage 0.0 (TID 69). 867 bytes result sent to driver 20/06/28 02:36:40 INFO Executor: Finished task 71.0 in stage 0.0 (TID 71). 824 bytes result sent to driver 20/06/28 02:36:40 INFO TaskSetManager: Starting task 72.0 in stage 0.0 (TID 72, localhost, executor driver, partition 72, PROCESS_LOCAL, 7866 bytes) 20/06/28 02:36:40 INFO TaskSetManager: Starting task 73.0 in stage 0.0 (TID 73, localhost, executor driver, partition 73, PROCESS_LOCAL, 7866 bytes) 20/06/28 02:36:40 INFO TaskSetManager: Finished task 69.0 in stage 0.0 (TID 69) in 48 ms on localhost (executor driver) (71/100) 20/06/28 02:36:40 INFO TaskSetManager: Finished task 71.0 in stage 0.0 (TID 71) in 12 ms on localhost (executor driver) (72/100) 20/06/28 02:36:40 INFO Executor: Running task 72.0 in stage 0.0 (TID 72) 20/06/28 02:36:40 INFO Executor: Running task 73.0 in stage 0.0 (TID 73) 20/06/28 02:36:40 INFO Executor: Finished task 72.0 in stage 0.0 (TID 72). 781 bytes result sent to driver 20/06/28 02:36:40 INFO TaskSetManager: Starting task 74.0 in stage 0.0 (TID 74, localhost, executor driver, partition 74, PROCESS_LOCAL, 7866 bytes) 20/06/28 02:36:40 INFO TaskSetManager: Finished task 72.0 in stage 0.0 (TID 72) in 14 ms on localhost (executor driver) (73/100) 20/06/28 02:36:40 INFO Executor: Running task 74.0 in stage 0.0 (TID 74) 20/06/28 02:36:40 INFO Executor: Finished task 74.0 in stage 0.0 (TID 74). 824 bytes result sent to driver 20/06/28 02:36:40 INFO TaskSetManager: Starting task 75.0 in stage 0.0 (TID 75, localhost, executor driver, partition 75, PROCESS_LOCAL, 7866 bytes) 20/06/28 02:36:40 INFO TaskSetManager: Finished task 74.0 in stage 0.0 (TID 74) in 14 ms on localhost (executor driver) (74/100) 20/06/28 02:36:40 INFO Executor: Running task 75.0 in stage 0.0 (TID 75) 20/06/28 02:36:40 INFO Executor: Finished task 73.0 in stage 0.0 (TID 73). 824 bytes result sent to driver 20/06/28 02:36:40 INFO TaskSetManager: Starting task 76.0 in stage 0.0 (TID 76, localhost, executor driver, partition 76, PROCESS_LOCAL, 7866 bytes) 20/06/28 02:36:40 INFO TaskSetManager: Finished task 73.0 in stage 0.0 (TID 73) in 47 ms on localhost (executor driver) (75/100) 20/06/28 02:36:40 INFO Executor: Finished task 75.0 in stage 0.0 (TID 75). 781 bytes result sent to driver 20/06/28 02:36:40 INFO TaskSetManager: Starting task 77.0 in stage 0.0 (TID 77, localhost, executor driver, partition 77, PROCESS_LOCAL, 7866 bytes) 20/06/28 02:36:40 INFO TaskSetManager: Finished task 75.0 in stage 0.0 (TID 75) in 23 ms on localhost (executor driver) (76/100) 20/06/28 02:36:40 INFO Executor: Running task 77.0 in stage 0.0 (TID 77) 20/06/28 02:36:40 INFO Executor: Running task 76.0 in stage 0.0 (TID 76) 20/06/28 02:36:40 INFO Executor: Finished task 77.0 in stage 0.0 (TID 77). 781 bytes result sent to driver 20/06/28 02:36:40 INFO TaskSetManager: Starting task 78.0 in stage 0.0 (TID 78, localhost, executor driver, partition 78, PROCESS_LOCAL, 7866 bytes) 20/06/28 02:36:40 INFO TaskSetManager: Finished task 77.0 in stage 0.0 (TID 77) in 19 ms on localhost (executor driver) (77/100) 20/06/28 02:36:40 INFO Executor: Running task 78.0 in stage 0.0 (TID 78) 20/06/28 02:36:40 INFO Executor: Finished task 76.0 in stage 0.0 (TID 76). 781 bytes result sent to driver 20/06/28 02:36:40 INFO Executor: Finished task 78.0 in stage 0.0 (TID 78). 824 bytes result sent to driver 20/06/28 02:36:40 INFO TaskSetManager: Starting task 79.0 in stage 0.0 (TID 79, localhost, executor driver, partition 79, PROCESS_LOCAL, 7866 bytes) 20/06/28 02:36:40 INFO TaskSetManager: Starting task 80.0 in stage 0.0 (TID 80, localhost, executor driver, partition 80, PROCESS_LOCAL, 7866 bytes) 20/06/28 02:36:40 INFO TaskSetManager: Finished task 76.0 in stage 0.0 (TID 76) in 32 ms on localhost (executor driver) (78/100) 20/06/28 02:36:40 INFO TaskSetManager: Finished task 78.0 in stage 0.0 (TID 78) in 11 ms on localhost (executor driver) (79/100) 20/06/28 02:36:40 INFO Executor: Running task 79.0 in stage 0.0 (TID 79) 20/06/28 02:36:40 INFO Executor: Running task 80.0 in stage 0.0 (TID 80) 20/06/28 02:36:40 INFO Executor: Finished task 79.0 in stage 0.0 (TID 79). 824 bytes result sent to driver 20/06/28 02:36:40 INFO TaskSetManager: Starting task 81.0 in stage 0.0 (TID 81, localhost, executor driver, partition 81, PROCESS_LOCAL, 7866 bytes) 20/06/28 02:36:40 INFO TaskSetManager: Finished task 79.0 in stage 0.0 (TID 79) in 25 ms on localhost (executor driver) (80/100) 20/06/28 02:36:40 INFO Executor: Finished task 80.0 in stage 0.0 (TID 80). 824 bytes result sent to driver 20/06/28 02:36:40 INFO TaskSetManager: Starting task 82.0 in stage 0.0 (TID 82, localhost, executor driver, partition 82, PROCESS_LOCAL, 7866 bytes) 20/06/28 02:36:40 INFO TaskSetManager: Finished task 80.0 in stage 0.0 (TID 80) in 27 ms on localhost (executor driver) (81/100) 20/06/28 02:36:40 INFO Executor: Running task 81.0 in stage 0.0 (TID 81) 20/06/28 02:36:40 INFO Executor: Running task 82.0 in stage 0.0 (TID 82) 20/06/28 02:36:40 INFO Executor: Finished task 81.0 in stage 0.0 (TID 81). 824 bytes result sent to driver 20/06/28 02:36:40 INFO TaskSetManager: Starting task 83.0 in stage 0.0 (TID 83, localhost, executor driver, partition 83, PROCESS_LOCAL, 7866 bytes) 20/06/28 02:36:40 INFO TaskSetManager: Finished task 81.0 in stage 0.0 (TID 81) in 13 ms on localhost (executor driver) (82/100) 20/06/28 02:36:40 INFO Executor: Running task 83.0 in stage 0.0 (TID 83) 20/06/28 02:36:40 INFO Executor: Finished task 83.0 in stage 0.0 (TID 83). 824 bytes result sent to driver 20/06/28 02:36:40 INFO TaskSetManager: Starting task 84.0 in stage 0.0 (TID 84, localhost, executor driver, partition 84, PROCESS_LOCAL, 7866 bytes) 20/06/28 02:36:40 INFO TaskSetManager: Finished task 83.0 in stage 0.0 (TID 83) in 20 ms on localhost (executor driver) (83/100) 20/06/28 02:36:40 INFO Executor: Finished task 82.0 in stage 0.0 (TID 82). 824 bytes result sent to driver 20/06/28 02:36:40 INFO TaskSetManager: Starting task 85.0 in stage 0.0 (TID 85, localhost, executor driver, partition 85, PROCESS_LOCAL, 7866 bytes) 20/06/28 02:36:40 INFO TaskSetManager: Finished task 82.0 in stage 0.0 (TID 82) in 36 ms on localhost (executor driver) (84/100) 20/06/28 02:36:40 INFO Executor: Running task 84.0 in stage 0.0 (TID 84) 20/06/28 02:36:40 INFO Executor: Finished task 84.0 in stage 0.0 (TID 84). 781 bytes result sent to driver 20/06/28 02:36:40 INFO TaskSetManager: Starting task 86.0 in stage 0.0 (TID 86, localhost, executor driver, partition 86, PROCESS_LOCAL, 7866 bytes) 20/06/28 02:36:40 INFO Executor: Running task 85.0 in stage 0.0 (TID 85) 20/06/28 02:36:40 INFO TaskSetManager: Finished task 84.0 in stage 0.0 (TID 84) in 13 ms on localhost (executor driver) (85/100) 20/06/28 02:36:40 INFO Executor: Running task 86.0 in stage 0.0 (TID 86) 20/06/28 02:36:40 INFO Executor: Finished task 86.0 in stage 0.0 (TID 86). 824 bytes result sent to driver 20/06/28 02:36:40 INFO TaskSetManager: Starting task 87.0 in stage 0.0 (TID 87, localhost, executor driver, partition 87, PROCESS_LOCAL, 7866 bytes) 20/06/28 02:36:40 INFO TaskSetManager: Finished task 86.0 in stage 0.0 (TID 86) in 29 ms on localhost (executor driver) (86/100) 20/06/28 02:36:40 INFO Executor: Running task 87.0 in stage 0.0 (TID 87) 20/06/28 02:36:40 INFO Executor: Finished task 85.0 in stage 0.0 (TID 85). 781 bytes result sent to driver 20/06/28 02:36:40 INFO Executor: Finished task 87.0 in stage 0.0 (TID 87). 824 bytes result sent to driver 20/06/28 02:36:40 INFO TaskSetManager: Starting task 88.0 in stage 0.0 (TID 88, localhost, executor driver, partition 88, PROCESS_LOCAL, 7866 bytes) 20/06/28 02:36:40 INFO TaskSetManager: Starting task 89.0 in stage 0.0 (TID 89, localhost, executor driver, partition 89, PROCESS_LOCAL, 7866 bytes) 20/06/28 02:36:40 INFO TaskSetManager: Finished task 85.0 in stage 0.0 (TID 85) in 52 ms on localhost (executor driver) (87/100) 20/06/28 02:36:40 INFO TaskSetManager: Finished task 87.0 in stage 0.0 (TID 87) in 15 ms on localhost (executor driver) (88/100) 20/06/28 02:36:40 INFO Executor: Running task 88.0 in stage 0.0 (TID 88) 20/06/28 02:36:40 INFO Executor: Running task 89.0 in stage 0.0 (TID 89) 20/06/28 02:36:40 INFO Executor: Finished task 88.0 in stage 0.0 (TID 88). 824 bytes result sent to driver 20/06/28 02:36:40 INFO TaskSetManager: Starting task 90.0 in stage 0.0 (TID 90, localhost, executor driver, partition 90, PROCESS_LOCAL, 7866 bytes) 20/06/28 02:36:40 INFO TaskSetManager: Finished task 88.0 in stage 0.0 (TID 88) in 22 ms on localhost (executor driver) (89/100) 20/06/28 02:36:40 INFO Executor: Running task 90.0 in stage 0.0 (TID 90) 20/06/28 02:36:40 INFO Executor: Finished task 90.0 in stage 0.0 (TID 90). 781 bytes result sent to driver 20/06/28 02:36:40 INFO TaskSetManager: Starting task 91.0 in stage 0.0 (TID 91, localhost, executor driver, partition 91, PROCESS_LOCAL, 7866 bytes) 20/06/28 02:36:40 INFO TaskSetManager: Finished task 90.0 in stage 0.0 (TID 90) in 18 ms on localhost (executor driver) (90/100) 20/06/28 02:36:40 INFO Executor: Running task 91.0 in stage 0.0 (TID 91) 20/06/28 02:36:40 INFO Executor: Finished task 91.0 in stage 0.0 (TID 91). 781 bytes result sent to driver 20/06/28 02:36:40 INFO TaskSetManager: Starting task 92.0 in stage 0.0 (TID 92, localhost, executor driver, partition 92, PROCESS_LOCAL, 7866 bytes) 20/06/28 02:36:40 INFO TaskSetManager: Finished task 91.0 in stage 0.0 (TID 91) in 14 ms on localhost (executor driver) (91/100) 20/06/28 02:36:40 INFO Executor: Running task 92.0 in stage 0.0 (TID 92) 20/06/28 02:36:40 INFO Executor: Finished task 89.0 in stage 0.0 (TID 89). 781 bytes result sent to driver 20/06/28 02:36:40 INFO Executor: Finished task 92.0 in stage 0.0 (TID 92). 781 bytes result sent to driver 20/06/28 02:36:40 INFO TaskSetManager: Starting task 93.0 in stage 0.0 (TID 93, localhost, executor driver, partition 93, PROCESS_LOCAL, 7866 bytes) 20/06/28 02:36:40 INFO TaskSetManager: Starting task 94.0 in stage 0.0 (TID 94, localhost, executor driver, partition 94, PROCESS_LOCAL, 7866 bytes) 20/06/28 02:36:40 INFO Executor: Running task 94.0 in stage 0.0 (TID 94) 20/06/28 02:36:40 INFO TaskSetManager: Finished task 89.0 in stage 0.0 (TID 89) in 66 ms on localhost (executor driver) (92/100) 20/06/28 02:36:40 INFO TaskSetManager: Finished task 92.0 in stage 0.0 (TID 92) in 15 ms on localhost (executor driver) (93/100) 20/06/28 02:36:40 INFO Executor: Running task 93.0 in stage 0.0 (TID 93) 20/06/28 02:36:40 INFO Executor: Finished task 93.0 in stage 0.0 (TID 93). 824 bytes result sent to driver 20/06/28 02:36:40 INFO TaskSetManager: Starting task 95.0 in stage 0.0 (TID 95, localhost, executor driver, partition 95, PROCESS_LOCAL, 7866 bytes) 20/06/28 02:36:40 INFO TaskSetManager: Finished task 93.0 in stage 0.0 (TID 93) in 16 ms on localhost (executor driver) (94/100) 20/06/28 02:36:40 INFO Executor: Running task 95.0 in stage 0.0 (TID 95) 20/06/28 02:36:40 INFO Executor: Finished task 94.0 in stage 0.0 (TID 94). 824 bytes result sent to driver 20/06/28 02:36:40 INFO Executor: Finished task 95.0 in stage 0.0 (TID 95). 824 bytes result sent to driver 20/06/28 02:36:40 INFO TaskSetManager: Starting task 96.0 in stage 0.0 (TID 96, localhost, executor driver, partition 96, PROCESS_LOCAL, 7866 bytes) 20/06/28 02:36:40 INFO TaskSetManager: Starting task 97.0 in stage 0.0 (TID 97, localhost, executor driver, partition 97, PROCESS_LOCAL, 7866 bytes) 20/06/28 02:36:40 INFO Executor: Running task 97.0 in stage 0.0 (TID 97) 20/06/28 02:36:40 INFO TaskSetManager: Finished task 94.0 in stage 0.0 (TID 94) in 28 ms on localhost (executor driver) (95/100) 20/06/28 02:36:40 INFO TaskSetManager: Finished task 95.0 in stage 0.0 (TID 95) in 14 ms on localhost (executor driver) (96/100) 20/06/28 02:36:40 INFO Executor: Running task 96.0 in stage 0.0 (TID 96) 20/06/28 02:36:40 INFO Executor: Finished task 96.0 in stage 0.0 (TID 96). 824 bytes result sent to driver 20/06/28 02:36:40 INFO TaskSetManager: Starting task 98.0 in stage 0.0 (TID 98, localhost, executor driver, partition 98, PROCESS_LOCAL, 7866 bytes) 20/06/28 02:36:40 INFO TaskSetManager: Finished task 96.0 in stage 0.0 (TID 96) in 19 ms on localhost (executor driver) (97/100) 20/06/28 02:36:40 INFO Executor: Running task 98.0 in stage 0.0 (TID 98) 20/06/28 02:36:40 INFO Executor: Finished task 98.0 in stage 0.0 (TID 98). 781 bytes result sent to driver 20/06/28 02:36:40 INFO Executor: Finished task 97.0 in stage 0.0 (TID 97). 824 bytes result sent to driver 20/06/28 02:36:40 INFO TaskSetManager: Starting task 99.0 in stage 0.0 (TID 99, localhost, executor driver, partition 99, PROCESS_LOCAL, 7866 bytes) 20/06/28 02:36:40 INFO Executor: Running task 99.0 in stage 0.0 (TID 99) 20/06/28 02:36:40 INFO TaskSetManager: Finished task 98.0 in stage 0.0 (TID 98) in 18 ms on localhost (executor driver) (98/100) 20/06/28 02:36:40 INFO TaskSetManager: Finished task 97.0 in stage 0.0 (TID 97) in 35 ms on localhost (executor driver) (99/100) 20/06/28 02:36:40 INFO Executor: Finished task 99.0 in stage 0.0 (TID 99). 781 bytes result sent to driver 20/06/28 02:36:40 INFO TaskSetManager: Finished task 99.0 in stage 0.0 (TID 99) in 12 ms on localhost (executor driver) (100/100) 20/06/28 02:36:40 INFO TaskSchedulerImpl: Removed TaskSet 0.0, whose tasks have all completed, from pool 20/06/28 02:36:40 INFO DAGScheduler: ResultStage 0 (reduce at SparkPi.scala:38) finished in 3.115 s 20/06/28 02:36:40 INFO DAGScheduler: Job 0 finished: reduce at SparkPi.scala:38, took 3.176151 s Pi is roughly 3.1422703142270314 20/06/28 02:36:40 INFO SparkUI: Stopped Spark web UI at http://hadoop101.yinzhengjie.org.cn:4040 20/06/28 02:36:40 INFO MapOutputTrackerMasterEndpoint: MapOutputTrackerMasterEndpoint stopped! 20/06/28 02:36:40 INFO MemoryStore: MemoryStore cleared 20/06/28 02:36:40 INFO BlockManager: BlockManager stopped 20/06/28 02:36:40 INFO BlockManagerMaster: BlockManagerMaster stopped 20/06/28 02:36:40 INFO OutputCommitCoordinator$OutputCommitCoordinatorEndpoint: OutputCommitCoordinator stopped! 20/06/28 02:36:40 INFO SparkContext: Successfully stopped SparkContext 20/06/28 02:36:40 INFO ShutdownHookManager: Shutdown hook called 20/06/28 02:36:40 INFO ShutdownHookManager: Deleting directory /tmp/spark-abc62f4f-e086-46b5-9660-567122352e37 20/06/28 02:36:40 INFO ShutdownHookManager: Deleting directory /tmp/spark-7e4020d1-9318-4c3e-9740-95e1a389bbdf [root@hadoop101.yinzhengjie.org.cn ~]#
三.运行wordCount程序
1>.准备数据源
[root@hadoop101.yinzhengjie.org.cn ~]# ll /tmp/data/ total 8 -rw-r--r-- 1 root root 46 Jun 28 03:14 1.txt -rw-r--r-- 1 root root 60 Jun 28 03:14 2.txt [root@hadoop101.yinzhengjie.org.cn ~]# [root@hadoop101.yinzhengjie.org.cn ~]# cat /tmp/data/1.txt hello java java python kafka shell spark java [root@hadoop101.yinzhengjie.org.cn ~]# [root@hadoop101.yinzhengjie.org.cn ~]# cat /tmp/data/2.txt hello golang bigdata shell java python world java spark c++ [root@hadoop101.yinzhengjie.org.cn ~]#
2>.运行spark程序
[root@hadoop101.yinzhengjie.org.cn ~]# spark-shell 20/06/28 02:50:47 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties Setting default log level to "WARN". To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel). Spark context Web UI available at http://hadoop101.yinzhengjie.org.cn:4040 Spark context available as 'sc' (master = local[*], app id = local-1593283854727). Spark session available as 'spark'. Welcome to ____ __ / __/__ ___ _____/ /__ _\ \/ _ \/ _ `/ __/ '_/ /___/ .__/\_,_/_/ /_/\_\ version 2.4.6 /_/ Using Scala version 2.11.12 (Java HotSpot(TM) 64-Bit Server VM, Java 1.8.0_201) Type in expressions to have them evaluated. Type :help for more information. scala> scala> sc.textFile("/tmp/data/").flatMap(_.split(" ")).map((_,1)).reduceByKey(_+_).collect res0: Array[(String, Int)] = Array((kafka,1), (world,1), (python,2), (golang,1), (hello,2), (java,5), (spark,2), (c++,1), (bigdata,1), (shell,2)) scala> scala> sc res1: org.apache.spark.SparkContext = org.apache.spark.SparkContext@67744663 scala> 数据流分析： sc: Spark的上下文对象。 textFile("/tmp/data/")： 读取本地文件input文件夹数据； flatMap(_.split(" "))： 压平操作，按照空格分割符将一行数据映射成一个个单词； map((_,1))： 对每一个元素操作，将单词映射为元组； reduceByKey(_+_)： 按照key将值进行聚合，相加； collect： 将数据收集到Driver端展示。
3>.查看webUI
当你的才华还撑不起你的野心的时候，你就应该静下心来学习。当你的能力还驾驭不了你的目标的时候，你就应该沉下心来历练。问问自己，想要怎样的人生。 欢迎加入基础架构自动化运维：598432640，大数据SRE进阶之路：959042252，DevOps进阶之路：526991186
