随笔分类 -  spark

1 2 下一页
spark学习笔记之二：宽依赖和窄依赖
摘要：1.如果父RDD里的一个partition只去向一个子RDD里的partition为窄依赖，否则为宽依赖（只要是shuffle操作）。 2.spark根据算子判断宽窄依赖： 窄依赖：map,filter,union 宽依赖:groupByKey,join 3.宽窄依赖用于切割 action算子生成的 阅读全文
posted @ 2017-07-27 17:17 书灯 阅读(21) 评论(0) 推荐(0) 编辑
spark--transform算子--union
摘要：import org.apache.spark.{SparkConf, SparkContext} /** * Created by liupeng on 2017/6/16. */ object T_union { System.setProperty("hadoop.home.dir","F:\ 阅读全文
posted @ 2017-07-19 10:08 书灯 阅读(2) 评论(0) 推荐(0) 编辑
spark--transform算子--sortByKey
摘要：import org.apache.spark.{SparkConf, SparkContext} /** * Created by liupeng on 2017/6/16. */ object T_sortByKey { System.setProperty("hadoop.home.dir", 阅读全文
posted @ 2017-07-19 10:05 书灯 阅读(0) 评论(0) 推荐(0) 编辑
spark--transform算子--sample
摘要：import org.apache.spark.{SparkConf, SparkContext} /** * Created by liupeng on 2017/6/16. */ object T_sample { System.setProperty("hadoop.home.dir","F: 阅读全文
posted @ 2017-07-19 09:49 书灯 阅读(2) 评论(0) 推荐(0) 编辑
spark--transform算子--repartition
摘要：import org.apache.spark.{SparkConf, SparkContext} import scala.collection.mutable.ArrayBuffer /** * Created by liupeng on 2017/6/16. */ object T_repar 阅读全文
posted @ 2017-07-19 09:39 书灯 阅读(5) 评论(0) 推荐(0) 编辑
spark--transform算子--reduceByKey
摘要：import org.apache.spark.{SparkConf, SparkContext} /** * Created by liupeng on 2017/6/16. */ object T_reduceByKey { System.setProperty("hadoop.home.dir 阅读全文
posted @ 2017-07-19 09:06 书灯 阅读(0) 评论(0) 推荐(0) 编辑
spark--transform算子--parallelized
摘要：import org.apache.spark.{SparkConf, SparkContext} /** * Created by liupeng on 2017/6/15. */ object T_parallelized { System.setProperty("hadoop.home.di 阅读全文
posted @ 2017-07-18 21:56 书灯 阅读(2) 评论(0) 推荐(0) 编辑
spark--transform算子--mapPartitionsWithIndex
摘要：import org.apache.spark.{SparkConf, SparkContext} import scala.collection.mutable.ArrayBuffer /** * Created by liupeng on 2017/6/15. */ object T_mapPa 阅读全文
posted @ 2017-07-18 21:52 书灯 阅读(6) 评论(0) 推荐(0) 编辑
spark--transform算子--mapPartitions
摘要：import org.apache.spark.{SparkConf, SparkContext} import scala.collection.immutable.HashMap import scala.collection.mutable.ArrayBuffer /** * Created 阅读全文
posted @ 2017-07-18 21:46 书灯 阅读(4) 评论(0) 推荐(0) 编辑
spark--transform算子--map
摘要：import org.apache.spark.{SparkConf, SparkContext} /** * Created by liupeng on 2017/6/15. */ object T_map { System.setProperty("hadoop.home.dir","F:\\h 阅读全文
posted @ 2017-07-18 21:43 书灯 阅读(0) 评论(0) 推荐(0) 编辑
spark--transform算子--join
摘要：import org.apache.spark.{SparkConf, SparkContext} /** * Created by liupeng on 2017/6/16. */ object T_join { System.setProperty("hadoop.home.dir","F:\\ 阅读全文
posted @ 2017-07-18 21:38 书灯 阅读(1) 评论(0) 推荐(0) 编辑
spark--transform算子--intersection
摘要：import org.apache.spark.{SparkConf, SparkContext} /** * Created by liupeng on 2017/6/16. */ object T_intersection { System.setProperty("hadoop.home.di 阅读全文
posted @ 2017-07-18 21:24 书灯 阅读(1) 评论(0) 推荐(0) 编辑
spark--transform算子--groupByKey
摘要：import org.apache.spark.{SparkConf, SparkContext} /** * Created by liupeng on 2017/6/16. */ object T_groupByKey { System.setProperty("hadoop.home.dir" 阅读全文
posted @ 2017-07-18 11:42 书灯 阅读(2) 评论(0) 推荐(0) 编辑
spark--transform算子--flatMap
摘要：import org.apache.spark.{SparkConf, SparkContext} /** * Created by liupeng on 2017/6/16. */ object T_flatMap { System.setProperty("hadoop.home.dir","F 阅读全文
posted @ 2017-07-18 11:41 书灯 阅读(2) 评论(0) 推荐(0) 编辑
spark--transform算子--filter
摘要：import org.apache.spark.{SparkConf, SparkContext} /** * Created by yz02 on 2017/6/15. */ object T_filter { System.setProperty("hadoop.home.dir","F:\\h 阅读全文
posted @ 2017-07-18 11:39 书灯 阅读(1) 评论(0) 推荐(0) 编辑
spark--transform算子--distinct
摘要：import org.apache.spark.{SparkConf, SparkContext} /** * Created by liupeng on 2017/6/16. */ object T_distinct { System.setProperty("hadoop.home.dir"," 阅读全文
posted @ 2017-07-18 11:15 书灯 阅读(1) 评论(0) 推荐(0) 编辑
spark--transform算子--cogroup
摘要：import org.apache.spark.{SparkConf, SparkContext} /** * Created by yz02 on 2017/6/16. */ object T_cogroup { System.setProperty("hadoop.home.dir","F:\\ 阅读全文
posted @ 2017-07-17 17:56 书灯 阅读(4) 评论(0) 推荐(0) 编辑
spark--transform算子--coalesce
摘要：import org.apache.spark.{SparkConf, SparkContext} import scala.collection.mutable.ArrayBuffer /** * Created by yz02 on 2017/6/15. */ object T_coalesce 阅读全文
posted @ 2017-07-17 17:52 书灯 阅读(4) 评论(0) 推荐(0) 编辑
spark--transform算子--cartesian
摘要：import org.apache.spark.{SparkConf, SparkContext} /** * Created by lp on 2017/6/16. */ object T_cartesian { System.setProperty("hadoop.home.dir","F:\\ 阅读全文
posted @ 2017-07-17 17:47 书灯 阅读(3) 评论(0) 推荐(0) 编辑
spark--actions算子--takeSample
摘要：import org.apache.spark.{SparkConf, SparkContext} /** * Created by yz02 on 2017/6/16. */ object A_takeSample { System.setProperty("hadoop.home.dir","F 阅读全文
posted @ 2017-07-13 09:41 书灯 阅读(4) 评论(0) 推荐(0) 编辑

1 2 下一页
点击右上角即可分享
微信分享提示