Spark常用函数讲解
object MapValues {
def main(args: Array[String]) {
val conf =
new
SparkConf().setMaster(
"local"
).setAppName(
"map"
)
val sc =
new
SparkContext(conf)
val list = List((
"mobin"
,
22
),(
"kpop"
,
20
),(
"lufei"
,
23
))
val rdd = sc.parallelize(list)
val mapValuesRDD = rdd.mapValues(_+
2
)
mapValuesRDD.foreach(println)
}
}
输出: (mobin,24) (kpop,22) (lufei,25)
2. flatMapValues(fun):对[K,V]型数据中的V值flatmap操作
//省略<br>val list = List(("mobin",22),("kpop",20),("lufei",23))
val rdd = sc.parallelize(list)
val mapValuesRDD = rdd.flatMapValues(x => Seq(x,
"male"
))
mapValuesRDD.foreach(println)
(mobin,22)
(mobin,male)
(kpop,20)
(kpop,male)
(lufei,23)
(lufei,male)
(mobin,male)
(kpop,20)
(kpop,male)
(lufei,23)
(lufei,male)
如果是mapValues会输出:
(mobin,List(22, male))
(kpop,List(20, male)) (lufei,List(23, male))
(kpop,List(20, male)) (lufei,List(23, male))
3. comineByKey(createCombiner,mergeValue,mergeCombiners,partitioner,mapSideCombine)
createCombiner:在第一次遇到Key时创建组合器函数,将RDD数据集中的V类型值转换C类型值(V => C),
mergeValue:合并值函数,再次遇到相同的Key时,将createCombiner道理的C类型值与这次传入的V类型值合并成一个C类型值(C,V)=>C,
mergeCombiners:合并组合器函数,将C类型值两两合并成一个C类型值
object CombineByKey {
def main(args: Array[String]) {
val conf =
new
SparkConf().setMaster(
"local"
).setAppName(
"combinByKey"
)
val sc =
new
SparkContext(conf)
val people = List((
"male"
,
"Mobin"
), (
"male"
,
"Kpop"
), (
"female"
,
"Lucy"
), (
"male"
,
"Lufei"
), (
"female"
,
"Amy"
))
val rdd = sc.parallelize(people)
val combinByKeyRDD = rdd.combineByKey(
(x: String) => (List(x),
1
),
(peo: (List[String], Int), x : String) => (x :: peo._1, peo._2 +
1
),
(sex1: (List[String], Int), sex2: (List[String], Int)) => (sex1._1 ::: sex2._1, sex1._2 + sex2._2))
combinByKeyRDD.foreach(println)
sc.stop()
}
}
(male,(List(Lufei, Kpop, Mobin),3))
(female,(List(Amy, Lucy),2))
(female,(List(Amy, Lucy),2))
foldByKey函数是通过调用CombineByKey函数实现的
func: Value将通过func函数按Key值进行合并(实际上是通过CombineByKey的mergeValue,mergeCombiners函数实现的,只不过在这里,这两个函数是相同的)
//省略
val people = List((
"Mobin"
,
2
), (
"Mobin"
,
1
), (
"Lucy"
,
2
), (
"Amy"
,
1
), (
"Lucy"
,
3
))
val rdd = sc.parallelize(people)
val foldByKeyRDD = rdd.foldByKey(
2
)(_+_)
foldByKeyRDD.foreach(println)
7.sortByKey(accending,numPartitions):返回以Key排序的(K,V)键值对组成的RDD,accending为true时表示升序,为false时表示降序,numPartitions设置分区数,提高作业并行度
8.cogroup(otherDataSet,numPartitions):对两个RDD(如:(K,V)和(K,W))相同Key的元素先分别做聚合,最后返回(K,Iterator<V>,Iterator<W>)形式的RDD,numPartitions设置分区数,提高作业并行度
val arr = List((
"A"
,
1
), (
"B"
,
2
), (
"A"
,
2
), (
"B"
,
3
))
val arr1 = List((
"A"
,
"A1"
), (
"B"
,
"B1"
), (
"A"
,
"A2"
), (
"B"
,
"B2"
))
val rdd1 = sc.parallelize(arr,
3
)
val rdd2 = sc.parallelize(arr1,
3
)
val groupByKeyRDD = rdd1.cogroup(rdd2)
groupByKeyRDD.foreach(println)
sc.stop
(B,(CompactBuffer(2, 3),CompactBuffer(B1, B2)))
(A,(CompactBuffer(1, 2),CompactBuffer(A1, A2)))
(A,(CompactBuffer(1, 2),CompactBuffer(A1, A2)))
9. join(otherDataSet,numPartitions):对两个RDD先进行cogroup操作形成新的RDD,再对每个Key下的元素进行笛卡尔积,numPartitions设置分区数,提高作业并行度
10.LeftOutJoin(otherDataSet,numPartitions):左外连接,包含左RDD的所有数据,如果右边没有与之匹配的用None表示,numPartitions设置分区数,提高作业并行度