Spark RDD Transformation和Action

spark -- Spark RDD Transformation和Action

目录

  • Transformation算子
    • 基本的初始化
  • 一、map、flatMap、mapParations、mapPartitionsWithIndex
    • 1.1map
    • 1.2flatMap
    • 1.3mapPartitions
    • 1.4mapPartitionsWithIndex
  • 二、reduce、reduceByKey
    • 2.1reduce
    • 2.2reduceByKey
  • 三、union,join和groupByKey 
    • 3.1union
    • 3.2groupByKey
    • 3.3join
  • 四、sample、cartesian 
    • 4.1sample
    • 4.2cartesian
  • 五、filter、distinct、intersection
    • 5.1filter
    • 5.2distinct
    • 5.3intersection
  • 六、coalesce、repartition、repartitionAndSortWithinPartitions
    • 6.1coalesce
    • 6.2 replication
    • 6.3repartitionAndSortWithinPartitions
  • 七、cogroup、sortBykey、aggregateByKey
    • 7.1cogroup
    • 7.2sortBykey
    • 7.3aggregateByKey

Transformation算子

基本的初始化

val config = new SparkConf().setAppName("MapPartitionsAPP").setMaster("local[2]") 
val sc = new SparkContext(config) // 获取spark 上下文

一、map、flatMap、mapParations、mapPartitionsWithIndex

1.1  map

def map: Unit ={
    val list = List("spark","hadoop","sqoop","hive","storm")
    val listRDD = sc.parallelize(list) //parallelize第二个参数可以指定RDD分区个数
  /**
  * 对于map算子,源JavaRDD的每个元素都会进行计算,由于是依次进行传参,所以他是有序的,新RDD的元素顺序与源RDD是相同的。而由有序又引出接下来的flatMap
  */ val listMapRDD = listRDD.map(name =>{ "hello word" + name }) listMapRDD.foreach(println(_)) }

1.2  flatMap

 def flatMap: Unit ={
    val list = List("spark sparkSQL","hadoop MapReduce","sqoop","hive","storm")
    val listRDD = sc.parallelize(list)
    val flatMapRDD = listRDD.flatMap(name => {
      name.split(" ").map(name =>"hello word"+name)
    })
    flatMapRDD.foreach(println(_))
  }

1.3   mapPartitions

def mapPartitions: Unit ={
    val list = List(1, 2, 3, 4, 5, 6)
    val listRDD = sc.parallelize(list,2)
    /**
      * map和flatMap都是依次进行参数传递的,
      * 但有时候需要RDD中的两个元素进行相应操作时(例如:算存款所得时,下一个月所得的利息是要原本金加上上一个月所得的本金56的),
      * 这两个算子便无法达到目的了,这是便需要mapPartitions算子,他传参的方式是将整个RDD传入,
      * 然后将一个迭代器传出生成一个新的RDD,由于整个RDD都传入了,所以便能完成前面说的业务。
      */
    listRDD.mapPartitions(iterator => {
      val newList: ListBuffer[String] = ListBuffer()
      while (iterator.hasNext){
        newList.append("hello " + iterator.next())
      }
      newList.toIterator
    }).foreach(name => println(name))
  }

1.4 mapPartitionsWithIndex

每次获取和处理的就是一个分区的数据,并且知道处理的分区的分区号

  def main(args: Array[String]): Unit = {
    val conf = new SparkConf().setAppName("MapPartitionsWithIndexAPP").setMaster("local[2]")
    val sc = new SparkContext(conf)
    val list  = List(1, 2, 3, 4, 5, 6, 7, 8, 9, 10)
    /**
      * mapPartitionsWithIndex
      * 每次获取和处理的就是一个分区的数据,并且知道处理的分区的分区号index
      */
    val listRDD = sc.parallelize(list).mapPartitionsWithIndex((index,iterator) => {
      val listBuffer:ListBuffer[String] = new ListBuffer
      while (iterator.hasNext){
        listBuffer.append(index+"_"+iterator.next())
      }
      listBuffer.iterator
    },true).foreach(println(_))
  }

二、reduce、reduceByKey

2.1reduce

def main(args: Array[String]): Unit = {
    val conf = new SparkConf().setAppName("ReduceAPP").setMaster("local[2]")
    val sc = new SparkContext(conf)
    val list = Array(1,2,3,4,5,1,2,3)
    val listRDD = sc.parallelize(list)
    /**
      * reduce其实是将RDD中的所有元素进行合并,
      * 当运行call方法时,会传入两个参数,
      * 在call方法中将两个参数合并后返回,而这个返回值回合一个新的RDD中的元素再次传入call方法中,继续合并,直到合并到只剩下一个元素时。
      */
    val resule =  listRDD.reduce((x,y) => x+y )
    println(resule)
  }

 

2.2  reduceByKey

 def main(args: Array[String]): Unit = {
    val conf = new SparkConf().setAppName("ReduceByKeyAPP").setMaster("local[2]")
    val sc = new SparkContext(conf)
    val list = List(("A", 99), ("B", 97), ("A", 89), ("B", 77))
    val mapRDD = sc.parallelize(list)
    /**
      * reduceByKey仅将RDD中所有K,V对中K值相同的V进行合并。
      */
    val resultRDD = mapRDD.reduceByKey((_+_))
    resultRDD.foreach(tuple => println(tuple._1 + "->"+tuple._2))
  }

三、union,join和groupByKey 

3.1union

def main(args: Array[String]): Unit = {
    val conf = new SparkConf().setAppName("ReduceByKeyAPP").setMaster("local[2]")
    val sc = new SparkContext(conf)
    val list1 = List(1,2,3,4)
    val list2 = List(2,2,3,4)
    val rdd1 = sc.parallelize(list1)
    val rdd2 = sc.parallelize(list2)
    /**
      * union 操作只是将两个RDD连接起来,相当于List的 ADDALL操作,local[2] 导致有两个分区
      */
    rdd1.union(rdd2).foreach(println(_))
  }

3.2groupByKey

def main(args: Array[String]): Unit = {
    val conf = new SparkConf().setAppName("ReduceByKeyAPP").setMaster("local[2]")
    val sc = new SparkContext(conf)
    val list = List(("hadoop", "MapReduce"), ("hadoop", "hive"), ("Spark", "SparkSQL"), ("Spark", "SpartStreaming"))
    val listRDD = sc.parallelize(list)
    /**
      * groupByKey是将PairRDD中拥有相同key值得元素归为一组
      */
    val groupByKeyRDD = listRDD.groupByKey()
    groupByKeyRDD.foreach(touple => {
      val key = touple._1
      val valuesiter = touple._2.iterator
      var people = ""
      while(valuesiter.hasNext){
        people = people + valuesiter.next + " "
      }
      println(key + " -> " + people)
    })
  }

3.3join

 def join(): Unit ={
    val list1 = List((1, "Apache"), (2, "Nginx"), (3, "Tomcat"))
    val list2 = List((1, 99), (2, 98), (3, 97))
    val list1RDD = sc.parallelize(list1)
    val list2RDD = sc.parallelize(list2)
    /**
      * join是将两个PairRDD合并,并将有相同key的元素分为一组,可以理解为groupByKey和Union的结合
      */
    val joinRDD = list1RDD.join(list2RDD)
    joinRDD.foreach(t => {
      println("学号:"+ t._1 +"   姓名:"+t._2._1 + "   成绩" + t._2._2)
    })
  }

 

posted @ 2018-11-12 14:53  局外人~~  阅读(279)  评论(0编辑  收藏  举报