Scala实验4.1
请到本教程官网的“下载专区”的“数据集”中下载 chapter5-data1.txt,该数据集包含
了某大学计算机系的成绩,数据格式如下所示:
Tom,DataBase,80
Tom,Algorithm,50
Tom,DataStructure,60
Jim,DataBase,90
Jim,Algorithm,60
Jim,DataStructure,80
(1) 该系总共有多少学生; val lines = sc.textFile("file:///usr/local/spark/sparksqldata/Data01.txt") val par = lines.map(row=>row.split(",")(0)) val distinct_par = par.distinct() //去重操作 distinct_par.count //取得总数
(2) 该系共开设来多少门课程; val lines = sc.textFile("file:///usr/local/spark/sparksqldata/Data01.txt") val par = lines.map(row=>row.split(",")(1)) val distinct_par = par.distinct() distinct_par.count
(3) Tom 同学的总成绩平均分是多少; val lines = sc.textFile("file:///usr/local/spark/sparksqldata/Data01.txt") val pare = lines.filter(row=>row.split(",")(0)=="Tom") pare.foreach(println) Tom,DataBase,26 Tom,Algorithm,12 Tom,OperatingSystem,16 Tom,Python,40 Tom,Software,60 pare.map(row=>(row.split(",")(0),row.split(",")(2).toInt)).mapValues(x=>(x,1)).reduceByKey((x,y ) => (x._1+y._1,x._2 + y._2)).mapValues(x => (x._1 / x._2)).collect() //res9: Array[(String, Int)] = Array((Tom,30))
(4) 求每名同学的选修的课程门数; val lines = sc.textFile("file:///usr/local/spark/sparksqldata/Data01.txt") val pare = lines.map(row=>(row.split(",")(0),row.split(",")(1))) pare.mapValues(x => (x,1)).reduceByKey((x,y) => (" ",x._2 + y._2)).mapValues(x => x._2).foreach(println)
(5) 该系 DataBase 课程共有多少人选修; val lines = sc.textFile("file:///usr/local/spark/sparksqldata/Data01.txt") val pare = lines.filter(row=>row.split(",")(1)=="DataBase") pare.count res1: Long = 126
(6) 各门课程的平均分是多少; val lines = sc.textFile("file:///usr/local/spark/sparksqldata/Data01.txt") val pare = lines.map(row=>(row.split(",")(1),row.split(",")(2).toInt)) pare.mapValues(x=>(x,1)).reduceByKey((x,y) => (x._1+y._1,x._2 + y._2)).mapValues(x => (x._1/ x._2)).collect()
(7)使用累加器计算共有多少人选了 DataBase 这门课。 val lines = sc.textFile("file:///usr/local/spark/sparksqldata/Data01.txt") val pare = lines.filter(row=>row.split(",")(1)=="DataBase").map(row=>(row.split(",")(1),1)) val accum = sc.longAccumulator("My Accumulator") pare.values.foreach(x => accum.add(x)) accum.value res19: Long = 126