对RDD分区的理解
举个例子:
val logFile = "file:///home/soyo/桌面/6.txt" val conf = new SparkConf().setAppName("Simple Application") val sc = new SparkContext(conf)
val RDD =sc.textFile(logFile)
RDD.saveAsTextFile("file:////home/soyo/桌面/667.txt")
val repartitionedRDD=context.repartition(8) //控制RDD分区的个数
repartitionedRDD.saveAsTextFile("file:////home/soyo/桌面/668.txt")
667.txt:
part-00000 _SUCCESS
668.txt:
part-00000 part-00001 part-00002 part-00003 part-00004 part-00005 part-00006 part-00007 _SUCCESS
如果设置.setMaster("local[2]")默认最少是2个分区,别的类同