spark 变量使用 broadcast、accumulator
broadcast
官方文档描述:
Broadcast a read-only variable to the cluster, returning a [[org.apache.spark.broadcast.Broadcast]] object for reading it in distributed functions. The variable will be sent to each cluster only once.
函数原型:
def broadcast[T](value: T): Broadcast[T]
广播变量允许程序员将一个只读的变量缓存在每台机器上,而不用在任务之间传递变量。广播变量可被用于有效地给每个节点一个大输入数据集的副本。Spark还尝试使用高效地广播算法来分发变量,进而减少通信的开销。 Spark的动作通过一系列的步骤执行,这些步骤由分布式的洗牌操作分开。Spark自动地广播每个步骤每个任务需要的通用数据。这些广播数据被序列化地缓存,在运行任务之前被反序列化出来。这意味着当我们需要在多个阶段的任务之间使用相同的数据,或者以反序列化形式缓存数据是十分重要的时候,显式地创建广播变量才有用。
源码分析:
def broadcast[T: ClassTag](value: T): Broadcast[T] = {
assertNotStopped()
if (classOf[RDD[_]].isAssignableFrom(classTag[T].runtimeClass)) {
// This is a warning instead of an exception in order to avoid breaking user programs that
// might have created RDD broadcast variables but not used them:
logWarning("Can not directly broadcast RDDs; instead, call collect() and "
+ "broadcast the result (see SPARK-5063)")
}
val bc = env.broadcastManager.newBroadcast[T](value, isLocal)
val callSite = getCallSite
logInfo("Created broadcast " + bc.id + " from " + callSite.shortForm)
cleaner.foreach(_.registerBroadcastForCleanup(bc))
bc
}
实例:
1 List<Integer> data = Arrays.asList(5, 1, 1, 4, 4, 2, 2); 2 JavaRDD<Integer> javaRDD = javaSparkContext.parallelize(data,5); 3 final Broadcast<List<Integer>> broadcast = javaSparkContext.broadcast(data); 4 JavaRDD<Integer> result = javaRDD.map(new Function<Integer, Integer>() { 5 List<Integer> iList = broadcast.value(); 6 @Override 7 public Integer call(Integer v1) throws Exception { 8 Integer isum = 0; 9 for(Integer i : iList) 10 isum += i; 11 return v1 + isum; 12 } 13 }); 14 System.out.println(result.collect());
accumulator
官方文档描述:
Create an [[org.apache.spark.Accumulator]] variable of a given type, which tasks can "add"
values to using the `add` method. Only the master can access the accumulator's `value`.
函数原型:
def accumulator[T](initialValue: T, accumulatorParam: AccumulatorParam[T]): Accumulator[T]
def accumulator[T](initialValue: T, name: String, accumulatorParam: AccumulatorParam[T])
: Accumulator[T]
累加器是仅仅被相关操作累加的变量,因此可以在并行中被有效地支持。它可以被用来实现计数器和sum。Spark原生地只支持数字类型的累加器,开发者可以添加新类型的支持。如果创建累加器时指定了名字,可以在Spark的UI界面看到。这有利于理解每个执行阶段的进程(对于Python还不支持) 。
累加器通过对一个初始化了的变量v调用SparkContext.accumulator(v)来创建。在集群上运行的任务可以通过add或者”+=”方法在累加器上进行累加操作。但是,它们不能读取它的值。只有驱动程序能够读取它的值,通过累加器的value方法。
累加器通过对一个初始化了的变量v调用SparkContext.accumulator(v)来创建。在集群上运行的任务可以通过add或者”+=”方法在累加器上进行累加操作。但是,它们不能读取它的值。只有驱动程序能够读取它的值,通过累加器的value方法。
源码分析:
def accumulator[T](initialValue: T, name: String)(implicit param: AccumulatorParam[T])
: Accumulator[T] = {
val acc = new Accumulator(initialValue, param, Some(name))
cleaner.foreach(_.registerAccumulatorForCleanup(acc))
acc
}
实例:
1 class VectorAccumulatorParam implements AccumulatorParam<Vector> { 2 @Override 3 //合并两个累加器的值。 4 //参数r1是一个累加数据集合 5 //参数r2是另一个累加数据集合 6 public Vector addInPlace(Vector r1, Vector r2) { 7 r1.addAll(r2); 8 return r1; 9 } 10 @Override 11 //初始值 12 public Vector zero(Vector initialValue) { 13 return initialValue; 14 } 15 @Override 16 //添加额外的数据到累加值中 17 //参数t1是当前累加器的值 18 //参数t2是被添加到累加器的值 19 public Vector addAccumulator(Vector t1, Vector t2) { 20 t1.addAll(t2); 21 return t1; 22 } 23 } 24 List<Integer> data = Arrays.asList(5, 1, 1, 4, 4, 2, 2); 25 JavaRDD<Integer> javaRDD = javaSparkContext.parallelize(data,5); 26 27 final Accumulator<Integer> accumulator = javaSparkContext.accumulator(0); 28 Vector initialValue = new Vector(); 29 for(int i=6;i<9;i++) 30 initialValue.add(i); 31 //自定义累加器 32 final Accumulator accumulator1 = javaSparkContext.accumulator(initialValue,new VectorAccumulatorParam()); 33 JavaRDD<Integer> result = javaRDD.map(new Function<Integer, Integer>() { 34 @Override 35 public Integer call(Integer v1) throws Exception { 36 accumulator.add(1); 37 Vector term = new Vector(); 38 term.add(v1); 39 accumulator1.add(term); 40 return v1; 41 } 42 }); 43 System.out.println(result.collect()); 44 System.out.println("~~~~~~~~~~~~~~~~~~~~~" + accumulator.value()); 45 System.out.println("~~~~~~~~~~~~~~~~~~~~~" + accumulator1.value());
原文引自:https://www.jianshu.com/p/082ef79c63c1?utm_campaign=maleskine&utm_content=note&utm_medium=pc_all_hots&utm_source=recommendation