Thursday, 25 April 2019

Accumulator vs broadcast variable

Accumulator is a variable,variable can be added through cummulative operation or associative operation efficiently achieve the process parallely.and also the driver only can have the value.used for sum.

val accum=sc.accumulator()
val a=1 to 10 toArray
val b=a.map{a=>accum(add(x))}
b.value

a variable shared over the cluster in a cache for read only instead of sharing the copy.

No comments:

Post a Comment