Accumulator is a variable,variable can be added through cummulative operation or associative operation efficiently achieve the process parallely.and also the driver only can have the value.used for sum.
val accum=sc.accumulator()
val a=1 to 10 toArray
val b=a.map{a=>accum(add(x))}
b.value
a variable shared over the cluster in a cache for read only instead of sharing the copy.
val accum=sc.accumulator()
val a=1 to 10 toArray
val b=a.map{a=>accum(add(x))}
b.value
a variable shared over the cluster in a cache for read only instead of sharing the copy.
No comments:
Post a Comment