Class

com.swoop.spark.accumulators

ByKeyAdditiveAccumulator

Related Doc: package accumulators

Permalink

class ByKeyAdditiveAccumulator[A, B] extends AccumulatorV2[(A, B), Map[A, B]]

An AccumulatorV2-style accumulator for collecting a map of sums.

This accumulator is used in com.swoop.spark.records.AccumulatorMetrics to collect metrics about the execution of record building.

Through the use of the scala.math.Numeric typeclass, the implementation can be used with any numeric type as well as any other value type that is "additive" (the implementation only uses zero and plus).

For example:

- You could define plus to be the equivalent of add for another accumulator, e.g., LongAccumulator and then you can accumulate counts, sums and averages by key.

- You could define plus on a set to be a union operation and then this accumulator will operate as a collect_set by key.

- You could define plus on an array to be concatenation. Combined with a maxLength parameter, this would allow a fast "first N by key" action that completes through a single map stage. The normal way, through a transformation, would require a grouping operation and a shuffle.

A

Key type for the map to aggregate into.

B

Value type, supported by a scala.math.Numeric typeclass.

Linear Supertypes
AccumulatorV2[(A, B), Map[A, B]], Serializable, Serializable, AnyRef, Any
Ordering
  1. Alphabetic
  2. By Inheritance
Inherited
  1. ByKeyAdditiveAccumulator
  2. AccumulatorV2
  3. Serializable
  4. Serializable
  5. AnyRef
  6. Any
Implicitly
  1. by any2stringadd
  2. by StringFormat
  3. by Ensuring
  4. by ArrowAssoc
  1. Hide All
  2. Show All
Visibility
  1. Public
  2. All

Instance Constructors

  1. new ByKeyAdditiveAccumulator()(implicit arg0: Numeric[B])

    Permalink

Value Members

  1. final def !=(arg0: Any): Boolean

    Permalink
    Definition Classes
    AnyRef → Any

  2. final def ##(): Int

    Permalink
    Definition Classes
    AnyRef → Any

  3. def +(other: String): String

    Permalink
    Implicit information
    This member is added by an implicit conversion from ByKeyAdditiveAccumulator[A, B] to any2stringadd[ByKeyAdditiveAccumulator[A, B]] performed by method any2stringadd in scala.Predef.
    Definition Classes
    any2stringadd

  4. def ->[B](y: B): (ByKeyAdditiveAccumulator[A, B], B)

    Permalink
    Implicit information
    This member is added by an implicit conversion from ByKeyAdditiveAccumulator[A, B] to ArrowAssoc[ByKeyAdditiveAccumulator[A, B]] performed by method ArrowAssoc in scala.Predef.
    Definition Classes
    ArrowAssoc
    Annotations
    @inline()

  5. final def ==(arg0: Any): Boolean

    Permalink
    Definition Classes
    AnyRef → Any

  6. def add(v: (A, B)): Unit

    Permalink
    Definition Classes
    ByKeyAdditiveAccumulator → AccumulatorV2

  7. final def asInstanceOf[T0]: T0

    Permalink
    Definition Classes
    Any

  8. def clone(): AnyRef

    Permalink
    Attributes
    protected[java.lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )

  9. def copy(): ByKeyAdditiveAccumulator[A, B]

    Permalink
    Definition Classes
    ByKeyAdditiveAccumulator → AccumulatorV2

  10. def copyAndReset(): ByKeyAdditiveAccumulator[A, B]

    Permalink
    Definition Classes
    ByKeyAdditiveAccumulator → AccumulatorV2

  11. def ensuring(cond: (ByKeyAdditiveAccumulator[A, B]) ⇒ Boolean, msg: ⇒ Any): ByKeyAdditiveAccumulator[A, B]

    Permalink
    Implicit information
    This member is added by an implicit conversion from ByKeyAdditiveAccumulator[A, B] to Ensuring[ByKeyAdditiveAccumulator[A, B]] performed by method Ensuring in scala.Predef.
    Definition Classes
    Ensuring

  12. def ensuring(cond: (ByKeyAdditiveAccumulator[A, B]) ⇒ Boolean): ByKeyAdditiveAccumulator[A, B]

    Permalink
    Implicit information
    This member is added by an implicit conversion from ByKeyAdditiveAccumulator[A, B] to Ensuring[ByKeyAdditiveAccumulator[A, B]] performed by method Ensuring in scala.Predef.
    Definition Classes
    Ensuring

  13. def ensuring(cond: Boolean, msg: ⇒ Any): ByKeyAdditiveAccumulator[A, B]

    Permalink
    Implicit information
    This member is added by an implicit conversion from ByKeyAdditiveAccumulator[A, B] to Ensuring[ByKeyAdditiveAccumulator[A, B]] performed by method Ensuring in scala.Predef.
    Definition Classes
    Ensuring

  14. def ensuring(cond: Boolean): ByKeyAdditiveAccumulator[A, B]

    Permalink
    Implicit information
    This member is added by an implicit conversion from ByKeyAdditiveAccumulator[A, B] to Ensuring[ByKeyAdditiveAccumulator[A, B]] performed by method Ensuring in scala.Predef.
    Definition Classes
    Ensuring

  15. final def eq(arg0: AnyRef): Boolean

    Permalink
    Definition Classes
    AnyRef

  16. def equals(arg0: Any): Boolean

    Permalink
    Definition Classes
    AnyRef → Any

  17. def finalize(): Unit

    Permalink
    Attributes
    protected[java.lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( classOf[java.lang.Throwable] )

  18. def formatted(fmtstr: String): String

    Permalink
    Implicit information
    This member is added by an implicit conversion from ByKeyAdditiveAccumulator[A, B] to StringFormat[ByKeyAdditiveAccumulator[A, B]] performed by method StringFormat in scala.Predef.
    Definition Classes
    StringFormat
    Annotations
    @inline()

  19. final def getClass(): Class[_]

    Permalink
    Definition Classes
    AnyRef → Any

  20. def hashCode(): Int

    Permalink
    Definition Classes
    AnyRef → Any

  21. final def id: Long

    Permalink
    Definition Classes
    AccumulatorV2

  22. final def isInstanceOf[T0]: Boolean

    Permalink
    Definition Classes
    Any

  23. final def isRegistered: Boolean

    Permalink
    Definition Classes
    AccumulatorV2

  24. def isZero: Boolean

    Permalink
    Definition Classes
    ByKeyAdditiveAccumulator → AccumulatorV2

  25. def merge(other: AccumulatorV2[(A, B), Map[A, B]]): Unit

    Permalink
    Definition Classes
    ByKeyAdditiveAccumulator → AccumulatorV2

  26. final def name: Option[String]

    Permalink
    Definition Classes
    AccumulatorV2

  27. final def ne(arg0: AnyRef): Boolean

    Permalink
    Definition Classes
    AnyRef

  28. final def notify(): Unit

    Permalink
    Definition Classes
    AnyRef

  29. final def notifyAll(): Unit

    Permalink
    Definition Classes
    AnyRef

  30. def reset(): Unit

    Permalink
    Definition Classes
    ByKeyAdditiveAccumulator → AccumulatorV2

  31. final def synchronized[T0](arg0: ⇒ T0): T0

    Permalink
    Definition Classes
    AnyRef

  32. def toString(): String

    Permalink
    Definition Classes
    AccumulatorV2 → AnyRef → Any

  33. lazy val value: Map[A, B]

    Permalink

    Returns a synchronized java.util.Map with the same key and value type as the accumulator.

    Returns a synchronized java.util.Map with the same key and value type as the accumulator.

    Definition Classes
    ByKeyAdditiveAccumulator → AccumulatorV2

  34. final def wait(): Unit

    Permalink
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )

  35. final def wait(arg0: Long, arg1: Int): Unit

    Permalink
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )

  36. final def wait(arg0: Long): Unit

    Permalink
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )

  37. final def writeReplace(): Any

    Permalink
    Attributes
    protected
    Definition Classes
    AccumulatorV2

  38. def →[B](y: B): (ByKeyAdditiveAccumulator[A, B], B)

    Permalink
    Implicit information
    This member is added by an implicit conversion from ByKeyAdditiveAccumulator[A, B] to ArrowAssoc[ByKeyAdditiveAccumulator[A, B]] performed by method ArrowAssoc in scala.Predef.
    Definition Classes
    ArrowAssoc

Inherited from AccumulatorV2[(A, B), Map[A, B]]

Inherited from Serializable

Inherited from Serializable

Inherited from AnyRef

Inherited from Any

Inherited by implicit conversion any2stringadd from ByKeyAdditiveAccumulator[A, B] to any2stringadd[ByKeyAdditiveAccumulator[A, B]]

Inherited by implicit conversion StringFormat from ByKeyAdditiveAccumulator[A, B] to StringFormat[ByKeyAdditiveAccumulator[A, B]]

Inherited by implicit conversion Ensuring from ByKeyAdditiveAccumulator[A, B] to Ensuring[ByKeyAdditiveAccumulator[A, B]]

Inherited by implicit conversion ArrowAssoc from ByKeyAdditiveAccumulator[A, B] to ArrowAssoc[ByKeyAdditiveAccumulator[A, B]]

Ungrouped