Class/Object

io.radanalytics.silex.rdd.multiplex

MuxRDDFunctions

Related Docs: object MuxRDDFunctions | package multiplex

Permalink

class MuxRDDFunctions[T] extends Serializable

Enhance RDDs with methods for generating multiplexed RDDs

T

the element type of the RDD

// enable multiplexing methods
import io.radanalytics.silex.rdd.multiplex.implicits._
// A boolean predicate on data elements
val pred: Int => Boolean = ....
// pos will contain data elements for which 'pred' was true.
// neg will contain elements for which 'pred' was false.
val (pos, neg) = data.flatMuxPartitions((data: Iterator[Int]) => {
  val pT = scala.collection.mutable.ArrayBuffer.empty[Int]
  val pF = scala.collection.mutable.ArrayBuffer.empty[Int]
  data.foreach { e => (if (pred(e)) pT else pF) += e }
  (pT, pF)
})
Linear Supertypes
Serializable, Serializable, AnyRef, Any
Ordering
  1. Alphabetic
  2. By Inheritance
Inherited
  1. MuxRDDFunctions
  2. Serializable
  3. Serializable
  4. AnyRef
  5. Any
  1. Hide All
  2. Show All
Visibility
  1. Public
  2. All

Instance Constructors

  1. new MuxRDDFunctions(self: RDD[T])(implicit arg0: ClassTag[T])

    Permalink

Value Members

  1. final def !=(arg0: Any): Boolean

    Permalink
    Definition Classes
    AnyRef → Any

  2. final def ##(): Int

    Permalink
    Definition Classes
    AnyRef → Any

  3. final def ==(arg0: Any): Boolean

    Permalink
    Definition Classes
    AnyRef → Any

  4. final def asInstanceOf[T0]: T0

    Permalink
    Definition Classes
    Any

  5. def clone(): AnyRef

    Permalink
    Attributes
    protected[java.lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )

  6. final def eq(arg0: AnyRef): Boolean

    Permalink
    Definition Classes
    AnyRef

  7. def equals(arg0: Any): Boolean

    Permalink
    Definition Classes
    AnyRef → Any

  8. def finalize(): Unit

    Permalink
    Attributes
    protected[java.lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( classOf[java.lang.Throwable] )

  9. def flatMux2Partitions[U1, U2](f: (Iterator[T]) ⇒ (TraversableOnce[U1], TraversableOnce[U2]), persist: StorageLevel = defaultSL)(implicit arg0: ClassTag[U1], arg1: ClassTag[U2]): (RDD[U1], RDD[U2])

    Permalink

    Obtain a tuple of RDDs where the jth component is obtained from flattening the corresponding components returned by applying (f) to each input partition.

    Obtain a tuple of RDDs where the jth component is obtained from flattening the corresponding components returned by applying (f) to each input partition.

    f

    Function maps data from a partition into a tuple of sequences

    persist

    The storage level to apply to the intermediate result returned by (f)

    returns

    The tuple of RDDs, as described above

  10. def flatMux2PartitionsWithIndex[U1, U2](f: (Int, Iterator[T]) ⇒ (TraversableOnce[U1], TraversableOnce[U2]), persist: StorageLevel = defaultSL)(implicit arg0: ClassTag[U1], arg1: ClassTag[U2]): (RDD[U1], RDD[U2])

    Permalink

    Obtain a tuple of RDDs where the jth component is obtained from flattening the corresponding components returned by applying (f) to each input partition and its id.

    Obtain a tuple of RDDs where the jth component is obtained from flattening the corresponding components returned by applying (f) to each input partition and its id.

    f

    Function maps data from a partition and its id into a tuple of sequences

    persist

    The storage level to apply to the intermediate result returned by (f)

    returns

    The tuple of RDDs, as described above

  11. def flatMux3Partitions[U1, U2, U3](f: (Iterator[T]) ⇒ (TraversableOnce[U1], TraversableOnce[U2], TraversableOnce[U3]), persist: StorageLevel = defaultSL)(implicit arg0: ClassTag[U1], arg1: ClassTag[U2], arg2: ClassTag[U3]): (RDD[U1], RDD[U2], RDD[U3])

    Permalink

    Obtain a tuple of RDDs where the jth component is obtained from flattening the corresponding components returned by applying (f) to each input partition.

    Obtain a tuple of RDDs where the jth component is obtained from flattening the corresponding components returned by applying (f) to each input partition.

    f

    Function maps data from a partition into a tuple of sequences

    persist

    The storage level to apply to the intermediate result returned by (f)

    returns

    The tuple of RDDs, as described above

  12. def flatMux3PartitionsWithIndex[U1, U2, U3](f: (Int, Iterator[T]) ⇒ (TraversableOnce[U1], TraversableOnce[U2], TraversableOnce[U3]), persist: StorageLevel = defaultSL)(implicit arg0: ClassTag[U1], arg1: ClassTag[U2], arg2: ClassTag[U3]): (RDD[U1], RDD[U2], RDD[U3])

    Permalink

    Obtain a tuple of RDDs where the jth component is obtained from flattening the corresponding components returned by applying (f) to each input partition and its id.

    Obtain a tuple of RDDs where the jth component is obtained from flattening the corresponding components returned by applying (f) to each input partition and its id.

    f

    Function maps data from a partition and its id into a tuple of sequences

    persist

    The storage level to apply to the intermediate result returned by (f)

    returns

    The tuple of RDDs, as described above

  13. def flatMux4Partitions[U1, U2, U3, U4](f: (Iterator[T]) ⇒ (TraversableOnce[U1], TraversableOnce[U2], TraversableOnce[U3], TraversableOnce[U4]), persist: StorageLevel = defaultSL)(implicit arg0: ClassTag[U1], arg1: ClassTag[U2], arg2: ClassTag[U3], arg3: ClassTag[U4]): (RDD[U1], RDD[U2], RDD[U3], RDD[U4])

    Permalink

    Obtain a tuple of RDDs where the jth component is obtained from flattening the corresponding components returned by applying (f) to each input partition.

    Obtain a tuple of RDDs where the jth component is obtained from flattening the corresponding components returned by applying (f) to each input partition.

    f

    Function maps data from a partition into a tuple of sequences

    persist

    The storage level to apply to the intermediate result returned by (f)

    returns

    The tuple of RDDs, as described above

  14. def flatMux4PartitionsWithIndex[U1, U2, U3, U4](f: (Int, Iterator[T]) ⇒ (TraversableOnce[U1], TraversableOnce[U2], TraversableOnce[U3], TraversableOnce[U4]), persist: StorageLevel = defaultSL)(implicit arg0: ClassTag[U1], arg1: ClassTag[U2], arg2: ClassTag[U3], arg3: ClassTag[U4]): (RDD[U1], RDD[U2], RDD[U3], RDD[U4])

    Permalink

    Obtain a tuple of RDDs where the jth component is obtained from flattening the corresponding components returned by applying (f) to each input partition and its id.

    Obtain a tuple of RDDs where the jth component is obtained from flattening the corresponding components returned by applying (f) to each input partition and its id.

    f

    Function maps data from a partition and its id into a tuple of sequences

    persist

    The storage level to apply to the intermediate result returned by (f)

    returns

    The tuple of RDDs, as described above

  15. def flatMux5Partitions[U1, U2, U3, U4, U5](f: (Iterator[T]) ⇒ (TraversableOnce[U1], TraversableOnce[U2], TraversableOnce[U3], TraversableOnce[U4], TraversableOnce[U5]), persist: StorageLevel = defaultSL)(implicit arg0: ClassTag[U1], arg1: ClassTag[U2], arg2: ClassTag[U3], arg3: ClassTag[U4], arg4: ClassTag[U5]): (RDD[U1], RDD[U2], RDD[U3], RDD[U4], RDD[U5])

    Permalink

    Obtain a tuple of RDDs where the jth component is obtained from flattening the corresponding components returned by applying (f) to each input partition.

    Obtain a tuple of RDDs where the jth component is obtained from flattening the corresponding components returned by applying (f) to each input partition.

    f

    Function maps data from a partition into a tuple of sequences

    persist

    The storage level to apply to the intermediate result returned by (f)

    returns

    The tuple of RDDs, as described above

  16. def flatMux5PartitionsWithIndex[U1, U2, U3, U4, U5](f: (Int, Iterator[T]) ⇒ (TraversableOnce[U1], TraversableOnce[U2], TraversableOnce[U3], TraversableOnce[U4], TraversableOnce[U5]), persist: StorageLevel = defaultSL)(implicit arg0: ClassTag[U1], arg1: ClassTag[U2], arg2: ClassTag[U3], arg3: ClassTag[U4], arg4: ClassTag[U5]): (RDD[U1], RDD[U2], RDD[U3], RDD[U4], RDD[U5])

    Permalink

    Obtain a tuple of RDDs where the jth component is obtained from flattening the corresponding components returned by applying (f) to each input partition and its id.

    Obtain a tuple of RDDs where the jth component is obtained from flattening the corresponding components returned by applying (f) to each input partition and its id.

    f

    Function maps data from a partition and its id into a tuple of sequences

    persist

    The storage level to apply to the intermediate result returned by (f)

    returns

    The tuple of RDDs, as described above

  17. def flatMuxPartitions[U](n: Int, f: (Iterator[T]) ⇒ Seq[TraversableOnce[U]], persist: StorageLevel = defaultSL)(implicit arg0: ClassTag[U]): Seq[RDD[U]]

    Permalink

    Obtain a sequence of (n) RDDs where the jth RDD is obtained from flattening the jth elements returned by applying (f) to each input partition.

    Obtain a sequence of (n) RDDs where the jth RDD is obtained from flattening the jth elements returned by applying (f) to each input partition.

    n

    The length of sequence returned from (f)

    f

    Function maps data from a partition into a sequence of (n) sequences of type U

    persist

    The storage level to apply to the intermediate result returned by (f)

    returns

    The sequence of RDDs, as described above

  18. def flatMuxPartitionsWithIndex[U](n: Int, f: (Int, Iterator[T]) ⇒ Seq[TraversableOnce[U]], persist: StorageLevel = defaultSL)(implicit arg0: ClassTag[U]): Seq[RDD[U]]

    Permalink

    Obtain a sequence of (n) RDDs where the jth RDD is obtained from flattening the jth elements returned by applying (f) to each input partition and its id

    Obtain a sequence of (n) RDDs where the jth RDD is obtained from flattening the jth elements returned by applying (f) to each input partition and its id

    n

    The length of sequence returned from (f)

    f

    Function maps data from a partition and its id into a sequence of (n) sequences of type U

    persist

    The storage level to apply to the intermediate result returned by (f)

    returns

    The sequence of RDDs, as described above

  19. final def getClass(): Class[_]

    Permalink
    Definition Classes
    AnyRef → Any

  20. def hashCode(): Int

    Permalink
    Definition Classes
    AnyRef → Any

  21. final def isInstanceOf[T0]: Boolean

    Permalink
    Definition Classes
    Any

  22. def mux2Partitions[U1, U2](f: (Iterator[T]) ⇒ (U1, U2), persist: StorageLevel = defaultSL)(implicit arg0: ClassTag[U1], arg1: ClassTag[U2]): (RDD[U1], RDD[U2])

    Permalink

    Obtain a tuple of RDDs where jth component is obtained from the corresponding component returned by applying (f) to each partition

    Obtain a tuple of RDDs where jth component is obtained from the corresponding component returned by applying (f) to each partition

    f

    Function maps data from a partition into a tuple of objects

    persist

    The storage level to apply to the intermediate result returned by (f)

    returns

    The tuple of RDDs, as described above

  23. def mux2PartitionsWithIndex[U1, U2](f: (Int, Iterator[T]) ⇒ (U1, U2), persist: StorageLevel = defaultSL)(implicit arg0: ClassTag[U1], arg1: ClassTag[U2]): (RDD[U1], RDD[U2])

    Permalink

    Obtain a tuple of RDDs where jth component is obtained from the corresponding component returned by applying (f) to each partition and its id

    Obtain a tuple of RDDs where jth component is obtained from the corresponding component returned by applying (f) to each partition and its id

    f

    Function maps data from a partition and its id into a tuple of objects

    persist

    The storage level to apply to the intermediate result returned by (f)

    returns

    The tuple of RDDs, as described above

  24. def mux3Partitions[U1, U2, U3](f: (Iterator[T]) ⇒ (U1, U2, U3), persist: StorageLevel = defaultSL)(implicit arg0: ClassTag[U1], arg1: ClassTag[U2], arg2: ClassTag[U3]): (RDD[U1], RDD[U2], RDD[U3])

    Permalink

    Obtain a tuple of RDDs where jth component is obtained from the corresponding component returned by applying (f) to each partition

    Obtain a tuple of RDDs where jth component is obtained from the corresponding component returned by applying (f) to each partition

    f

    Function maps data from a partition into a tuple of objects

    persist

    The storage level to apply to the intermediate result returned by (f)

    returns

    The tuple of RDDs, as described above

  25. def mux3PartitionsWithIndex[U1, U2, U3](f: (Int, Iterator[T]) ⇒ (U1, U2, U3), persist: StorageLevel = defaultSL)(implicit arg0: ClassTag[U1], arg1: ClassTag[U2], arg2: ClassTag[U3]): (RDD[U1], RDD[U2], RDD[U3])

    Permalink

    Obtain a tuple of RDDs where jth component is obtained from the corresponding component returned by applying (f) to each partition and its id

    Obtain a tuple of RDDs where jth component is obtained from the corresponding component returned by applying (f) to each partition and its id

    f

    Function maps data from a partition and its id into a tuple of objects

    persist

    The storage level to apply to the intermediate result returned by (f)

    returns

    The tuple of RDDs, as described above

  26. def mux4Partitions[U1, U2, U3, U4](f: (Iterator[T]) ⇒ (U1, U2, U3, U4), persist: StorageLevel = defaultSL)(implicit arg0: ClassTag[U1], arg1: ClassTag[U2], arg2: ClassTag[U3], arg3: ClassTag[U4]): (RDD[U1], RDD[U2], RDD[U3], RDD[U4])

    Permalink

    Obtain a tuple of RDDs where jth component is obtained from the corresponding component returned by applying (f) to each partition

    Obtain a tuple of RDDs where jth component is obtained from the corresponding component returned by applying (f) to each partition

    f

    Function maps data from a partition into a tuple of objects

    persist

    The storage level to apply to the intermediate result returned by (f)

    returns

    The tuple of RDDs, as described above

  27. def mux4PartitionsWithIndex[U1, U2, U3, U4](f: (Int, Iterator[T]) ⇒ (U1, U2, U3, U4), persist: StorageLevel = defaultSL)(implicit arg0: ClassTag[U1], arg1: ClassTag[U2], arg2: ClassTag[U3], arg3: ClassTag[U4]): (RDD[U1], RDD[U2], RDD[U3], RDD[U4])

    Permalink

    Obtain a tuple of RDDs where jth component is obtained from the corresponding component returned by applying (f) to each partition and its id

    Obtain a tuple of RDDs where jth component is obtained from the corresponding component returned by applying (f) to each partition and its id

    f

    Function maps data from a partition and its id into a tuple of objects

    persist

    The storage level to apply to the intermediate result returned by (f)

    returns

    The tuple of RDDs, as described above

  28. def mux5Partitions[U1, U2, U3, U4, U5](f: (Iterator[T]) ⇒ (U1, U2, U3, U4, U5), persist: StorageLevel = defaultSL)(implicit arg0: ClassTag[U1], arg1: ClassTag[U2], arg2: ClassTag[U3], arg3: ClassTag[U4], arg4: ClassTag[U5]): (RDD[U1], RDD[U2], RDD[U3], RDD[U4], RDD[U5])

    Permalink

    Obtain a tuple of RDDs where jth component is obtained from the corresponding component returned by applying (f) to each partition

    Obtain a tuple of RDDs where jth component is obtained from the corresponding component returned by applying (f) to each partition

    f

    Function maps data from a partition into a tuple of objects

    persist

    The storage level to apply to the intermediate result returned by (f)

    returns

    The tuple of RDDs, as described above

  29. def mux5PartitionsWithIndex[U1, U2, U3, U4, U5](f: (Int, Iterator[T]) ⇒ (U1, U2, U3, U4, U5), persist: StorageLevel = defaultSL)(implicit arg0: ClassTag[U1], arg1: ClassTag[U2], arg2: ClassTag[U3], arg3: ClassTag[U4], arg4: ClassTag[U5]): (RDD[U1], RDD[U2], RDD[U3], RDD[U4], RDD[U5])

    Permalink

    Obtain a tuple of RDDs where jth component is obtained from the corresponding component returned by applying (f) to each partition and its id

    Obtain a tuple of RDDs where jth component is obtained from the corresponding component returned by applying (f) to each partition and its id

    f

    Function maps data from a partition and its id into a tuple of objects

    persist

    The storage level to apply to the intermediate result returned by (f)

    returns

    The tuple of RDDs, as described above

  30. def muxPartitions[U](n: Int, f: (Iterator[T]) ⇒ Seq[U], persist: StorageLevel = defaultSL)(implicit arg0: ClassTag[U]): Seq[RDD[U]]

    Permalink

    Obtain a sequence of (n) RDDs where the jth RDD is obtained from the jth element returned by applying (f) to each input partition.

    Obtain a sequence of (n) RDDs where the jth RDD is obtained from the jth element returned by applying (f) to each input partition.

    n

    The length of sequence returned from (f)

    f

    Function maps data from a partition into a sequence of (n) objects of type U

    persist

    The storage level to apply to the intermediate result returned by (f)

    returns

    The sequence of RDDs, as described above

  31. def muxPartitionsWithIndex[U](n: Int, f: (Int, Iterator[T]) ⇒ Seq[U], persist: StorageLevel = defaultSL)(implicit arg0: ClassTag[U]): Seq[RDD[U]]

    Permalink

    Obtain a sequence of (n) RDDs where the jth RDD is obtained from the jth element returned by applying (f) to each input partition and its id.

    Obtain a sequence of (n) RDDs where the jth RDD is obtained from the jth element returned by applying (f) to each input partition and its id.

    n

    The length of sequence returned from (f)

    f

    Function maps data from a partition, along with that partition's (id) value, into a sequence of (n) objects of type U

    persist

    The storage level to apply to the intermediate result returned by (f)

    returns

    The sequence of RDDs, as described above

  32. final def ne(arg0: AnyRef): Boolean

    Permalink
    Definition Classes
    AnyRef

  33. final def notify(): Unit

    Permalink
    Definition Classes
    AnyRef

  34. final def notifyAll(): Unit

    Permalink
    Definition Classes
    AnyRef

  35. final def synchronized[T0](arg0: ⇒ T0): T0

    Permalink
    Definition Classes
    AnyRef

  36. def toString(): String

    Permalink
    Definition Classes
    AnyRef → Any

  37. final def wait(): Unit

    Permalink
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )

  38. final def wait(arg0: Long, arg1: Int): Unit

    Permalink
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )

  39. final def wait(arg0: Long): Unit

    Permalink
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )

Inherited from Serializable

Inherited from Serializable

Inherited from AnyRef

Inherited from Any

Ungrouped