com.redhat.et.silex.rdd.multiplex

MuxRDDFunctions

class MuxRDDFunctions[T] extends Serializable

Enhance RDDs with methods for generating multiplexed RDDs

T

the element type of the RDD

// enable multiplexing methods
import com.redhat.et.silex.rdd.multiplex.implicits._

// A boolean predicate on data elements
val pred: Int => Boolean = ....

// pos will contain data elements for which 'pred' was true.
// neg will contain elements for which 'pred' was false.
val (pos, neg) = data.flatMuxPartitions((data: Iterator[Int]) => {
  val pT = scala.collection.mutable.ArrayBuffer.empty[Int]
  val pF = scala.collection.mutable.ArrayBuffer.empty[Int]
  data.foreach { e => (if (pred(e)) pT else pF) += e }
  (pT, pF)
})
Linear Supertypes
Serializable, Serializable, AnyRef, Any
Ordering
  1. Alphabetic
  2. By inheritance
Inherited
  1. MuxRDDFunctions
  2. Serializable
  3. Serializable
  4. AnyRef
  5. Any
  1. Hide All
  2. Show all
Learn more about member selection
Visibility
  1. Public
  2. All

Instance Constructors

  1. new MuxRDDFunctions(self: RDD[T])(implicit arg0: ClassTag[T])

Value Members

  1. final def !=(arg0: AnyRef): Boolean

    Definition Classes
    AnyRef
  2. final def !=(arg0: Any): Boolean

    Definition Classes
    Any
  3. final def ##(): Int

    Definition Classes
    AnyRef → Any
  4. final def ==(arg0: AnyRef): Boolean

    Definition Classes
    AnyRef
  5. final def ==(arg0: Any): Boolean

    Definition Classes
    Any
  6. final def asInstanceOf[T0]: T0

    Definition Classes
    Any
  7. def clone(): AnyRef

    Attributes
    protected[java.lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  8. final def eq(arg0: AnyRef): Boolean

    Definition Classes
    AnyRef
  9. def equals(arg0: Any): Boolean

    Definition Classes
    AnyRef → Any
  10. def finalize(): Unit

    Attributes
    protected[java.lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( classOf[java.lang.Throwable] )
  11. def flatMux2Partitions[U1, U2](f: (Iterator[T]) ⇒ (TraversableOnce[U1], TraversableOnce[U2]), persist: StorageLevel = defaultSL)(implicit arg0: ClassTag[U1], arg1: ClassTag[U2]): (RDD[U1], RDD[U2])

    Obtain a tuple of RDDs where the jth component is obtained from flattening the corresponding components returned by applying (f) to each input partition.

    Obtain a tuple of RDDs where the jth component is obtained from flattening the corresponding components returned by applying (f) to each input partition.

    f

    Function maps data from a partition into a tuple of sequences

    persist

    The storage level to apply to the intermediate result returned by (f)

    returns

    The tuple of RDDs, as described above

  12. def flatMux2PartitionsWithIndex[U1, U2](f: (Int, Iterator[T]) ⇒ (TraversableOnce[U1], TraversableOnce[U2]), persist: StorageLevel = defaultSL)(implicit arg0: ClassTag[U1], arg1: ClassTag[U2]): (RDD[U1], RDD[U2])

    Obtain a tuple of RDDs where the jth component is obtained from flattening the corresponding components returned by applying (f) to each input partition and its id.

    Obtain a tuple of RDDs where the jth component is obtained from flattening the corresponding components returned by applying (f) to each input partition and its id.

    f

    Function maps data from a partition and its id into a tuple of sequences

    persist

    The storage level to apply to the intermediate result returned by (f)

    returns

    The tuple of RDDs, as described above

  13. def flatMux3Partitions[U1, U2, U3](f: (Iterator[T]) ⇒ (TraversableOnce[U1], TraversableOnce[U2], TraversableOnce[U3]), persist: StorageLevel = defaultSL)(implicit arg0: ClassTag[U1], arg1: ClassTag[U2], arg2: ClassTag[U3]): (RDD[U1], RDD[U2], RDD[U3])

    Obtain a tuple of RDDs where the jth component is obtained from flattening the corresponding components returned by applying (f) to each input partition.

    Obtain a tuple of RDDs where the jth component is obtained from flattening the corresponding components returned by applying (f) to each input partition.

    f

    Function maps data from a partition into a tuple of sequences

    persist

    The storage level to apply to the intermediate result returned by (f)

    returns

    The tuple of RDDs, as described above

  14. def flatMux3PartitionsWithIndex[U1, U2, U3](f: (Int, Iterator[T]) ⇒ (TraversableOnce[U1], TraversableOnce[U2], TraversableOnce[U3]), persist: StorageLevel = defaultSL)(implicit arg0: ClassTag[U1], arg1: ClassTag[U2], arg2: ClassTag[U3]): (RDD[U1], RDD[U2], RDD[U3])

    Obtain a tuple of RDDs where the jth component is obtained from flattening the corresponding components returned by applying (f) to each input partition and its id.

    Obtain a tuple of RDDs where the jth component is obtained from flattening the corresponding components returned by applying (f) to each input partition and its id.

    f

    Function maps data from a partition and its id into a tuple of sequences

    persist

    The storage level to apply to the intermediate result returned by (f)

    returns

    The tuple of RDDs, as described above

  15. def flatMux4Partitions[U1, U2, U3, U4](f: (Iterator[T]) ⇒ (TraversableOnce[U1], TraversableOnce[U2], TraversableOnce[U3], TraversableOnce[U4]), persist: StorageLevel = defaultSL)(implicit arg0: ClassTag[U1], arg1: ClassTag[U2], arg2: ClassTag[U3], arg3: ClassTag[U4]): (RDD[U1], RDD[U2], RDD[U3], RDD[U4])

    Obtain a tuple of RDDs where the jth component is obtained from flattening the corresponding components returned by applying (f) to each input partition.

    Obtain a tuple of RDDs where the jth component is obtained from flattening the corresponding components returned by applying (f) to each input partition.

    f

    Function maps data from a partition into a tuple of sequences

    persist

    The storage level to apply to the intermediate result returned by (f)

    returns

    The tuple of RDDs, as described above

  16. def flatMux4PartitionsWithIndex[U1, U2, U3, U4](f: (Int, Iterator[T]) ⇒ (TraversableOnce[U1], TraversableOnce[U2], TraversableOnce[U3], TraversableOnce[U4]), persist: StorageLevel = defaultSL)(implicit arg0: ClassTag[U1], arg1: ClassTag[U2], arg2: ClassTag[U3], arg3: ClassTag[U4]): (RDD[U1], RDD[U2], RDD[U3], RDD[U4])

    Obtain a tuple of RDDs where the jth component is obtained from flattening the corresponding components returned by applying (f) to each input partition and its id.

    Obtain a tuple of RDDs where the jth component is obtained from flattening the corresponding components returned by applying (f) to each input partition and its id.

    f

    Function maps data from a partition and its id into a tuple of sequences

    persist

    The storage level to apply to the intermediate result returned by (f)

    returns

    The tuple of RDDs, as described above

  17. def flatMux5Partitions[U1, U2, U3, U4, U5](f: (Iterator[T]) ⇒ (TraversableOnce[U1], TraversableOnce[U2], TraversableOnce[U3], TraversableOnce[U4], TraversableOnce[U5]), persist: StorageLevel = defaultSL)(implicit arg0: ClassTag[U1], arg1: ClassTag[U2], arg2: ClassTag[U3], arg3: ClassTag[U4], arg4: ClassTag[U5]): (RDD[U1], RDD[U2], RDD[U3], RDD[U4], RDD[U5])

    Obtain a tuple of RDDs where the jth component is obtained from flattening the corresponding components returned by applying (f) to each input partition.

    Obtain a tuple of RDDs where the jth component is obtained from flattening the corresponding components returned by applying (f) to each input partition.

    f

    Function maps data from a partition into a tuple of sequences

    persist

    The storage level to apply to the intermediate result returned by (f)

    returns

    The tuple of RDDs, as described above

  18. def flatMux5PartitionsWithIndex[U1, U2, U3, U4, U5](f: (Int, Iterator[T]) ⇒ (TraversableOnce[U1], TraversableOnce[U2], TraversableOnce[U3], TraversableOnce[U4], TraversableOnce[U5]), persist: StorageLevel = defaultSL)(implicit arg0: ClassTag[U1], arg1: ClassTag[U2], arg2: ClassTag[U3], arg3: ClassTag[U4], arg4: ClassTag[U5]): (RDD[U1], RDD[U2], RDD[U3], RDD[U4], RDD[U5])

    Obtain a tuple of RDDs where the jth component is obtained from flattening the corresponding components returned by applying (f) to each input partition and its id.

    Obtain a tuple of RDDs where the jth component is obtained from flattening the corresponding components returned by applying (f) to each input partition and its id.

    f

    Function maps data from a partition and its id into a tuple of sequences

    persist

    The storage level to apply to the intermediate result returned by (f)

    returns

    The tuple of RDDs, as described above

  19. def flatMuxPartitions[U](n: Int, f: (Iterator[T]) ⇒ Seq[TraversableOnce[U]], persist: StorageLevel = defaultSL)(implicit arg0: ClassTag[U]): Seq[RDD[U]]

    Obtain a sequence of (n) RDDs where the jth RDD is obtained from flattening the jth elements returned by applying (f) to each input partition.

    Obtain a sequence of (n) RDDs where the jth RDD is obtained from flattening the jth elements returned by applying (f) to each input partition.

    n

    The length of sequence returned from (f)

    f

    Function maps data from a partition into a sequence of (n) sequences of type U

    persist

    The storage level to apply to the intermediate result returned by (f)

    returns

    The sequence of RDDs, as described above

  20. def flatMuxPartitionsWithIndex[U](n: Int, f: (Int, Iterator[T]) ⇒ Seq[TraversableOnce[U]], persist: StorageLevel = defaultSL)(implicit arg0: ClassTag[U]): Seq[RDD[U]]

    Obtain a sequence of (n) RDDs where the jth RDD is obtained from flattening the jth elements returned by applying (f) to each input partition and its id

    Obtain a sequence of (n) RDDs where the jth RDD is obtained from flattening the jth elements returned by applying (f) to each input partition and its id

    n

    The length of sequence returned from (f)

    f

    Function maps data from a partition and its id into a sequence of (n) sequences of type U

    persist

    The storage level to apply to the intermediate result returned by (f)

    returns

    The sequence of RDDs, as described above

  21. final def getClass(): Class[_]

    Definition Classes
    AnyRef → Any
  22. def hashCode(): Int

    Definition Classes
    AnyRef → Any
  23. final def isInstanceOf[T0]: Boolean

    Definition Classes
    Any
  24. def mux2Partitions[U1, U2](f: (Iterator[T]) ⇒ (U1, U2), persist: StorageLevel = defaultSL)(implicit arg0: ClassTag[U1], arg1: ClassTag[U2]): (RDD[U1], RDD[U2])

    Obtain a tuple of RDDs where jth component is obtained from the corresponding component returned by applying (f) to each partition

    Obtain a tuple of RDDs where jth component is obtained from the corresponding component returned by applying (f) to each partition

    f

    Function maps data from a partition into a tuple of objects

    persist

    The storage level to apply to the intermediate result returned by (f)

    returns

    The tuple of RDDs, as described above

  25. def mux2PartitionsWithIndex[U1, U2](f: (Int, Iterator[T]) ⇒ (U1, U2), persist: StorageLevel = defaultSL)(implicit arg0: ClassTag[U1], arg1: ClassTag[U2]): (RDD[U1], RDD[U2])

    Obtain a tuple of RDDs where jth component is obtained from the corresponding component returned by applying (f) to each partition and its id

    Obtain a tuple of RDDs where jth component is obtained from the corresponding component returned by applying (f) to each partition and its id

    f

    Function maps data from a partition and its id into a tuple of objects

    persist

    The storage level to apply to the intermediate result returned by (f)

    returns

    The tuple of RDDs, as described above

  26. def mux3Partitions[U1, U2, U3](f: (Iterator[T]) ⇒ (U1, U2, U3), persist: StorageLevel = defaultSL)(implicit arg0: ClassTag[U1], arg1: ClassTag[U2], arg2: ClassTag[U3]): (RDD[U1], RDD[U2], RDD[U3])

    Obtain a tuple of RDDs where jth component is obtained from the corresponding component returned by applying (f) to each partition

    Obtain a tuple of RDDs where jth component is obtained from the corresponding component returned by applying (f) to each partition

    f

    Function maps data from a partition into a tuple of objects

    persist

    The storage level to apply to the intermediate result returned by (f)

    returns

    The tuple of RDDs, as described above

  27. def mux3PartitionsWithIndex[U1, U2, U3](f: (Int, Iterator[T]) ⇒ (U1, U2, U3), persist: StorageLevel = defaultSL)(implicit arg0: ClassTag[U1], arg1: ClassTag[U2], arg2: ClassTag[U3]): (RDD[U1], RDD[U2], RDD[U3])

    Obtain a tuple of RDDs where jth component is obtained from the corresponding component returned by applying (f) to each partition and its id

    Obtain a tuple of RDDs where jth component is obtained from the corresponding component returned by applying (f) to each partition and its id

    f

    Function maps data from a partition and its id into a tuple of objects

    persist

    The storage level to apply to the intermediate result returned by (f)

    returns

    The tuple of RDDs, as described above

  28. def mux4Partitions[U1, U2, U3, U4](f: (Iterator[T]) ⇒ (U1, U2, U3, U4), persist: StorageLevel = defaultSL)(implicit arg0: ClassTag[U1], arg1: ClassTag[U2], arg2: ClassTag[U3], arg3: ClassTag[U4]): (RDD[U1], RDD[U2], RDD[U3], RDD[U4])

    Obtain a tuple of RDDs where jth component is obtained from the corresponding component returned by applying (f) to each partition

    Obtain a tuple of RDDs where jth component is obtained from the corresponding component returned by applying (f) to each partition

    f

    Function maps data from a partition into a tuple of objects

    persist

    The storage level to apply to the intermediate result returned by (f)

    returns

    The tuple of RDDs, as described above

  29. def mux4PartitionsWithIndex[U1, U2, U3, U4](f: (Int, Iterator[T]) ⇒ (U1, U2, U3, U4), persist: StorageLevel = defaultSL)(implicit arg0: ClassTag[U1], arg1: ClassTag[U2], arg2: ClassTag[U3], arg3: ClassTag[U4]): (RDD[U1], RDD[U2], RDD[U3], RDD[U4])

    Obtain a tuple of RDDs where jth component is obtained from the corresponding component returned by applying (f) to each partition and its id

    Obtain a tuple of RDDs where jth component is obtained from the corresponding component returned by applying (f) to each partition and its id

    f

    Function maps data from a partition and its id into a tuple of objects

    persist

    The storage level to apply to the intermediate result returned by (f)

    returns

    The tuple of RDDs, as described above

  30. def mux5Partitions[U1, U2, U3, U4, U5](f: (Iterator[T]) ⇒ (U1, U2, U3, U4, U5), persist: StorageLevel = defaultSL)(implicit arg0: ClassTag[U1], arg1: ClassTag[U2], arg2: ClassTag[U3], arg3: ClassTag[U4], arg4: ClassTag[U5]): (RDD[U1], RDD[U2], RDD[U3], RDD[U4], RDD[U5])

    Obtain a tuple of RDDs where jth component is obtained from the corresponding component returned by applying (f) to each partition

    Obtain a tuple of RDDs where jth component is obtained from the corresponding component returned by applying (f) to each partition

    f

    Function maps data from a partition into a tuple of objects

    persist

    The storage level to apply to the intermediate result returned by (f)

    returns

    The tuple of RDDs, as described above

  31. def mux5PartitionsWithIndex[U1, U2, U3, U4, U5](f: (Int, Iterator[T]) ⇒ (U1, U2, U3, U4, U5), persist: StorageLevel = defaultSL)(implicit arg0: ClassTag[U1], arg1: ClassTag[U2], arg2: ClassTag[U3], arg3: ClassTag[U4], arg4: ClassTag[U5]): (RDD[U1], RDD[U2], RDD[U3], RDD[U4], RDD[U5])

    Obtain a tuple of RDDs where jth component is obtained from the corresponding component returned by applying (f) to each partition and its id

    Obtain a tuple of RDDs where jth component is obtained from the corresponding component returned by applying (f) to each partition and its id

    f

    Function maps data from a partition and its id into a tuple of objects

    persist

    The storage level to apply to the intermediate result returned by (f)

    returns

    The tuple of RDDs, as described above

  32. def muxPartitions[U](n: Int, f: (Iterator[T]) ⇒ Seq[U], persist: StorageLevel = defaultSL)(implicit arg0: ClassTag[U]): Seq[RDD[U]]

    Obtain a sequence of (n) RDDs where the jth RDD is obtained from the jth element returned by applying (f) to each input partition.

    Obtain a sequence of (n) RDDs where the jth RDD is obtained from the jth element returned by applying (f) to each input partition.

    n

    The length of sequence returned from (f)

    f

    Function maps data from a partition into a sequence of (n) objects of type U

    persist

    The storage level to apply to the intermediate result returned by (f)

    returns

    The sequence of RDDs, as described above

  33. def muxPartitionsWithIndex[U](n: Int, f: (Int, Iterator[T]) ⇒ Seq[U], persist: StorageLevel = defaultSL)(implicit arg0: ClassTag[U]): Seq[RDD[U]]

    Obtain a sequence of (n) RDDs where the jth RDD is obtained from the jth element returned by applying (f) to each input partition and its id.

    Obtain a sequence of (n) RDDs where the jth RDD is obtained from the jth element returned by applying (f) to each input partition and its id.

    n

    The length of sequence returned from (f)

    f

    Function maps data from a partition, along with that partition's (id) value, into a sequence of (n) objects of type U

    persist

    The storage level to apply to the intermediate result returned by (f)

    returns

    The sequence of RDDs, as described above

  34. final def ne(arg0: AnyRef): Boolean

    Definition Classes
    AnyRef
  35. final def notify(): Unit

    Definition Classes
    AnyRef
  36. final def notifyAll(): Unit

    Definition Classes
    AnyRef
  37. final def synchronized[T0](arg0: ⇒ T0): T0

    Definition Classes
    AnyRef
  38. def toString(): String

    Definition Classes
    AnyRef → Any
  39. final def wait(): Unit

    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  40. final def wait(arg0: Long, arg1: Int): Unit

    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  41. final def wait(arg0: Long): Unit

    Definition Classes
    AnyRef
    Annotations
    @throws( ... )

Inherited from Serializable

Inherited from Serializable

Inherited from AnyRef

Inherited from Any

Ungrouped