com.hazelcast.jet.hadoop

## Class HdfsSources

- java.lang.Object
- - com.hazelcast.jet.hadoop.HdfsSources

-     public final class HdfsSources
      extends Object

  Contains factory methods for Apache Hadoop HDFS sources.

- - ### Method Summary

    All Methods Static Methods Concrete Methods
    Modifier and Type | Method and Description
    --- | ---
    `static <K,V> BatchSource<Map.Entry<K,V>>` | `hdfs(org.apache.hadoop.mapred.JobConf jobConf)` Convenience for `hdfs(JobConf, DistributedBiFunction)` with `Map.Entry` as its output type.
    `static <K,V,E> BatchSource<E>` | `hdfs(org.apache.hadoop.mapred.JobConf jobConf, DistributedBiFunction<K,V,E> projectionFn)` Returns a source that reads records from Apache Hadoop HDFS and emits the results of transforming each record (a key-value pair) with the supplied mapping function.

    - ### Methods inherited from class java.lang.Object

      `clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait`

- - ### Method Detail

    - #### hdfs

          @Nonnull
          public static <K,V,E> BatchSource<E> hdfs(@Nonnull
                                                             org.apache.hadoop.mapred.JobConf jobConf,
                                                             @Nonnull
                                                             DistributedBiFunction<K,V,E> projectionFn)

      Returns a source that reads records from Apache Hadoop HDFS and emits the results of transforming each record (a key-value pair) with the supplied mapping function.

      This source splits and balances the input data among Jet processors, doing its best to achieve data locality. To this end the Jet cluster topology should be aligned with Hadoop's — on each Hadoop member there should be a Jet member.

      Default local parallelism for this processor is 2 (or less if less CPUs are available).

      This source does not save any state to snapshot. If the job is restarted, all entries will be emitted again.

      Type Parameters:
        `K` - key type of the records
        `V` - value type of the records
        `E` - the type of the emitted value
      Parameters:
        `jobConf` - JobConf for reading files with the appropriate input format and path
        `projectionFn` - function to create output objects from key and value. If the projection returns a `null` for an item, that item will be filtered out

    - #### hdfs

          @Nonnull
          public static <K,V> BatchSource<Map.Entry<K,V>> hdfs(@Nonnull
                                                                        org.apache.hadoop.mapred.JobConf jobConf)

      Convenience for `hdfs(JobConf, DistributedBiFunction)` with `Map.Entry` as its output type.

Copyright © 2018. All rights reserved.