com.hazelcast.jet.pipeline

## Class FileSourceBuilder

- java.lang.Object
- - com.hazelcast.jet.pipeline.FileSourceBuilder

-     public final class FileSourceBuilder
      extends Object

  Builder for a file source which reads lines from files in a directory (but not its subdirectories) and emits output object created by `mapOutputFn`
  Since:
    3.0

- - ### Method Summary

    All Methods Instance Methods Concrete Methods
    Modifier and Type | Method and Description
    --- | ---
    `BatchSource<String>` | `build()` Convenience for `build(BiFunctionEx)`.
    `<T> BatchSource<T>` | `build(BiFunctionEx<String,String,? extends T> mapOutputFn)` Builds a custom file `BatchSource` with supplied components and the output function `mapOutputFn`.
    `StreamSource<String>` | `buildWatcher()` Convenience for `buildWatcher(BiFunctionEx)`.
    `<T> StreamSource<T>` | `buildWatcher(BiFunctionEx<String,String,? extends T> mapOutputFn)` Builds a source that emits a stream of lines of text coming from files in the watched directory (but not its subdirectories).
    `FileSourceBuilder` | `charset(Charset charset)` Sets the character set used to encode the files.
    `FileSourceBuilder` | `glob(String glob)` Sets the globbing mask, see `getPathMatcher()`.
    `FileSourceBuilder` | `sharedFileSystem(boolean sharedFileSystem)` Sets if files are in a shared storage visible to all members.

    - ### Methods inherited from class java.lang.Object

      `clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait`

- - ### Method Detail

    - #### glob

          public FileSourceBuilder glob(@Nonnull
                                        String glob)

      Sets the globbing mask, see `getPathMatcher()`. Default value is `"*"` which means all files.

    - #### sharedFileSystem

          public FileSourceBuilder sharedFileSystem(boolean sharedFileSystem)

      Sets if files are in a shared storage visible to all members. Default value is `false`

      If `sharedFileSystem` is `true`, Jet will assume all members see the same files. They will split the work so that each member will read a part of the files. If `sharedFileSystem` is `false`, each member will read all files in the directory, assuming the are local.

    - #### charset

          public FileSourceBuilder charset(@Nonnull
                                           Charset charset)

      Sets the character set used to encode the files. Default value is `StandardCharsets.UTF_8`.

      Setting this component does not have any effect if builder is used by Avro module.

    - #### build

          public BatchSource<String> build()

      Convenience for `build(BiFunctionEx)`. Source emits lines to downstream without any transformation.

    - #### build

          public <T> BatchSource<T> build(BiFunctionEx<String,String,? extends T> mapOutputFn)

      Builds a custom file `BatchSource` with supplied components and the output function `mapOutputFn`.

      The source does not save any state to snapshot. If the job is restarted, it will re-emit all entries.

      Any `IOException` will cause the job to fail. The files must not change while being read; if they do, the behavior is unspecified.

      The default local parallelism for this processor is 2 (or 1 if just 1 CPU is available).

      Type Parameters:
        `T` - the type of the items the source emits
      Parameters:
        `mapOutputFn` - the function which creates output object from each line. Gets the filename and line as parameters

    - #### buildWatcher

          public StreamSource<String> buildWatcher()

      Convenience for `buildWatcher(BiFunctionEx)`.

    - #### buildWatcher

          public <T> StreamSource<T> buildWatcher(BiFunctionEx<String,String,? extends T> mapOutputFn)

      Builds a source that emits a stream of lines of text coming from files in the watched directory (but not its subdirectories). It will emit only new contents added after startup: both new files and new content appended to existing ones.

      If, during the scanning phase, the source observes a file that doesn't end with a newline, it will assume that there is a line just being written. This line won't appear in its output.

      The source completes when the directory is deleted. However, in order to delete the directory, all files in it must be deleted and if you delete a file that is currently being read from, the job may encounter an `IOException`. The directory must be deleted on all nodes if `sharedFileSystem` is `false`.

      Any `IOException` will cause the job to fail.

      The source does not save any state to snapshot. If the job is restarted, lines added after the restart will be emitted, which gives at-most-once behavior.

      The default local parallelism for this processor is 2 (or 1 if just 1 CPU is available).

      ### Limitation on Windows

      On Windows the `WatchService` is not notified of appended lines until the file is closed. If the file-writing process keeps the file open while appending, the processor may fail to observe the changes. It will be notified if any process tries to open that file, such as looking at the file in Explorer. This holds for Windows 10 with the NTFS file system and might change in future. You are advised to do your own testing on your target Windows platform.

      ### Use the latest JRE

      The underlying JDK API (`WatchService`) has a history of unreliability and this source may experience infinite blocking, missed, or duplicate events as a result. Such problems may be resolved by upgrading the JRE to the latest version.

      ### Appending lines using an text editor

      If you're testing this source, you might think of using a text editor to append the lines. However, it might not work as expected because some editors write to a temp file and then rename it or append extra newline character at the end which gets overwritten if more text is added in the editor. Best way to append is to use `echo text >> yourfile`.
      Type Parameters:
        `T` - the type of the items the source emits
      Parameters:
        `mapOutputFn` - the function which creates output object from each line. Gets the filename and line as parameters

Copyright © 2019. All rights reserved.