com.hazelcast.jet.pipeline

## Class SourceBuilder.Stream<T>

- java.lang.Object
- - com.hazelcast.jet.pipeline.SourceBuilder.Stream<T>

- Type Parameters:
    `T` - type of emitted objects
  Enclosing class:
    SourceBuilder<C>

      public final class SourceBuilder.Stream<T>
      extends Object

  See `SourceBuilder.stream(String, FunctionEx)`.
  Since:
    3.0

- - ### Method Summary

    All Methods Instance Methods Concrete Methods
    Modifier and Type | Method and Description
    --- | ---
    `StreamSource<T>` | `build()` Builds and returns the unbounded stream source.
    `<S> SourceBuilder.FaultTolerant<SourceBuilder.Stream<T>,S>` | `createSnapshotFn(FunctionEx<? super C,? extends S> createSnapshotFn)` Sets the function Jet calls when it's creating a snapshot of the current job state.
    `SourceBuilder.Stream<T>` | `destroyFn(ConsumerEx<? super C> pDestroyFn)` Sets the function that Jet will call when it is done cleaning up after an execution.
    `SourceBuilder.Stream<T>` | `distributed(int preferredLocalParallelism)` Declares that you're creating a distributed source.
    `<T_NEW> SourceBuilder.Stream<T_NEW>` | `fillBufferFn(BiConsumerEx<? super C,? super SourceBuilder.SourceBuffer<T_NEW>> fillBufferFn)` Sets the function that Jet will call whenever it needs more data from your source.

    - ### Methods inherited from class java.lang.Object

      `clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait`

- - ### Method Detail

    - #### fillBufferFn

          @Nonnull
          public <T_NEW> SourceBuilder.Stream<T_NEW> fillBufferFn(@Nonnull
                                                                           BiConsumerEx<? super C,? super SourceBuilder.SourceBuffer<T_NEW>> fillBufferFn)

      Sets the function that Jet will call whenever it needs more data from your source. The function receives the context object obtained from `createFn` and Jet's buffer object. It should add some items to the buffer, ideally those it can produce without making any blocking calls. On any given invocation the function may also choose not to add any items. Jet will automatically employ an exponential backoff strategy to avoid calling your function in a tight loop, if the previous call didn't add any items to the buffer.
      Type Parameters:
        `T_NEW` - type of the emitted items
      Parameters:
        `fillBufferFn` - function that fills the buffer with source data
      Returns:
        this builder with the item type reset to the one inferred from `fillBufferFn`

    - #### destroyFn

          @Nonnull
          public SourceBuilder.Stream<T> destroyFn(@Nonnull
                                                            ConsumerEx<? super C> pDestroyFn)

      Sets the function that Jet will call when it is done cleaning up after an execution. It gives you the opportunity to release any resources that your context object may be holding. Jet also calls this function when the user cancels or restarts the job.

    - #### distributed

          @Nonnull
          public SourceBuilder.Stream<T> distributed(int preferredLocalParallelism)

      Declares that you're creating a distributed source. On each member of the cluster Jet will create as many processors as you specify with the `preferredLocalParallelism` parameter. If you call this, you must ensure that all the source processors are coordinated and not emitting duplicated data. The `createFn` can consult `processorContext.totalParallelism()` and `processorContext.globalProcessorIndex()`. Jet calls `createFn` exactly once with each `globalProcessorIndex` from 0 to `totalParallelism - 1` and you can use this to make all the instances agree on which part of the data to emit.

      If you don't call this method, there will be only one processor instance running on an arbitrary member.

      Parameters:
        `preferredLocalParallelism` - the requested number of processors on each cluster member

    - #### createSnapshotFn

          @Nonnull
          public <S> SourceBuilder.FaultTolerant<SourceBuilder.Stream<T>,S> createSnapshotFn(@Nonnull
                                                                                                      FunctionEx<? super C,? extends S> createSnapshotFn)

      Sets the function Jet calls when it's creating a snapshot of the current job state. This happens in all Jet jobs that have a processing guarantee configured.

      When Jet restarts a job, it first initializes your source as if starting a new job, and then passes the snapshot object you returned here to `restoreSnapshotFn`. After that it starts calling `fillBufferFn`, which must resume emitting the stream from the same item it was about to emit when the snapshot was taken.

      The object you return must be serializable. Each source processor will call the function once per snapshot.

      Here's an example of a fault-tolerant generator of an infinite sequence of integers:

           StreamSource<Integer> source = SourceBuilder
                   .stream("name", processorContext -> new AtomicInteger())
                   .<Integer>fillBufferFn((numToEmit, buffer) -> {
                       for (int i = 0; i < 100; i++) {
                           buffer.add(numToEmit.getAndIncrement());
                       }
                   })
                   .createSnapshotFn(numToEmit -> numToEmit.get())
                   .restoreSnapshotFn((numToEmit, states) -> numToEmit.set(states.get(0)))
                   .build();

      Type Parameters:
        `S` - type of the snapshot object

    - #### build

          @Nonnull
          public StreamSource<T> build()

      Builds and returns the unbounded stream source.

Copyright © 2019. All rights reserved.