com.hazelcast.jet.core

## Interface Processor

- All Known Implementing Classes:
    AbstractProcessor

      public interface Processor

  When Jet executes a DAG, it creates one or more instances of `Processor` on each cluster member to do the work of a given vertex. The vertex's `localParallelism` property controls the number of processors per member.

  The processor is a single-threaded processing unit that performs the computation needed to transform zero or more input data streams into zero or more output streams. Each input/output stream corresponds to an edge on the vertex. The correspondence between a stream and an edge is established via the edge's ordinal.

  The special case of zero input streams applies to a source vertex, which gets its data from the environment. The special case of zero output streams applies to a sink vertex, which pushes its data to the environment.

  The processor accepts input from instances of `Inbox` and pushes its output to an instance of `Outbox`.

  See the `isCooperative()` for important restrictions to how the processor should work.

  Since:
    3.0

- - ### Nested Class Summary

    Nested Classes
    Modifier and Type | Interface and Description
    --- | ---
    `static interface` | `Processor.Context` Context passed to the processor in the `init()` call.

  - ### Method Summary

    All Methods Instance Methods Default Methods
    Modifier and Type | Method and Description
    --- | ---
    `default void` | `close()` Called as the last method in the processor lifecycle.
    `default boolean` | `complete()` Called after all the inbound edges' streams are exhausted.
    `default boolean` | `completeEdge(int ordinal)` Called after the edge input with the supplied `ordinal` is exhausted.
    `default boolean` | `finishSnapshotRestore()` Called after a job was restarted from a snapshot and the processor has consumed all the snapshot data.
    `default void` | `init(Outbox outbox, Processor.Context context)` Initializes this processor with the outbox that the processing methods must use to deposit their output items.
    `default boolean` | `isCooperative()` Tells whether this processor is able to participate in cooperative multithreading.
    `default void` | `process(int ordinal, Inbox inbox)` Called with a batch of items retrieved from an inbound edge's stream.
    `default void` | `restoreFromSnapshot(Inbox inbox)` Called when a batch of items is received during the "restore from snapshot" operation.
    `default boolean` | `saveToSnapshot()` Stores its snapshotted state by adding items to the outbox's snapshot bucket.
    `default boolean` | `tryProcess()` This method will be called periodically and only when the current batch of items in the inbox has been exhausted.
    `default boolean` | `tryProcessWatermark(Watermark watermark)` Tries to process the supplied watermark.

- - ### Method Detail

    - #### isCooperative

          default boolean isCooperative()

      Tells whether this processor is able to participate in cooperative multithreading. This means that each invocation of a processing method will take a reasonably small amount of time (up to a millisecond). Violations will manifest themselves as increased latency due to slower switching of processors.

      A cooperative processor should also not attempt any blocking operations, such as I/O operations, waiting for locks/semaphores or sleep operations. Violations of this rule will manifest as less than 100% CPU usage under maximum load (note that this is possible for other reasons too, for example if the network is the bottleneck or if parking time is too high). The processor must also return as soon as the outbox rejects an item (that is when the `offer()` method returns `false`).

      If this processor declares itself cooperative, it will share a thread with other cooperative processors. Otherwise it will run in a dedicated Java thread.

      Jet prefers cooperative processors because they result in a greater overall throughput. A processor should be non-cooperative only if it involves blocking operations, which would cause all other processors on the same shared thread to starve.

      Processor instances of a single vertex are allowed to return different values, but a single processor instance must always return the same value.

      The default implementation returns `true`.

    - #### init

          default void init(@Nonnull
                            Outbox outbox,
                            @Nonnull
                            Processor.Context context)
                     throws Exception

      Initializes this processor with the outbox that the processing methods must use to deposit their output items. This method will be called exactly once and strictly before any calls to processing methods (`process(int, Inbox)` and `complete()`).

      The default implementation does nothing.

      Throws:
        `Exception`

    - #### process

          default void process(int ordinal,
                               @Nonnull
                               Inbox inbox)

      Called with a batch of items retrieved from an inbound edge's stream. The items are in the inbox and this method may process zero or more of them, removing each item after it is processed. Does not remove an item until it is done with it.

      If the method returns with items still present in the inbox, it will be called again before proceeding to call any other methods. There is at least one item in the inbox when this method is called.

      The default implementation throws an exception. It is suitable for source processors.

      Parameters:
        `ordinal` - ordinal of the inbound edge
        `inbox` - the inbox containing the pending items

    - #### tryProcessWatermark

          default boolean tryProcessWatermark(@Nonnull
                                              Watermark watermark)

      Tries to process the supplied watermark. The value is always greater than in the previous call. The watermark is delivered for processing after it has been received from all the edges.

      The implementation may choose to process only partially and return `false`, in which case it will be called again later with the same `timestamp` before any other processing method is called. Before the method returns `true`, it should emit the watermark to the downstream processors.

      The default implementation throws an exception. For any non-sink processor you must provide an implementation that at least forwards the watermark. A sink processor may simply return `true`.

      ### Caution for Jobs With the At-Least-Once Guarantee

      Jet propagates the value of the watermark by sending watermark items interleaved with the regular stream items. If a job configured with the at-least-once processing guarantee gets restarted, the same watermark, like any other stream item, can be delivered again. Therefore the processor may be asked to process a watermark older than the one it had already processed before the restart.
      Parameters:
        `watermark` - watermark to be processed
      Returns:
        `true` if this watermark has now been processed, `false` otherwise.

    - #### tryProcess

          default boolean tryProcess()

      This method will be called periodically and only when the current batch of items in the inbox has been exhausted. It can be used to produce output in the absence of input or to do general maintenance work.

      If the call returns `false`, it will be called again before proceeding to call any other method. Default implementation returns `true`.

      If this method tried to offer to the outbox and the offer call returned false, this method must also return false and retry to offer in the next call.

    - #### completeEdge

          default boolean completeEdge(int ordinal)

      Called after the edge input with the supplied `ordinal` is exhausted. If it returns `false`, it will be called again before proceeding to call any other method.

      If this method tried to offer to the outbox and the offer call returned false, this method must also return false and retry the offer in the next call.

      Returns:
        `true` if the processor is now done completing the edge, `false` otherwise.

    - #### complete

          default boolean complete()

      Called after all the inbound edges' streams are exhausted. If it returns `false`, it will be invoked again until it returns `true`. For example, a streaming source processor will return `false` forever. Unlike other methods which guarantee that no other method is called until they return `true`, `saveToSnapshot()` can be called even though this method returned `false`. If you returned because `Outbox.offer()` returned `false`, make sure to first offer the pending item to the outbox in `saveToSnapshot()` before continuing to offer to snapshot.

      After this method is called, no other processing methods are called on this processor, except for `saveToSnapshot()`.

      Non-cooperative processors are required to return from this method from time to time to give the system a chance to check for snapshot requests and job cancellation. The time the processor spends in this method affects the latency of snapshots and job cancellations.

      Returns:
        `true` if the completing step is now done, `false` otherwise.

    - #### saveToSnapshot

          default boolean saveToSnapshot()

      Stores its snapshotted state by adding items to the outbox's snapshot bucket. If this method returns `false`, it will be called again before proceeding to call any other method.

      This method will only be called after a call to `process()` returns with an empty inbox. After all the input is exhausted, it is also called between `complete()` calls. Once `complete()` returns `true`, this method won't be called anymore.

      The default implementation takes no action and returns `true`.

    - #### restoreFromSnapshot

          default void restoreFromSnapshot(@Nonnull
                                           Inbox inbox)

      Called when a batch of items is received during the "restore from snapshot" operation. The type of items in the inbox is `Map.Entry`. May emit items to the outbox.

      If it returns with items still present in the inbox, it will be called again before proceeding to call any other methods. It is never called with an empty inbox.

      The default implementation throws an exception.

    - #### finishSnapshotRestore

          default boolean finishSnapshotRestore()

      Called after a job was restarted from a snapshot and the processor has consumed all the snapshot data.

      If it returns `false`, it will be called again before proceeding to call any other methods.

      If this method tried to offer to the outbox and the offer call returned false, this method must also return false and retry the offer in the next call.

      The default implementation takes no action and returns `true`.

    - #### close

          default void close()
                      throws Exception

      Called as the last method in the processor lifecycle. It is called whether the job was successful or not, and strictly before `ProcessorSupplier.close(java.lang.Throwable)` is called on this member. The method might get called even if `init(com.hazelcast.jet.core.Outbox, com.hazelcast.jet.core.Processor.Context)` method was not yet called.

      The method will be called right after `complete()` returns `true`, that is before the job is finished. The job might still be running other processors.

      If this method throws an exception, it is logged but it won't be reported as a job failure or cause the job to fail.

      Throws:
        `Exception`

Copyright © 2019. All rights reserved.