In this Release Notes document, you will find the new features and fixes of the
Hazelcast Jet release. The numbers in the square brackets refer to the issue number
in Hazelcast Jet GitHub repository. You can reach to the full descriptions of the
issues at https://github.com/hazelcast/hazelcast-jet/issues/<issue number>.

The label in the brackets refer to the respective module.

** Hazelcast Jet 3.2 **

1. New Features

[pipeline-api] Add test sources and assertion sinks which can be used for testing pipelines [#1457]
[pipeline-api] {map/flatMap/filter}Stateful operator which enables new kinds of stateful processing [#1516]
[core] Add ability to query job metrics through Job API and add option to store metrics for completed jobs [#1459]
[s3] Add sources and sinks for Amazon AWS S3 [#1538]

2. Enhancements

[pipeline-api] Add ability to adapt AggregateOperation1 to Hazelcast Aggregtor [#1513]
[pipeline-api] IMDG map and cache reader is with a much faster, cooperative reader [#1514]
[pipeline-api] IMDG map writer is replaced with faster, cooperative writer [#1539, #1545]
[jet-cli] Add support for YAML configuration to CLI [#1475]
[code-samples] Code samples have been moved to the main Jet repository [#1546]
[core] Support for multigraphs in DAG [#1332]
[core] Cap number of job results in the cluster to 1,000 and add configuration property [#1495]
[core] Replace job ID generator with FlakeID [#1488]
[core] Update Hazelcast to 3.12.3 [#1673]
[core] Add ability to configure both min and max idle time for both cooperative/non-cooperative threads [#1507]
[core] Make it optional to enable/disable metrics for a specific job [#1509]
[core] Add extended information to DOT format output [#1526]

3. Fixes

[pipeline-api] Timestamp transform should match parallelism of upstream transform [#1563]
[pipeline-api] Add missing serializability checks for sources and sinks [#1455]
[pipeline-api] Default to lower parallelism of 2 for non-cooperative context factory [#1571]
[jet-cli] Saved snapshots and running jobs should be sorted by name and sanitized [#1556]
[jet-cli] Fix default group name in cluster.sh [#1577]
[core] Do not use blocking client threads where possible [#1485]
[core] Fix for checking system property for shutdown hook registration [#1461]
[core] Fix for mixing and matching Hazelcast XML and YAML configuration with Jet XML or YAML [#1474]
[core] Avoid creating job resources map if it won't be needed [#1493]
[core] Do not attempt to clear snapshot maps on job starts [#1494]
[core] Fix race which may happen between cancellation and restarting [#1544]
[core] Fix race which may happen during multiple clients calling the same job [#1557]
[core] Fix for races when getting job status and config that could result in job being reported as missing [#1567]
[core] Make snapshots logs less noisy [#1573]
[core] Fix for deadlock when many jobs are restarting at the same time [#1583]
[core] Add more details to exceptions if a job fails with TopologyChangedException [#1595]
[core] Add consistent exception if job.join() is called before or after job is completed [#1600, #1612]

** Hazelcast Jet 3.1 **

1. New Features

[pipeline-api] Enable composite transforms using `apply` operator [#1352]
[pipeline-api] Add support for fault tolerant sources to SourceBuilder API [#1394]
[core] YAML configuration support [#1371]
[integration] Source and sink for Elasticsearch [https://github.com/hazelcast/hazelcast-jet-contrib/tree/master/elasticsearch]
[integration] Source and sink for InfluxDB [https://github.com/hazelcast/hazelcast-jet-contrib/tree/master/influxdb]
[beam] First release of Hazelcast Jet Beam Runner [https://beam.apache.org/documentation/runners/jet/]

2. Enhancements

[pipeline-api] New AggregateOperations: `filtering` and `flatMapping` [#1369]
[pipeline-api] Consecutive transform stages will be merged into one vertex during execution [#1325]
[pipeline-api] Hash-join operator now supports non-distinct keys on the secondary inputs [#1327]
[pipeline-api] Idle time out for streaming sources are now configurable using
               StreamSource.setPartitionIdleTimeout() [#1328]
[pipeline-api] Optimizations for batch grouping operator to release memory eagerly [#1345]
[kafka] Kafka client updated to version 2.2

3. Fixes

[jet-cli] Improve parsing of command line arguments for jet.sh [#1409, #1411, #1449]
[core] Improve handling of errors if operation packets are lost [#1377, #1435]
[core] Configured classloader in Hazelcast Config should be set as the parent of job classloader [#1395]
[core] Job classloader will now print a warning if it's used after a job is completed [#1315, #1446]
[core] Snapshot creation should be handled gracefully even if execution is already completed on a member [#1397]
[core] Added check to ensure that any explicit Hazelcast IMDG dependency has same version as the shaded one [#1412]
[core] Fix StackOverFlowError when a very long path in DAG is used [#1430]
[kafka] Improve Kafka source responsiveness when broker is down [#1429]
[kafka] Improve handling of TimeoutException in Kafka sink [#1433]

4. Breaking Changes

[pipeline-api] Previously it was possible to close the buffer in SourceBuilder when building a streaming source,
               however a streaming source by definition should never close. buffer.close() method now throws when
               called inside a streaming source. [#1394]
[core] Signatures of SourceProcessors.convenientSourceP and convenientTimestampedSourceP
       have been updated to support fault tolerance [#1394]

** Hazelcast Jet 3.0 **

1. New Features

[jet-enterprise] Support for job upgrades [#1154]
[jet-enterprise] Support for lossless recovery through use of Hot Restart [#1159, #1174]
[pipeline-api] Add async support for mapUsingContext [#1193]
[pipeline-api] Support for early window results [#1244]
[core] New Command Line interface, `jet.sh` for job management [#1202]
[core] Implement support for exclusive job execution [#1203]
[core] Update IMDG version to 3.12 [#1343]

2. Enhancements

[pipeline-api] Remove watermarks from a pipeline with no windowing stages [#1091]
[pipeline-api] Optimize mapWithMerging to use less memory [#1107]
[pipeline-api] Improve performance of global aggregation [#1242]
[pipeline-api] Don't require User Code Deployment when reading from remote IMDG journal [#1306]
[core] Automatically restart job if a partition is being migrated when reading from IMap [#1243]
[core] Reuse same maps when taking snapshots [#1104]
[core] Added new metrics for monitoring watermarks and latencies [#1173]
[core] Make minimum idle time in execution service configurable [#1317]
[code-samples] Introduce Tensorflow integration sample [#172]
[code-samples] Introduce Early Window Results sample [#181]


3. Fixes

[core] Support non-smart clients by routing job requests to master [#1129]
[core] Jet client should be automatically shut down when using JetBootstrap [#1225]
[core] Support for JDK9-11 builds [#1223]
[core] Prevent mutation of JobConfig affecting the job after job is submitted [#1233]
[core] Fix race in cooperative worker shutdown [#1296]
[core] Simplify garbage collection for job resources [#1320]
[core] Fix for potential race during initialization of multiple JetInstances [#1322]
[core] Improve exception when JetBootstrap is used without the submit script [#1321]
[pipeline-api] Global session windows should have local parallelism of one [#1300]
[hadoop] Fix creation of RecordWriter to receive its own copy [#1309]

4. Breaking Changes

[pipeline-api] When reading from a source, user must now specify how the timestamps should be handled [#1166]
[pipeline-api] DistributedFunction.constantKey() is removed [#1242]
[pipeline-api] Some methods in ContextFactory have been renamed [#1285]
[pipeline-api] mapUsingIMap and mapUsingReplicated map operations have been simplified and now make use of async calls [#1292]
[pipeline-api] Functional interfaces such as `DistributedFunction` have been renamed as `FunctionEx`. Unused functional interfaces
have been removed [#1293]
[pipeline-api] Window operators now emit `KeyedWindowResult` or `WindowResult` instead of `TimestampedItem` or `TimestampedEntry` [#1244]
[pipeline-api] TimestampedItem and TimestampedEntry have been removed [#1307]
[core-api] Edge.allToOne() now has a constant key parameter [#1242]
[core-api] Watermark policies except `limitingLag` is removed [#1184]
[core-api] maxRetain option from EdgeConfig is removed [#1184]
[core] MetricsConfig.setMetricsForDataStructures is renamed to setMetricsForDataStructuresEnabled [#1201]
[core] jet-submit.sh script has been superseded by `jet.sh submit` command [#1202]
[core] When submitting jobs, if a job with same name already exists an exception will be thrown [#1203]
[core] Group password is deprecated and removed [#1302]

** Hazelcast Jet 0.7.2 **

1. Fixes

[core] Fix for missing config schema file in previous release

** Hazelcast Jet 0.7.1 **

1. New Features

[kubernetes] Full support for running Hazelcast Jet on Kubernetes
[kubernetes] Hazelcast Jet Helm Charts are released

2. Enhancements

[pipeline-api] TopN and BottomN aggregate operations are added [#1140]

3. Fixes

[pipeline-api] All of ClientConfig must be serialized when connecting to remote IMDG sources and sinks [#1147]
[core] Add scaleUpDelay config option to XML config [#1148]
[core] Extract lambdas to avoid allocating new lambdas during processing [#1106, #1249]
[core] Fix for a race during snapshot creation which could result in snapshot not completing [#1101]
[core] Don't materialize distinct keys in SessionWindowP [#1121]
[core] Do not add Watermarks to a stream if there are no windowing stages [#1144]

** Hazelcast Jet 0.7 **

1. New Features

[pipeline-api] Add JMS connectors [#874]
[pipeline-api] New Merge and distinct operations and simpler co-aggregation [#885]
[pipeline-api] Add Avro connector for working with Avro files [#890, #925]
[pipeline-api] Add map/flatMap/filterUsingContext for keyed streams [#892]
[pipeline-api] Add JDBC connector [#931]
[pipeline-api] Add convenient enrichment from IMap and ReplicatedMap [#924]
[pipeline-api] Add a source builder for building custom sources [#989]
[core] Expose metrics through JMX [#926]
[core] Add full job elasticity with graceful shutdown and suspend and resume [#946]

2. Enhancements

[pipeline-api] Add support for DOT file visualization for DAG and Pipeline API [#891]
[pipeline-api] Add name to SinkBuilder [#894]
[pipeline-api] File sources and sinks have been converted to use builders for more convenience [#913]
[pipeline-api] Add exception throwing methods to distributed lambdas [#951]
[pipeline-api] Add "AggregateOperation.andThen" [#959]
[pipeline-api] Add Tuple4 and Tuple5 [#964]
[core] Improve Watermark diagnostics logging [#872]
[core] Processors should be closed as soon as processor is completed [#934]
[core] jobId(), executionId() and jobConfig() are now available on Processor.Context [#934]
[core] Use job name in logs [#939]
[core] Add JetClientConfig which uses the default jet group name and password [#950]
[core] Update Hazelcast to 3.10.5 [#1005]

3. Fixes

[pipeline-api] Using results of a window aggregation in another window shouldn't cause a window shift [#898]
[pipeline-api] mapToOutputFn in aggregations should support returning nulls [#938]
[pipeline-api] Fix for mapWithMerging which shouldn't use a stateful flush function [#968]
[kafka] Handle InterruptException in StreamKafkaP.close() [#870]
[java.util.stream] java.util.stream implementation is removed. It has been superceeded by the Pipeline API [#983]
[spring] Spring module should no longer add dependency to hazelcast when used with spring boot [#977]
[spring] Spring context should be injected to wrapped processors as well [#999]

4. Breaking Changes

[pipeline-api] API for co-aggregation has been simplified
[pipeline-api] File sources and sinks have been replaced with a builder API through Sources.filesBuilder()
[pipeline-api] SinkBuilder now requires a name and takes Processor.Context as input in createFn
[core] AbstractProcessor.setCooperative is removed
[core] Remove tempDir from config as it was unused
[core] Processor.close() no longer receives the exception for the job

** Hazelcast Jet 0.6.1 **

1. New Features

[core] Add support for configuring a custom classloader per job [770]

2. Enhancements

[pipeline-api] Add a noop sink [827]
[core] Optimize in memory layout of snapshot data, reducing memory usage by up to 75% [822]
[core] Update Hazelcast to 3.10-GA [833]
[core] Idle CPU usage improvements when there's a large number of non-cooperative processors [816]
[core] Utility methods to load JetConfig from XML file or resource directly [856]

3. Fixes

[pipeline-api] Fix generic output type for aggregate operations with a custom mapToOutputFn [823]
[pipeline-api] SinkBuilder shouldn't call destroyFn if createFn wasn't called [826]
[core] Don't shrink receive window during lulls which can lead to reduced throughput [798]
[kafka] Kafka source should handle InterruptException in all cases
[hadoop] When matching HDFS hostnames against member hostnames, use alternative ways than just reverse DNS [858]

** Hazelcast Jet 0.6 **

1. New Features

[pipeline-api] Windowing support and overhaul of Pipeline API to work with streaming sources [731]
[pipeline-api] New updating and merging map sinks which update the value in place [621]
[pipeline-api] Custom sink builder support for Pipeline API [762]
[core] Improved job lifecycle features such as naming jobs and getting jobs by name [666]
[core] Ability to restart jobs manually to make use of newly joined members [693]
[spring] Add support for annotation and XML based spring configuration [757]

2. Enhancements

[pipeline-api] Predictable vertex names when convering Pipelines to DAG [616]
[pipeline-api] Improved AggregateOperations.allOf() to be type-safe
[pipeline-api] Add support for stateful transform operations in the pipeline API [744]
[pipeline-api] Add ability to set parallelism and name of each stage separately [731]
[pipeline-api] Add generic Predicate factory methods [695]
[core] Support for idle streaming sources through a configurable idle timeout
       where idle processors are excluded from Watermark coalescing. [660]
[core] Add support for initializing processors with Hazelcast Managed Context [645]
[core] Improved watermark coalescing and processing [649]
[core] Make parallelism information available to Processor.Context [754]
[core] Include Hazelcast Javadocs in Jet distribution [652]
[core] Add Java 9 build support and automatic module names [720]
[core] Add ability to emit file name from file sources [729]
[core] Several Javadoc improvements
[java.util.stream] New DistributedStream instances now can work with Pipeline API sources
[kafka] Kafka version updated to 1.0.0
[hadoop] Hadoop version updated to 2.8.3

3. Fixes

[core] Avoid race between job cancellation and completion [604]
[core] Do not send a join request for a job unless Job.join() is explicitly called [628]
[core] When master node leaves cluster, job would restart even if auto-restart was disabled [683]
[core] When using map source, returning null in a projection would cause rest of the items to be dropped [669]
[core] When DAG fails to deserialize on job submission from client, the exception was not returned to user [794]
[core] Fix for `ClassNotFoundException` when restoring from snapshot when the key class is not on Jet classpath [778]
[core] Assert that always same item offered to Outbox after a rejection [808]
[kafka] Kafka sink should ensure that all records are sent before snapshot [611]
[kafka/hadoop] Package names reorganized to have better support for module systems

4. Breaking Changes

[pipeline-api] Grouping and aggregation in Pipeline API now done through .groupingKey() and .aggregate() methods
[pipeline-api] Package names for Pipeline API have been reorganized.
[core] Job cancellation can now only be done through Job.cancel() rather than Job.getFuture().cancel()
[core] IStreamMap, IStreamList and IStreamCache respectively renamed to IMapJet, IListJet and ICacheJet
[core] Dedicated tryProcessWatermark() method is now responsible for processing Watermarks in a processor
and watermarks will be forwarded automatically once it returns true.
[core] start-jet.sh, submit-jet.sh and other scripts renamed to have prefix "jet" instead.
[core] `CloseableProcessorSupplier` is replaced with a `close()`` method on Processor instead.
[core] `complete` methods on `ProcessorMetaSupplier` and `ProcessorSupplier` have been renamed to `close()`
[java.util.stream] DistributedStream creation moved to DistributedStream class from Map and List instances. We strongly
suggest to switch to Pipeline API instead of java.util.stream API.

** Hazelcast Jet 0.5 **

1. New Features

[core] New Pipeline API with support for group by, hash join and co group [497]
[core] Event journal streaming source for Map and Cache [487]
[core] Improved Jet job lifecycle management including auto-restart [492]
[core] Support for in-memory snapshots and at-least and exactly-once processing [500]

2. Enhancements

[core] Add projection and predicate capability to map reader [490]
[core] Hazelcast version updated to 3.9 [474]
[core] InsertWatermarksProcessor will now log late events [551]
[core] Add preferred local parallelism to all the sources and sinks [552]
[core] Socket source now uses non-blocking IO [554]
[core] Add ProcessorMetaSupplier.complete() method to support global cleanup [571]
[java.util.stream] Support for creating a stream from any source [360]
[kafka] Add snapshotting support for Kafka reader [500]
[kafka] Add optional projection function to Kafka reader [534]
[kafka] Add ability to create own ProducerRecord instances for Kafka writer [588]

3. Fixes

[core] Fix NPE when releasing uninitialized ResourceStore  [478]
[core] Add cache manager in order not to require a dependency on javax.cache [485]
[core] JetClassLoader should not throw on findResource [549]
[core] Fix ignored charset in StreamSocketP [553]
[core] Suppress exception for flow control packet when member has already left cluster [556]
[core] Kafka source should throw when partition count < global parallelism [559]
[core] Traversers.traverseStream should close stream when exhausted [560]
[core] Add missing serializer for Session type [562]
[hdfs] Make sure proper global cleanup is performed on job cancellation [572]

4. Code Samples Improvements

* Overall module reorganization roughly divided to code samples using pipeline API and core API.
* Several examples migrated to use the new pipeline API
* New examples using hash join enrichment and co group
* New code samples illustrating the new event journal and fault tolerance features
* New code samples showing how to use HDFS with java.util.stream

