This document lists the new features, enhancements, fixed issues and, removed or deprecated features for 3.10.x releases. Note that all the Hazelcast IMDG Open Source features and enhancements are also included in this Hazelcast IMDG Enterprise release notes. The numbers in the square brackets refer to the issues in Hazelcast's GitHub repositories.

1. Enhancements

==== 3.10-BETA-2 ====

- Added ID count to flake ID generators statistics. [#12612]
- It is still required to provide a group and password while configuring with Spring: the attribute required should be removed from the tag group both for name and password elements. [#12462]
- A SEVERE error is thrown when a source cluster attempts a connection to a destination cluster and the groupUsername or groupPassword are incorrect in the WAN configuration. The error message should be improved. [#1925]
- The entries dropped on each WAN replication endpoint should be monitored.[#1853]


==== 3.10-BETA-1 ====

- Split-Brain Healing for Hazelcast Data Structures with High-Density (HD) Memory Store: Added implementations to allow the merging of HD Memory Store backed data structures.
- Multiple Users/Roles in a Single Security Mapping: Added support for allowing multiple principals, i.e., users or roles/groups, to be attributed to a security realm. Please see the Authorization section for more information.
- Extended Split-Brain Protection: Split-Brain protection has been implemented for all the Hazelcast data structures (it was only for IMap, ICache, IQueue and ILock before 3.10). Please see the Distributed Data Structures chapter.
- Shorter Split-Brain Detection Window: In addition to the current quorum function, which uses the list of members to determine whether a minimum number of members is available in the cluster, a new quorum implementation has been introduced to shorten the time between the occurrence of a network partitioning and its detection: it takes advantage of information about the cluster health (heartbeats and ICMP pings).
- Extended Merge Policies: Merge policies applied after a Split-Brain syndrome has been implemented for more Hazelcast data structures (it was for IMap, ICache and Replicated Map before 3.10).
- Flake ID Generator: Introduced to overcome the duplicate ID generation in case of a network split, by the deprecated IdGenerator data structure. Please see the FlakeIdGenerator section.
- PN-Counter: Introduced CRDT PN-Counter (Positive-Negative Counter), which allows incrementing and decrementing a counter value and propagating the results to all cluster members. Please see the PN-Counter section.

The following are the other improvements performed to solve the enhancement issues opened by the Hazelcast customers/team.

- Group passwords should not be serialized in ConfigCheck. [#12303]
- Code comments for Hazelcast.shutdownAll() should be improved: it talks about shutting down all running Hazelcast instances in the JVM. However, native clients, which are also Hazelcast instances, are not shut down when this method is called. [#12217]
- An initialization exception should be thrown when Symmetric Encryption is configured on Hazelcast Open Source edition. [#12203]
- IAtomicLong and IAtomicReference miss MergePolicyConfig support in ClientDynamicClusterConfig. [#12126]
- ConfigXmlGenerator masking should be optional: it masks certain fields like passwords. This is unnecessary since the user creating the hazelcast.xml should already know the values or they should be input by the user installing the application. By masking the fields automatically, the user is forced to manually edit the hazelcast.xml creating a manual step. [#12012]
- The CharConverter conversion behavior should be improved: When given an empty string, IllegalArgumentException should be thrown instead of the obscure StringIndexOutOfBoundsException. Also, when given a Number, a Character should be returned instead of an Integer. [#11981]
- The Javadoc for IMap must be enhanced to clarify the value vs. reference semantics. [#11816]
- The class com.hazelcast.query.Predicates contains public factory methods for query predicates. Most of the methods are missing Javadoc. [#11734]
- The value of Main-Class in hazelcast-all.jar should be fixed. The class StartServer should be used instead of ConsoleApp. Currently, when the JAR is executed, the ConsoleApp demo application is started. [#11655]
- The error message for failed client connections should not talk about "address in the config" as multicast discovery may have been used. [#11652]
- Doing a rolling upgrade without a Hazelcast Enterprise key should produce a better error message. [#11602]
- IMap contract should describe how it interacts with Map{Loader/Store}. [#10969]
- New HTTP connections should be logged with DEBUG or a lower level. [#10924]
- Out-of-the-box Hazelcast on Docker experience should be improved due to the "This node is not requested endpoint" errors. [#9219]
- WAN synchronization does not work if no endpoint is preset; therefore it is incompatible with endpoint discovery. If a publisher exists, it should be considered useable even if it has no endpoints. [#1534]
- Assertions should be added to methods which are supposed to be accessed by partition threads only. [#1262]
- There should be a flag to allocate all configured High-Density memory eagerly during Hazelcast startup. [#953]
- High-Density Memory Store backed near cache should be sizeable based on entry count. [#950]
- IMap: Fail-fast when eviction-percentage option is used in combination with High-Density Store. [#665]


2. Fixes

==== 3.10-BETA-2 ====

- ReplicatedMapConfig must not be mutated while constructing read-only variant. [#12620]
- Exception in Node constructor doesn't shutdown services properly. [#12587]
- CacheNotExistsException must be thrown when cache configuration does not exist. [#12593]
- Index configurations must be respected in query caches. [#12577]
- Issues with replicated map and lite members: (i) replicated map cannot be used after lite member promotion, (ii) replicated map runs into NPE after a split-brain healing. [#12537]
- Eviction causes absence of an entry in IMap and MapStore at the same time: just after the method evictAll is invoked, the map is cleared and the eviction is done in a different thread. This causes the objects not being available both IMap and MapStore. [#12455]
- JCache 1.1 TCK: the test org.jsr107.tck.event.CacheListenerTest passes, but there are assertion errors in the log. [#12390]
- An index aware predicate does not invoke its apply method: only the index operation is performed and the apply filter is silently ignored. [#12352]
- Query with predicates on IMap does not use index when running locally. [#12351]
- EventJournal loses data if two members terminate: Scenario is starting a cluster with four members and a client, producing data for the event journal, terminating one member and terminating another member after some time. When checking the total count of events, it is seen that some data is lost. [#12300]
- When minority of members are removed from the network, read/write failures occur on the cluster. [#12240]
- A merge policy cannot be defined for replicated maps using declarative configuration (XML). [#12223]
- All dynamic configurations are sent out in a prejoin operation regardless of the cluster version. [#12151]
- When using PagingPredicate, setting a too big page ends up with IllegalArgumentException. [#12079]
- There is a performance regression on predicate queries for 3.9.1 and 3.8.7 versions. [#12018]
- Each new configuration method introduced in 3.10 must be overridden in the client-side dynamic configuration support class. [#12010]
- For some of the scheduled jobs, the method getLastRunDuration() returns negative values for quick executions. [#11929]
- Exception in ResponseThreadRunnable causes the REST API to fail: When there is an exception while processing a single command, the entire thread is killed. It causes an unrecoverable condition when the system does not process any incoming REST commands, and hence the memory leak (incoming commands are accumulated in blockingQueue). [#11722]
- Accessing a query cache with a reasonable amount of data can result in an NPE. [#11675]
- IMap.add and IMap.remove: EntryListener randomly hangs. [#11470]
- Members not rejoining cluster after an elongated network disconnectivity: Assuming that, initially there is a cluster of 8 members and when the connectivity for a member is broken it gets eliminated from the cluster. The original cluster contains now 7 members and this is working as expected. However, when the connectivity is restored, the disconnected member is not able to join the original cluster and remains isolated. [#11267]
- There is a minor decrease in the performance of indexed searches for 3.7.6 and higher versions. [#11231]
- The code comments for time-to-live expiration is not clear. [#11787]
- When setting the same key value twice, NPE is thrown on the member. [#10556]
- JCache fails to initialize when the type of a key or value is not available on a remote member. [#8972]
- In Hazelcast 3.7, PollOperation invocation is failing to complete due to operation-heartbeat-timeout. [#8831]
- Custom LoggerFactory is instantiating two times. [#5641]
- Threads leak when Hazelcast fails to start due to an exception in the method SSLChannelFactory.create(). [#1978]
- Client exception is received when the initial WAN setup fails: A client makes a connection to a newly-started running cluster. The initial IMap.set call from the client causes the IMap to be created on server side, which also triggers the WAN initialization. However, when WAN initialization fails, the client receives an exception. [#1936]
- Rolling upgrade compatibility is broken in PostJoinMapOperation. [#1898]
- WAN replication: when an IMap existed in the source, it does not get created in the target; it must be created in the target first. [#1866]
- The WAN replication property discovery.useEndpointPrivateAddress is not documented. [#1834]
- Missing permission mappings allows read/modify actions regardless of the configuration. [#1059]


==== 3.10-BETA-1 ====

- Health Monitor reports load statistics incorrectly. [#12190]
- Time-to-live for IMap is not working as documented: outdated warnings should be removed from the code comments. [#12144]
- There are multiple top-level types missing in HazelcastNamespaceHandler, for example serializer. [#12121]
- The full stack trace of QuorumException should be logged into the log file instead of the console. [#12188]
- ConfigXmlGeneratorTest: There are missing tests for many attributes of multiple data structures. [#12119]
- Beta annotation from DurableExecutor classes should be removed. [#12083]
- Hazelcast should not allow to add dynamic configurations for concurrent data structures during rolling upgrades: new configurations have been introduced for IAtomicLong and IAtomicReference for the split-brain healing. There may come more for the split-brain protection; these should not be added dynamically when the cluster is on version 3.9, since old members will fail to process them. [#12000]
- Using MemberAddressProvider with custom discovery strategy SPI does not seem to work. [#11997]
- Hit count gets incremented by two (instead of one) when entry is updated using EntryProcessor. [#11993]
- The objects BufferObjectDataInput and BufferObjectDataOutput are pooled. While being used for serialization, their version may be set, however when cleared (so they can be returned to the pool), their version is not reset to UNKNOWN. [#11900]
- The default network interface selection chooses a wrong interface in some cases. [#11795]
- The memory limit setting for a queue store is ignored and the entries are getting inserted into both the store and the queue, even If the limit is reached. [#11682]
- When trying to integrate Spring and Hazelcast with MapStore implementation, the JdbcTemplate cannot be autowired in the MapStore class. [#11656]
- PER_NODE capacity calculation algorithm is not precise. [#11646]
- Currently ILock operations are not logged at any level. It would be very helpful diagnostic log output can be enabled, with the log lines containing the name of the ILock, the operation (create, lock, unlock, destroy) and the usual timestamp, and thread ID information. [#11622]
- The Echo task used to execute the operations demo in ConsoleApp fails to deserialize because of missing no-arg constructor. [#11612]
- Wrong Hazelcast configuration XSD schema is used when minor/major or minor has more than 1 digit (e.g., 3.1x). [#11586]
- The XSD (hazelcast-spring.xsd) should work independently of the hazelcast-spring.jar. [#11577]
- IMap with MapStore has duplicate keys. [#11462]
- Backups' view of the Ringbuffer differs from that of the primary's in case of an exception is thrown by the store underlying the Ringbuffer. [#11209]
- It does not seem to be possible to configure a list of cipher suites nor a list of SSL/TLS protocols. Client authorization is implemented, but it has to be passed in a populated Properties object. [#10750]
- Transaction could not obtain a lock for the key while calling the method TransactionalMap.getForUpdate(). [#9374]
- JCache fails to initialize when the type of a key or value is not available on a remote member. [#8972]
- There is no way to set the HazelcastClient name and/or InstanceName programmatically. So, the method HazelcastClient.getHazelcastClientByName(String name) does not work. [#7289]
- IMap JMX statistics are evicted together with the map entry: When system starts and caches are populated, and get the hits, the statistics presents the relevant values for localHits and localGetOperationCount attributes. But after 1 hour of idle and some entries are evicted, these attributes become less and less, till 0. Looks like these statistics are stored together with the entry itself and they are evicted together with the entry. [#4321]
- NativeOutOfMemoryError is thrown after sequent Split-Brain/Merge events. [#1902]
- WAN replication queue size statistics is not correct when snapshot.enabled=true. [#1798]
- Behavior of WAN replication when serialization compression is enabled on both clusters is not as expected: WAN serialization format when compression is enabled should be fixe. [#1750]


3. Coverity Fixes

- The methods readObject(...) and writeObject(...) should call the method from the superclass instead of handling the attributeName field themselves in the classes LikePredicate and RegexPredicate. [#11768]
- The field retryCounter is not atomically updated in the method RestartingMemberIterator.retry(Set). [#11750]


4. Behavioral Changes

Up to 3.9.x, the method Config.findCacheConfig looks up a CacheSimpleConfig: if none is found, then null is returned. This behavior has been not like all the other findXXConfig methods in the following ways:

- findXXConfig never returns null, it falls back to defaults, creates the missing data structure configuration and stores it in Config.
- findXXConfig returns a read-only copy of the actual configuration, while findCacheConfig returns the found configuration object itself.

With 3.10, the method Config.findCacheConfig is aligned to behave like the other findXXConfig methods; it now returns a read-only copy and does fallback to defaults. If you who want the old behavior for this method in 3.10.x, you should use the method Config.findCacheConfigOrNull.

5. Known Problems

- PartitionPredicates only works if you upgrade all of your members to 3.9.3. It may not work
  - when running a mixed cluster having members from 3.9.0, 3.9.1, 3.9.2 and 3.9.3, and
  - when rolling up from 3.8.x to 3.9.x.


6. Removed/Deprecated Features
- The data structure IdGenerator is deprecated. It could produce duplicate IDs in case of a network split, even with split-brain protection being enabled. Use Flake ID Generator for an alternative implementation which does not suffer from the mentioned issue.
- Following system properties are deprecated:
  - hazelcast.master.confirmation.interval.seconds
  - hazelcast.max.no.master.confirmation.seconds
  - hazelcast.mc.max.visible.instance.count
