This document lists the new features, enhancements, fixed issues and, removed or deprecated features for Hazelcast Platform 5.3-BETA-2 release. The numbers in the square brackets refer to the issues and pull requests in Hazelcast's GitHub repository.

## New Features

* **Connector for Kafka Connect source**: Import data from an external system directly into a Hazelcast data pipeline
without the need of a Kafka cluster.
* **Partition-Aware SQL Client**:  Send the SQL commands only to the members having the relevant data, which reduces the network hops and improves the query performances.
* **Connector for MongoDB**: Set up read-through/write-through caching in MongoDB via this connector without writing any code.

## Breaking Changes

* Renamed the `DataLinkFactory` interface as `DataConnection`. [#24224]
* Changed the default cloud coordinator URL from `coordinator.hazelcast.cloud` to `api.viridian.hazelcast.com`.
The default configuration now connects to https://viridian.hazelcast.com/sign-in?next=/[Hazelcast Viridian^] instead of Hazelcast Cloud.
If you want to continue accessing your Hazelcast Cloud clusters, you need to set the `hazelcast.client.cloud.url` property to `https://coordinator.hazelcast.cloud` in your configuration. [#23290]

## Enhancements

### Cloud

* Hazelcast was sending requests to Kubernetes API when deploying an application with embedded Hazelcast and `service-dns` (DNS lookup mode) specified to a Kubernetes cluster. This was causing the requests to be unsuccessful and the application not to start. This mechanism has been improved by creating Kubernetes client only for the DNS lookup mode. [#23883]
* When advanced networking is enabled, the Kubernetes discovery plugin might have been discovering several endpoints (per each port)
for each member's pod. The discovery plugin now matches only the private IP per endpoint, ignoring the port values. [#23766]
* Added support of link:https://aws.amazon.com/about-aws/whats-new/2022/10/amazon-machine-images-support-instance-metadata-service-version-2-default/#:~:text=Posted%20On%3A%20Oct%203%2C%202022,depth%20against%20unauthorized%20metadata%20access.[IMDSv2^] for Hazelcast's AWS Discovery plugin. [#23545]
* Enhanced the handling of errors coming from the Kubernetes API:
  ** Not detecting the intent correctly whenever the cluster is shutdown, so the cluster was not restarted successfully.
  ** Extensive amount of Kubernetes API calls were being logged when Automatic State Management feature is enabled.
  [#23538]
* Added the support of discovering ECS and EC2 members on AWS. See [Deploying a Cluster on Amazon AWS].
https://github.com/hazelcast/hazelcast/pull/22411[#22411]
* Enhanced the experience of using Hazelcast with persistence under Kubernetes by removing requirements for manual interventions. Previously, when there is a restart on such a cluster, the cluster states were needed to be changed manually. For some cases, this might have led to data loss. Now, the shutdown process is automated. See [Configuring Kubernetes] for details. [#21844]

### Storage

* Disk tier option of Tiered Storage is now disabled by default. [#23747]

### Distribution

* Added the Kafka connector to the distribution. [#23312]
* Shaded dependencies for Hazelcast Platform have been combined in a dedicated package (`com.hazelcast.shaded`). [#23124]

### Networking

* Added socket options for per-socket keep-alive configuration: `keep-count`, `keep-idle-seconds`, and `keep-interval-seconds`.
You can set these options using either the advanced network configurations or Hazelcast system properties.
See [Configuring TCP Keep-Alive]. [#23816], [#24050]

### SQL Engine

* Implemented the `GET_DDL` function. [#23595]
* Implemented the `SHOW DATA LINKS` statement to show the connections created by the `CREATE CONNECTIONS` statement. [#24028]
* Added support of providing declarative partitioning strategies based on attributes. These attributes represent specific partition keys for a map.
You can also configure this strategy dynamically. [#23535], [#23954]
* Improved the nested loop joins algorithm by making the `CreateDagVisitor` interface generic. [#23368]
* Implemented the `TO_CHAR` function which is capable of formatting dates, timestamps, and numbers. [#22529]
* Implemented `JSON=VARCHAR` comparison. [#22746]

### Jet Engine

* Each Jet engine job was creating a client for connecting to the remote cluster, resulting in an excessive number of clients.
This has been fixed by introducing a single data store which can be reused for all job instances. [#23472], [#23849]
* Added MongoDB read (source) and write (sink) processors. [#23348]
* Added new messages (`uploadJobMetaData` and `uploadJobMultipart`) to the binary client protocol to be able to
execute non-Java codes for streaming data pipelines, i.e., to submit Jet jobs. [#23212]
* Introduced `JobListener` as an alternative to retrieve a job status via the `Job.getStatus()` method. [#23193]
* Updated the [job API] to add the ability to distinguish the user-cancelled jobs from the failed ones. [#22924]
* Added `flock` to guard all the concurrent `pip` executions (upgrading `pip` and `protobuf` versions) in the Jet-to-Python script. [#22829]

### Security

* Added the TLS configuration property `keyMaterialDuration` which allows using a configurable timeout for caching keyStores and trustStores.
See [TLS Configuration] for the details on how to configure this property. [#23307]

### Serialization

* Hazelcast now allows serializing unsupported types via [zero-configuration] provided that there is an explicit serializer implemented for the unsupported type. [#23449]
* Setting a generic record now fails if the value is not of the same type of generic record. [#23522]
* Introduced clearer exception messages for Generic MapStore, instead of `UndefinedErrorCodeException`. [#22765]

### WAN Replication

* Added the REST endpoint `/hazelcast/rest/wan/sync/progress` which allows tracking the WAN synchronization progress. [#22958]

### Connectors

* Added compound identifier for the `externalName()` method. [#23772]
* Added an option for the MongoDB source to perform reading with a single processor instance. Usually distributing the operations is a best practice; however, the `$function` aggregation method is not supported by the Atlas Serverless instances. In this case, this newly introduced option can be used to to query such instances in one processor. See the option's [https://github.com/hazelcast/hazelcast/blob/master/hazelcast-sql/src/main/java/com/hazelcast/jet/sql/impl/connector/mongodb/Options.java#L77](documentation). [#24198]
* Added additional permission checks to JDBC connectors (source and sink). The previous checks in some cases used only the `jdbc:` name in `ConnectorPermission`. The newly introduced checks get the JDBC URL from existing connection metadata and check if the permission is granted. If want to grant a `ConnectorPermission` to all JDBC URLs, you may use wildcards, e.g., `jdbc:*`. [#23716]
* Added `DataConnection` for Kafka consumers and producers, and for MongoDB [#23886], [#23837]
* Jet engine's connector for MongoDB is now a part of the Hazelcast Platform distribution. [#22565]

### Metrics

* Added name of the job to the [job specific metrics]. [#23744]
* Added Tiered Storage compactor metrics to be exposed by the tools and interfaces such as Management Center, JMX, and diagnostics. [#23643]
* Added a new tag to the CP Subsystem metrics to display the client names for CP sessions on Hazelcast Management Center. [#23387]
* Added entry eviction and expiration counts to local map statistics and metrics. [#23318]

### Other Enhancements

* Introduced the `hazelcast.discovery.public.address.fallback` property to enable smart client
connectivity to Hazelcast clusters deployed on Kubernetes, and having advanced network configuration enabled. [#23707]
* _External Data Store_ references have been renamed as _Data Connections_. This includes the renaming of the `external-data-store` configuration element as `data-connection`. [#24224]
* Added the ability to test whether the connection with a JDBC data link is valid (`testConnection` method for `JdbcDataConnection`). [#23405]
* Upgraded Apache Calcite to 1.32.0. [#22849]

## Fixes

* Fixed an issue where multiple Hazelcast members were able to use the same base directory for Tiered Storage, causing crashes. #5770
* Fixed an issue where the members were crashing while they are loading data into the NATIVE memory and using JSON or Kryo serialization. #5384
* Fixed an issue where the `DELETE FROM` or `UPDATE` functions, when used with fully-qualified names, was failing in a JDBC table. [#23476]
* Fixed an issue where Data Definition Language (DDL) could not be generated for a map with High-Density Memory Store and having no indexes. [#24054]
* Fixed an issue where SQL statements were failing when a class (to determine the fields of a key/value pair) no longer exists but the mapping is still valid. [#24003]
* Fixed an issue where `lastAccessTime` and `expirationTime` was not updated when an entry is accessed via `executeOnEntries`. [#23926]
* Fixed an issue where the Hazelcast configuration was modified by the Jet engine, if the wildcard configuration is used and Jet is enabled; meaning the configurations made by the user was not being reflected. [#23848]
* Fixed an issue where Hazelcast was failing to connect to a database when using Debezium, and the key of a record is null. [#23768]
* Fixed an issue where Debezium connector for MongoDB could not keep the connection after some time. [#23689]
* Fixed an issue where Hazelcast transaction manager was timing out before the timeout value set for the transaction commit. [#23471]
* Fixed an issue where the unordered `mapUsingServiceAsync` was losing items when there are no watermarks. [#23271]
* Fixed an issue where the state of a job was being corrupted if a member is forcefully shut down and the job is restarted from snapshot. [#23201]
* Fixed an issue where disabling metrics and JMX on a Hazelcast client was failing the members to start. [#23360]
* Fixed an issue where a JDBC related exception message was mentioning data source reference instead of the table name. [#23122]
* Fixed an issue where the beginning time of a job was being recorded after the job is submitted, which was causing a race condition. [#23065]
* Fixed an issue where the `hz-cli` shell script was failing to run in bash on Windows systems since it uses a colon-separated `CLASSPATH`
where each path is a Linux-style path. However, Java on Windows expects a semicolon-separated `CLASSPATH` where each path is a Windows-style path.
As a consequence, the `hz-cli` could not be used from bash on Windows. [#22884]
* Fixed a memory leak due to incomplete clean-up of backup replica sync operations. [#22769]
* Fixed an issue where a map was loading old values with map.set() when map store offloading is enabled. [#22602]
* Fixed an issue where a CP Subsystem object could be destroyed without a permission even the object has security and permission configurations enabled. [#22475]
* Fixed an issue where external data store configurations could not be added dynamically. [#22450]
* Fixed an issue where replication over WAN was failing on the source cluster members, when there are multiple batch publishers configured in a single WAN replication. [#22437]
* Fixed an issue where the map entries recovered from persistence were not expiring after their time-to-live durations. [#22279]
* Fixed an issue where Hazelcast 5.1.x members in a Spring Boot application could not establish a connection to each other when deployed
in Kubernetes cluster with Istio Envoy Proxy enabled. [#22256]

== Contributors

We would like to thank the contributors from our open source community
who worked on this release:

* [Ilia Vianni](https://github.com/ivyanni)
