Package com.hazelcast.jet.s3

Class S3Sinks

java.lang.Object
com.hazelcast.jet.s3.S3Sinks
public final class S3Sinks extends Object
Contains factory methods for creating AWS S3 sinks.

  • Method Summary

    Modifier and Type
    Method
    Description
    static <T> com.hazelcast.jet.pipeline.Sink<? super T>
    s3(String bucketName, com.hazelcast.function.SupplierEx<? extends software.amazon.awssdk.services.s3.S3Client> clientSupplier)
    Convenience for s3(String, String, Charset, SupplierEx, FunctionEx) Uses Object.toString() to convert the items to lines.
    static <T> com.hazelcast.jet.pipeline.Sink<? super T>
    s3(String bucketName, String prefix, Charset charset, com.hazelcast.function.SupplierEx<? extends software.amazon.awssdk.services.s3.S3Client> clientSupplier, com.hazelcast.function.FunctionEx<? super T,String> toStringFn)
    Creates an AWS S3 Sink which writes items to files into the given bucket.

    Methods inherited from class java.lang.Object

    clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

  • Method Details

    • s3

      @Nonnull public static <T> com.hazelcast.jet.pipeline.Sink<? super T> s3(@Nonnull String bucketName, @Nonnull com.hazelcast.function.SupplierEx<? extends software.amazon.awssdk.services.s3.S3Client> clientSupplier)
      Convenience for s3(String, String, Charset, SupplierEx, FunctionEx) Uses Object.toString() to convert the items to lines.

    • s3

      @Nonnull public static <T> com.hazelcast.jet.pipeline.Sink<? super T> s3(@Nonnull String bucketName, @Nullable String prefix, @Nonnull Charset charset, @Nonnull com.hazelcast.function.SupplierEx<? extends software.amazon.awssdk.services.s3.S3Client> clientSupplier, @Nonnull com.hazelcast.function.FunctionEx<? super T,String> toStringFn)
      Creates an AWS S3 Sink which writes items to files into the given bucket. Sink converts each item to string using given toStringFn and writes it as a line. The sink creates a file in the bucket for each processor instance. Name of the file will include an user provided prefix (if defined) and processor's global index, for example the processor having the index 2 with prefix my-object- will create the object my-object-2.

      No state is saved to snapshot for this sink. If the job is restarted previously written files will be overwritten.

      The default local parallelism for this sink is 1.

      Here is an example which reads from a map and writes the entries to given bucket using Object.toString() to convert the values to a line. 

      
 Pipeline p = Pipeline.create();
 p.readFrom(Sources.map("map"))
  .writeTo(S3Sinks.s3("bucket", "my-map-", StandardCharsets.UTF_8,
      () -> S3Client.create(),
      Object::toString
 ));
      Type Parameters:
      T - type of the items the sink accepts
      Parameters:
      bucketName - the name of the bucket
      prefix - the prefix to be included in the file name
      charset - the charset to be used when encoding the strings
      clientSupplier - S3 client supplier
      toStringFn - the function which converts each item to its string representation