DEV Community

vivek
vivek

Posted on

Apache kafka-connect with AmazonS3

Apache Kafka connect with amazonS3

Use case

  1. Using windows machine
  2. I am connecting apache kafka connect with amazon-s3
    Kafka version-kafka_2.13-2.8.0
    Version-confluentinc-kafka-connect-s3-10.0.2
    Modified in properties

  3. While starting connector
    connect-standalone.bat config\connect-standalone.properties config\quickstart-s3.properties
    Error log-
    [2021-09-15 15:33:28,056] ERROR Stopping due to error (org.apache.kafka.connect.cli.ConnectStandalone:126)
    java.lang.NoClassDefFoundError: io/confluent/connect/storage/StorageSinkConnectorConfig

File-connectStatdalone.properties

bootstrap.servers=localhost:9092

key.converter=org.apache.kafka.connect.json.JsonConverter
value.converter=org.apache.kafka.connect.json.JsonConverter

key.converter.schemas.enable=true
value.converter.schemas.enable=true

offset.storage.file.filename=/tmp/connect.offsets

offset.storage.file.filename=E:/kafka_2.13-2.8.0/test.txt

Flush much faster than normal, which is useful for testing/debugging

offset.flush.interval.ms=10000

plugin.path=E:/opt/connectors/kafka-connect-s3-4/lib

quickstart-s3.properties

name=s3-sink
connector.class=io.confluent.connect.s3.S3SinkConnector
tasks.max=1
s3.credentials.provider.class=com.amazonaws.auth.DefaultAWSCredentialsProviderChain
topics=s3_bucket
aws.access.key.id=AKIAY63JZIPD65AZTOV7
aws.secret.access.key=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
s3.region=ap-south-1
s3.bucket.name=kafkabucket3
s3.part.size=5242880
flush.size=3

storage.class=io.confluent.connect.s3.storage.S3Storage

format.class=io.confluent.connect.s3.format.avro.AvroFormat

format.class=io.confluent.connect.s3.format.json.JsonFormat
schema.generator.class=io.confluent.connect.storage.hive.schema.DefaultSchemaGenerator
partitioner.class=io.confluent.connect.storage.partitioner.DefaultPartitioner

schema.compatibility=NONE

================

Please help

Top comments (0)