Apache Kafka connect with amazonS3
Use case
- Using windows machine
I am connecting apache kafka connect with amazon-s3
Kafka version-kafka_2.13-2.8.0
Version-confluentinc-kafka-connect-s3-10.0.2
Modified in propertiesWhile starting connector
connect-standalone.bat config\connect-standalone.properties config\quickstart-s3.properties
Error log-
[2021-09-15 15:33:28,056] ERROR Stopping due to error (org.apache.kafka.connect.cli.ConnectStandalone:126)
java.lang.NoClassDefFoundError: io/confluent/connect/storage/StorageSinkConnectorConfig
File-connectStatdalone.properties
bootstrap.servers=localhost:9092
key.converter=org.apache.kafka.connect.json.JsonConverter
value.converter=org.apache.kafka.connect.json.JsonConverter
key.converter.schemas.enable=true
value.converter.schemas.enable=true
offset.storage.file.filename=/tmp/connect.offsets
offset.storage.file.filename=E:/kafka_2.13-2.8.0/test.txt
Flush much faster than normal, which is useful for testing/debugging
offset.flush.interval.ms=10000
plugin.path=E:/opt/connectors/kafka-connect-s3-4/lib
quickstart-s3.properties
name=s3-sink
connector.class=io.confluent.connect.s3.S3SinkConnector
tasks.max=1
s3.credentials.provider.class=com.amazonaws.auth.DefaultAWSCredentialsProviderChain
topics=s3_bucket
aws.access.key.id=AKIAY63JZIPD65AZTOV7
aws.secret.access.key=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
s3.region=ap-south-1
s3.bucket.name=kafkabucket3
s3.part.size=5242880
flush.size=3
storage.class=io.confluent.connect.s3.storage.S3Storage
format.class=io.confluent.connect.s3.format.avro.AvroFormat
format.class=io.confluent.connect.s3.format.json.JsonFormat
schema.generator.class=io.confluent.connect.storage.hive.schema.DefaultSchemaGenerator
partitioner.class=io.confluent.connect.storage.partitioner.DefaultPartitioner
schema.compatibility=NONE
================
Please help
Top comments (0)