Aiven Amazon S3 Sink Connector

Kafka Connectors have been deprecated and will be removed on October, 1st 2024. Please refer to the deprecation notice for more information.

Aiven Amazon S3 Sink Connector allows you to continuously store the data from your Kafka Topics in Amazon S3. In this guide, we will walk you through creating a Amazon S3 Sink Connector.

Get Started

Create a Kafka Cluster

If you do not have a Kafka cluster and/or topic already, follow these steps to create one.

Prepare the Amazon S3 Environment

If you already have a Amazon S3 environment with the following information, skip this step and continue from the Create The Connector section. Note that the user with the given access keys, should have permission to modify the given bucket.

• aws.access.key.id
• aws.secret.access.key
• aws.s3.bucket.name
• aws.s3.region

Go to AWS S3 Console Create or select a bucket. Note that this bucket name will be used later to configure the connector.

To make this guide simple, we will allow public access to this bucket(not recommended in production).

You can disable public access and allow only following IP’s coming from Upstash :

52.48.149.7
52.213.40.91
174.129.75.41
34.195.190.47
52.58.175.235
18.158.44.120
63.34.151.162
54.247.137.96
3.78.151.126
3.124.80.204
34.236.200.33
44.195.74.73

Aside from bucket name and public access changes, default configurations should be fine for this guide.

Next, we will create a user account with permissions to modify S3 buckets Go to AWS IAM , then “Access Management” and “Users”. Click on “Add Users”.

Give a name to the user and continue with the next screen.

On the “Set Permissions” screen, we will give the “AmazonFullS3Access” to this user.

This gives more permissions than needed. You can create a custom policy with following json for more restrictive policy.

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "VisualEditor0",
            "Effect": "Allow",
            "Action": [
                "s3:PutObject",
                "s3:GetObject",
                "s3:ListBucketMultipartUploads",
                "s3:AbortMultipartUpload",
                "s3:ListMultipartUploadParts"
            ],
            "Resource": "*"
        }
    ]
}

After creating the user, we will go into the details of that user to create a key. Click on the user, then go to the “Security Credentials”. An the “Access Keys” section, click on the “Create access key” button.

We will choose “Application running outside AWS” and create the access key. Don’t forget to store access key id and secret key. We will use these two when creating the connector.

Create the Connector

Go to the Connectors tab, and create your first connector by clicking the New Connector button.

Choose your connector as Aiven Amazon S3 Connector

Enter the required properties.

The advanced screen is for any other configuration that the selected connector supports. At the top of this screen, you can find a link to related documentation. We can proceed with what we have and click the Connect button directly.

Congratulations! You have created an Aiven Amazon S3 Sink Connector.

As you put data into your selected topics, the data will be written into Amazon S3. You can see the data coming from your related bucket in the Amazon Console.