How to set up Redpanda Kafka

Atlan supports the S3 extraction method for fetching metadata from Redpanda Kafka. This method uses Atlan's kafka-extractor tool to fetch metadata.

Create user in Redpanda Kafka

To create a new user for extracting metadata from Redpanda Kafka:

1. Log in to your Redpanda Console and select your active cluster.
2. From the left menu of your cluster's Overview page, click the Security tab.
3. From the upper right of the Security page, click Create User to create a new user.
4. In the Create User dialog, enter the following details: 
   1. For Username, enter a name for the new user — for example, Atlan integration.
   2. For Password, set a password for the new user.
   3. From the Mechanism dropdown, select a SCRAM mechanism.
   4. Click Create to finish creating the new user.
5. From the list of users on the Security page, select the newly created user to edit the associated Access Control Lists (ACLs).
6. In the Edit ACL dialog, enter the following details:
   1. For Topics, click Add Topic ACL to allow the following operations — Describe and DescribeConfigs.
   2. For Consumer Groups, click Add Consumer Group ACL to allow the following operation — Describe.
   3. For Transactional ID, click Add Transactional ID ACL to allow the following operation — Describe.
   4. For Clusters, click Add Cluster ACL to allow the following operations — Describe and DescribeConfigs.
   5. Once you have added all the required operations, click OK to finish setup.