How to set up Confluent Kafka

Atlan supports the API authentication method for fetching metadata from Confluent Kafka. This method uses an API key and API secret to fetch metadata.

Create an API key

To create a resource-specific API key for crawling Confluent Kafka:

1. Log in to your Confluent Cloud instance with a OrganizationAdmin, EnvironmentAdmin, or CloudClusterAdmin role.
2. From the Cloud Console, select your active cluster.
3. Under Cluster Overview in the left menu for your active cluster, click API Keys.
4. In the upper right of the API Keys page, click + Add key.
5. On the Create key page, enter the following details:
   1. For Access Control, under Select Scope for API key, select Granular access to define a specific set of access rules and then click Next to proceed.
   2. For Service Account, click Create a new one and enter the following details:
      1. For New service account name, enter a meaningful name for your API key — for example, Atlan.
      2. (Optional) For Description, add a description for your API key — for example, Atlan crawler connection.
      3. Click Next to proceed.
   3. For Add ACLs to service account, click + Add ACLs to add and allow the following minimum permissions required for your Confluent Kafka resources:
      
      • Cluster: DescribeConfigs
      • Group: Describe
      • Topic: Describe, DescribeConfigs
   4. Once you've added all the permissions, click Next to proceed.
   5. For Create key, under Get your API key, copy or download your API key and secret and store them in a secure location.
      🚨 Careful! The API secret cannot be retrieved later.