How to set up Confluent Kafka

Atlan supports the API authentication method for fetching metadata from Confluent Kafka. This method uses an API key and API secret to fetch metadata.

Create an API Key

This section provides steps for creating an API key to access metadata from your Confluent Kafka environment.

Resource-Specific API Key

To create a resource-specific API key for crawling Confluent Kafka, follow these steps:

1. Log in to your Confluent Cloud instance with a OrganizationAdmin, EnvironmentAdmin, or CloudClusterAdmin role.
2. From the Cloud Console, select your active cluster.
3. Under Cluster Overview in the left menu for your active cluster, click API Keys.
4. In the upper right of the API Keys page, click + Add key.
5. On the Create key page, enter the following details:
   1. For Access Control, under Select Scope for API key, select Granular access to define a specific set of access rules, then click Next to proceed.
   2. For Service Account, click Create a new one and enter the following details:
      1. For New service account name, enter a meaningful name for your API key. For example, Atlan.
      2. (Optional) For Description, add a description for your API key. For example, Atlan crawler connection.
      3. Click Next to proceed.
   3. For Add ACLs to service account, click + Add ACLs to add and allow the following minimum permissions required for your Confluent Kafka resources:
      
      • Cluster: DescribeConfigs
      • Group: Describe
      • Topic: Describe, DescribeConfigs
   4. Once you've added all the permissions, click Next to proceed.
   5. For Create key, under Get your API key, copy or download the API key and secret. Make sure to store them securely, as the secret can't be retrieved later.

(Optional) Cloud API Key

To access Kafka metrics(sizeInBytes), you need a Cloud API key. Follow these steps to generate one:

1. Click the hamburger menu (☰) icon in the top right corner to open the menu.
2. In the menu, click API Keys.
3. On the API Key listing screen, click the Add API Key button to add a new API key.
4. On the Select an account for API key screen, select the account appropriate for your service or user account, then click Next.
5. On the Select resource scope for API key screen, choose Cloud resource management, then click Next.
6. On the API details screen, enter the required details:
• Name: Provide a unique and meaningful name for your API key. For example, Atlan Kafka Metrics Key.
• Description: Add a description that illustrates the purpose of the key. For example, Atlan Kafka metrics API key with read-only permissions.
7. Click Create API Key to generate the key.
8. On the API key download screen, copy the API key and secret or click Download API Key to save them as a file. Make sure to store them securely, as the secret can't be retrieved later.
9. Click Complete to finish the API key creation process. You will be able to see the generated API key on the API key listing screen.