How to set up Apache Kafka

Atlan supports the basic authentication method for fetching metadata from Apache Kafka. This method uses a username and password to fetch metadata. 

Create user in Apache Kafka

We recommend creating a new user in Apache Kafka for integration with Atlan. Next, you will need to implement authentication and authorization for that user on your Apache Kafka brokers.

To enable user authentication and authorization on the broker side, you will need to perform two steps on each broker:

• Configure valid credentials
• Configure the proper security protocol and authorizer implementation

You can configure the broker with its user credentials and then authorize the client's user credentials. These credentials, along with the login module specification, are stored in a JAAS login configuration file.

Here is the JAAS file you can run:

KafkaServer {
    org.apache.kafka.common.security.plain.PlainLoginModule required
    username="<kafka admin username>"
    password="<kafka admin password>"
    user_<username> = “<password>”;
};

• Replace username with the username you want to use in Atlan.
• Replace password with the password you want to use in Atlan.

Pass in this file as a JVM configuration option when running the broker, using -Djava.security.auth.login.config=[path_to_jaas_file]:

• [path_to_jaas_file] can be something like: config/jaas-kafka-server.conf
• This can be done by setting the KAFKA_OPTS environment variable, for example:

  export KAFKA_OPTS="-Djava.security.auth.login.config=<path-to-jaas-file>/jaas-kafka-server.conf"

Set permissions

The following minimum permissions are required for these resources:

• Topic: Describe, DescribeConfigs
• Group: Describe
• Cluster: DescribeConfigs

To set the minimum permissions required to crawl Apache Kafka:

# Give permission to topics
./bin/kafka-acls.sh --topic '*' --add --allowprincipals user:atlan --operations Read,Describe,DescribeConfigs --allow-host '*' --config /<broker-config-path>/server.properties

# Give permission to groups
./bin/kafka-acls.sh --consumer-group '*' --add --allowprincipals user:atlan --operations Read,Describe --allow-host '*' --config /<broker-config-path>/config/server.properties

# Give permission to cluster
./bin/kafka-acls.sh --cluster --add --allowprincipals user:atlan --operations Describe,DescribeConfigs --allow-host '*' --config /<broker-config-path>/config/server.properties