How to set up Apache Kafka

πŸ€“ Who can do this? You will probably need your Apache Kafka administrator to run these commands β€” you may not have access yourself.

Atlan supports the basic authentication method for fetching metadata from Apache Kafka. This method uses a username and password to fetch metadata.

The SASL/SCRAM authentication method is currently not supported.

Create user in Apache Kafka

We recommend creating a new user in Apache Kafka for integration with Atlan. Next, you will need to implement authentication and authorization for that user on your Apache Kafka brokers.

To enable user authentication and authorization on the broker side, you will need to perform two steps on each broker:

  • Configure valid credentials
  • Configure the proper security protocol and authorizer implementation

You can configure the broker with its user credentials and then authorize the client's user credentials. These credentials, along with the login module specification, are stored in a JAAS login configuration file.

Here is the JAAS file you can run:

KafkaServer {
    org.apache.kafka.common.security.plain.PlainLoginModule required
    username="<kafka admin username>"
    password="<kafka admin password>"
    user_<username> = β€œ<password>”;
};
  • Replace username with the username you want to use in Atlan.
  • Replace password with the password you want to use in Atlan.

Pass in this file as a JVM configuration option when running the broker, using -Djava.security.auth.login.config=[path_to_jaas_file]:

  • [path_to_jaas_file] can be something like: config/jaas-kafka-server.conf
  • This can be done by setting the KAFKA_OPTS environment variable, for example:
    export KAFKA_OPTS="-Djava.security.auth.login.config=<path-to-jaas-file>/jaas-kafka-server.conf"
🚨 Careful! Once you have configured the Apache Kafka brokers, you may need to restart your brokers to pick up the new configuration.

Set permissions

πŸ’ͺ Did you know? Atlan does not make any API requests or queries that will update the resources in your Kafka environment.

The following minimum permissions are required for these resources:

  • Topic: Describe, DescribeConfigs
  • Group: Describe
  • Cluster: DescribeConfigs

To set the minimum permissions required to crawl Apache Kafka:

# Give permission to topics
./bin/kafka-acls.sh --topic '*' --add --allowprincipals user:atlan --operations Read,Describe,DescribeConfigs --allow-host '*' --config /<broker-config-path>/server.properties

# Give permission to groups
./bin/kafka-acls.sh --consumer-group '*' --add --allowprincipals user:atlan --operations Read,Describe --allow-host '*' --config /<broker-config-path>/config/server.properties

# Give permission to cluster
./bin/kafka-acls.sh --cluster --add --allowprincipals user:atlan --operations Describe,DescribeConfigs --allow-host '*' --config /<broker-config-path>/config/server.properties

Related articles

Was this article helpful?
0 out of 0 found this helpful