Atlan supports the basic authentication method for fetching metadata from Apache Kafka. This method uses a username and password to fetch metadata.
The SASL/SCRAM authentication method is currently not supported.
Create user in Apache Kafka
We recommend creating a new user in Apache Kafka for integration with Atlan. Next, you will need to implement authentication and authorization for that user on your Apache Kafka brokers.
To enable user authentication and authorization on the broker side, you will need to perform two steps on each broker:
- Configure valid credentials
- Configure the proper security protocol and authorizer implementation
You can configure the broker with its user credentials and then authorize the client's user credentials. These credentials, along with the login module specification, are stored in a JAAS login configuration file.
Here is the JAAS file you can run:
KafkaServer {
org.apache.kafka.common.security.plain.PlainLoginModule required
username="<kafka admin username>"
password="<kafka admin password>"
user_<username> = β<password>β;
};
- Replace
username
with the username you want to use in Atlan. - Replace
password
with the password you want to use in Atlan.
Pass in this file as a JVM configuration option when running the broker, using -Djava.security.auth.login.config=[path_to_jaas_file]
:
[path_to_jaas_file]
can be something like:config/jaas-kafka-server.conf
- This can be done by setting the
KAFKA_OPTS
environment variable, for example:export KAFKA_OPTS="-Djava.security.auth.login.config=<path-to-jaas-file>/jaas-kafka-server.conf"
Set permissions
The following minimum permissions are required for these resources:
- Topic:
Describe
,DescribeConfigs
- Group:
Describe
- Cluster:
DescribeConfigs
To set the minimum permissions required to crawl Apache Kafka:
# Give permission to topics
./bin/kafka-acls.sh --topic '*' --add --allowprincipals user:atlan --operations Read,Describe,DescribeConfigs --allow-host '*' --config /<broker-config-path>/server.properties
# Give permission to groups
./bin/kafka-acls.sh --consumer-group '*' --add --allowprincipals user:atlan --operations Read,Describe --allow-host '*' --config /<broker-config-path>/config/server.properties
# Give permission to cluster
./bin/kafka-acls.sh --cluster --add --allowprincipals user:atlan --operations Describe,DescribeConfigs --allow-host '*' --config /<broker-config-path>/config/server.properties