How to set up Apache Kafka

Atlan supports different authentication mechanisms to securely access your Apache Kafka cluster. If the cluster is configured with "No Auth" (authentication not needed), Atlan connects directly. If the cluster requires authentication, you must configure it for Atlan to access your Apache Kafka cluster.

Atlan supports the following authentication flows:

• Basic authentication using a username and password with SASL_PLAIN mechanism
• SCRAM authentication using a username and password with SASL_SCRAM mechanism

Use basic authentication with SASL_PLAIN mechanism

With basic authentication using SASL_PLAIN, Atlan authenticates with Kafka using a username and password. To authenticate Atlan with Kafka using SASL_PLAIN, complete the following steps on each broker:

1. Create user by defining the user credentials in a `JAAS` login configuration file:

   KafkaServer {
       org.apache.kafka.common.security.plain.PlainLoginModule required
       username="<kafka admin username>"
       password="<kafka admin password>"
       user_<username> = "<password>";
   };

   • Replace <kafka admin username> and <kafka admin password> with the administrator credentials for Kafka.
   • Replace <username> with the username you want to use in Atlan.
   • Replace <password> with the password you want to use in Atlan.

2. Pass the JAAS file as a JVM configuration option when running the broker:

   export KAFKA_OPTS="-Djava.security.auth.login.config=<path-to-jaas-file>/jaas-kafka-server.conf"

3. Atlan does not make any API requests or queries that update the resources in your Kafka cluster. Set the minimum necessary permissions for resources:

   • Grant topic permissions to read and describe topics with the following command:

     ./bin/kafka-acls.sh --topic '*' --add --allow-principals user:atlan --operations Read,Describe,DescribeConfigs --allow-host '*' --config /<broker-config-path>/server.properties

   • Grant consumer group permissions to read and describe consumer groups with the following command:

     ./bin/kafka-acls.sh --consumer-group '*' --add --allow-principals user:atlan --operations Read,Describe --allow-host '*' --config /<broker-config-path>/config/server.properties

   • Grant cluster permissions to describe cluster configurations with the following command:

     ./bin/kafka-acls.sh --cluster --add --allow-principals user:atlan --operations Describe,DescribeConfigs --allow-host '*' --config /<broker-config-path>/config/server.properties

4. Once you have configured the Apache Kafka brokers, restart your brokers to pick up the new configuration.

Use SCRAM authentication with SASL_SCRAM mechanism

SCRAM (Salted Challenge Response Authentication Mechanism) provides more security than SASL_PLAIN. To use SCRAM authentication, complete the following steps on each broker:

1. Create the SCRAM user using SCRAM-SHA-256 or SCRAM-SHA-512 mechanism based on the mechanism set up on your Apache Kafka cluster:

   • Use SCRAM-SHA-256 mechanism:

     bin/kafka-configs.sh --zookeeper localhost:2181 --alter --entity-type users --entity-name <username> --add-config "SCRAM-SHA-256=[iterations=4096,password=<password>]"

     • Replace <username> with the username you want to use in Atlan.
     • Replace <password> with the password you want to use in Atlan.

   • Use SCRAM-SHA-512 mechanism:

     kafka-configs.sh --zookeeper localhost:2181 --alter --entity-type users --entity-name <username> --add-config "SCRAM-SHA-512=[iterations=4096,password=<password>]"

     • Replace <username> with the username you want to use in Atlan.
     • Replace <password> with the password you want to use in Atlan.

   • Use both SCRAM-SHA-256 and SCRAM-SHA-512 mechanisms:

     kafka-configs.sh --zookeeper localhost:2181 --alter --entity-type users --entity-name <username> --add-config "SCRAM-SHA-256=[iterations=4096,password=<password>],SCRAM-SHA-512=[iterations=4096,password=<password>]"

     • Replace <username> with the username you want to use in Atlan.
     • Replace <password> with the password you want to use in Atlan.

2. Verify the user configuration:

   kafka-configs.sh --zookeeper localhost:2181 --describe --entity-type users --entity-name <username>

   • Replace <username> with the username you want to use in Atlan.

3. The SCRAM authentication needs a JAAS file. If the file doesn't exist, create one with the following content:

   KafkaServer {
       org.apache.kafka.common.security.plain.PlainLoginModule required
       username="<kafka admin username>"
       password="<kafka admin password>"
   };

4. Pass in the JAAS file as a JVM configuration option when running the broker:

   export KAFKA_OPTS="-Djava.security.auth.login.config=<path-to-jaas-file>/jaas-kafka-server.conf"

5. Atlan does not make any API requests or queries that update the resources in your Kafka cluster. Set the minimum necessary permissions for resources:

   • Grant topic permissions to read and describe topics with the following command:

     ./bin/kafka-acls.sh --topic '*' --add --allow-principals user:atlan --operations Read,Describe,DescribeConfigs --allow-host '*' --config /<broker-config-path>/server.properties

   • Grant consumer group permissions to read and describe consumer groups with the following command:

     ./bin/kafka-acls.sh --consumer-group '*' --add --allow-principals user:atlan --operations Read,Describe --allow-host '*' --config /<broker-config-path>/config/server.properties

   • Grant cluster permissions to describe cluster configurations with the following command:

     ./bin/kafka-acls.sh --cluster --add --allow-principals user:atlan --operations Describe,DescribeConfigs --allow-host '*' --config /<broker-config-path>/config/server.properties

6. Once you have configured the Apache Kafka brokers, restart your brokers to pick up the new configuration.