How to set up a private network link to Amazon MSK

AWS PrivateLink creates a secure, private connection between services running in AWS, ensuring that traffic between services remains within the AWS network. This document describes the steps to set this up between Amazon MSK and Atlan.

Prerequisites

Before you can set up private network connectivity, ensure the following:

• Amazon MSK version: Apache Kafka 2.7.1 or higher.
• Authentication type: only IAM role-based authentication is supported.
• Cluster instance type: must be larger than t3.small.
• Region alignment: both your Amazon MSK cluster and Atlan tenant must reside in the same AWS region.

For more information, refer to Requirements and Limitations for Multi-VPC Private Connectivity.

Request Atlan's details

To configure private network connectivity between your AWS account and Atlan, contact Atlan support for the following details:

• Atlan's AWS account ID

Enable private network link

To verify or enable AWS PrivateLink for Amazon MSK:

1. Sign in to the AWS Management Console and open the Amazon MSK Console.
2. From the left menu, click Clusters.
3. On the Clusters page, under Cluster name, select the cluster for which you want to enable private network link.
4. On your cluster page, below the overview section, click the Properties tab.
5. In the Properties tab, navigate to the Networking settings section to verify or enable AWS PrivateLink connectivity:
   
   • If you have verified that AWS PrivateLink is turned on, skip to Grant access to Atlan.
   • If AWS PrivateLink is turned off, click the Edit button and then click Turn on multi-VPC connectivity to enable it.
     1. In the Turn on multi-VPC connectivity page, for Authentication type, click IAM role-based authentication.
     2. At the bottom of the screen, click Turn on selection. The cluster will undergo a rolling update, which may take several minutes to a few hours to complete.

Grant access to Atlan

Once AWS PrivateLink is enabled for your Amazon MSK cluster, you will need to update the cluster policy to grant access to Atlan.

To update your Amazon MSK cluster policy:

1. Sign in to the AWS Management Console and open the Amazon MSK Console.
2. From the left menu, click Clusters.
3. On the Clusters page, under Cluster name, select the cluster for which you enabled private network link.
4. On your cluster page, below the overview section, click the Properties tab.
5. In the Properties tab, navigate to the Security settings section and then click Edit cluster policy.
6. In the Edit cluster policy page, under Cluster policy, configure the following:
   1. Click Basic as the new cluster policy.
   2. For Account ID(s) that need cluster access, enter Atlan's AWS account ID.
   3. Click the Include Kafka service principal checkbox to allow Atlan access to Kafka services only.
   4. Click Save changes to save your selections.

Notify Atlan support team

Once you've completed the steps above, contact the Atlan support team again and provide the following details for your Amazon MSK cluster:

• Amazon MSK Cluster ARN — the unique identifier of your cluster

Atlan will create a managed VPC connection to your Amazon MSK cluster. Once completed, Atlan support will send you the cluster connection string (bootstrap servers) required for accessing Amazon MSK via AWS PrivateLink.

You can now enter the cluster connection string for the Bootstrap servers field to crawl Amazon MSK. Atlan will securely connect to your Amazon MSK cluster using AWS PrivateLink.