AWS PrivateLink creates a secure, private connection between services running in AWS. This document describes the steps to set this up between Databricks and Atlan.
🤓 Who can do this? You will need Databricks support, and probably your Databricks administrator involved — you may not have access or the tools to run these tasks.
Prerequisites
- Databricks must be set up on the E2 version of the platform and Enterprise pricing tier.
- Your Databricks workspace must be in an AWS region that supports the E2 version of the platform, and not the
us-west-1region. Your Databricks workspace must also be hosted in the same region as Atlan. - Your Databricks workspace must use customer-managed VPC. (Note that you cannot update an existing Databricks-managed VPC to a customer-managed VPC.)
For all details, see Databricks documentation.
Notify Atlan support
Once setup is completed, provide Atlan support with the following information:
- The AWS region of your Databricks instance.
There are additional steps that Atlan will need to complete:
- Creating a security group
- Creating an endpoint
Once the Atlan team has confirmed that the configuration is ready, please continue with the remaining steps.
Accept the endpoint connection request
You can either:
- Accept the endpoint connection request from Atlan via API.
- Accept the endpoint connection request from Atlan from the Databricks console.
Once the endpoint connection is accepted, Atlan support will finish the configuration on the Atlan side.
When you use this endpoint in the configuration for crawling Databricks, Atlan will connect to Databricks over AWS PrivateLink.