How to set up Databricks

Set up authentication for your Databricks instance using one of the following two options:

Interactive cluster

Confirm cluster setup

To confirm an all-purpose interactive cluster is configured:

1. From the left menu of any page of your Databricks instance, click Compute.
2. Under the All-purpose clusters tab ensure you have a cluster defined.
3. Click the link under the Name column of the table to open your cluster.
4. Under the Configuration tab, ensure the Autopilot options to Terminate after ... minutes is enabled.
5. At the bottom of the Configuration tab, expand the Advanced options expandable.
   1. Under the Advanced options expandable, open the JDBC/ODBC tab.
   2. Confirm that all of the fields in this tab are populated, and copy them for use in crawling: Server Hostname, Port, and HTTP Path.

Generate a personal access token

To generate a personal access token:

1. From the left menu of any page of your Databricks instance, at the bottom, click Settings and then User Settings.
2. Under the Access tokens tab, click the Generate new token button.
3. In the Generate new token dialog:
   1. For Comment enter a description of the token's intended use (for example, Atlan crawler).
   2. For Lifetime (days) consider removing the number. This will allow the token to be used indefinitely — it will not need to be refreshed. If you do enter a number, remember that you will need to generate a new token periodically and enter
      🚨 Careful! If you do enter a number, remember that you will need to periodically regenerate it and update Atlan's crawler configuration with the new token each time.
   3. At the bottom of the dialog click Generate.
4. Copy and save the generated token somewhere, and then click Done.

SQL warehouse (formerly SQL endpoint)

Confirm warehouse setup

To confirm a SQL warehouse is configured:

1. From the left menu of any page of your Databricks instance, open the drop-down just below the databricks logo and change to SQL.
2. From the refreshed left menu, click SQL Warehouses.
3. Click the link under the Name column of the table to open your SQL warehouse.
4. Under the Connection details tab, confirm that all of the fields are populated and copy them for use in crawling: Server hostname, Port, HTTP path.

Generate a personal access token

To generate a personal access token:

1. In the lower-right corner of the Connection details tab of the SQL warehouse, click the link to Create a personal access token.
2. In the resulting User Settings page, under the Personal access tokens tab, click the Generate new token button.
3. In the Generate token dialog:
   1. For Comment enter a description of the token's intended use (for example, Atlan crawler).
   2. For Lifetime (days) consider removing the number. This will allow the token to be used indefinitely — it will not need to be refreshed. If you do enter a number, remember that you will need to generate a new token periodically and enter
      🚨 Careful! If you do enter a number, remember that you will need to periodically regenerate it and update Atlan's crawler configuration with the new token each time.
   3. At the bottom of the dialog click Generate.
4. Copy and save the generated token somewhere, and then click Done.

(Optional) Set permissions for Unity Catalog

If you're managing permissions selectively through Unity Catalog, then Atlan will require certain privileges to crawl metadata for tables and views.

The caller must be a metastore admin or an owner of (or have the SELECT privilege on) the tables. For the latter case, the caller must also be the owner or have the USE_CATALOG privilege on the parent catalog and the USE_SCHEMA privilege on the parent schema.