How to set up Databricks

Have more questions? Submit a request
šŸ¤“ Who can do this? Check that you have Admin and Databricks SQL access for the Databricks workspace. This is required for both authentication options described below. If you do not have this access, contact your Databricks administrator.

Set up authentication for your Databricks instance using one of the following two options:

Interactive cluster

Confirm cluster setup

To confirm an all-purpose interactive cluster is configured:

  1. From the left menu of any page of your Databricks instance, click Compute.
  2. UnderĀ theĀ All-purpose clusters tab ensure you have a cluster defined.
  3. Click the link under theĀ Name column of the table to open your cluster.
  4. Under theĀ Configuration tab, ensure theĀ Autopilot options toĀ Terminate after ... minutes is enabled.
  5. At the bottom of the Configuration tab, expand the Advanced options expandable.
    1. Under theĀ Advanced options expandable, open theĀ JDBC/ODBC tab.
    2. Confirm that all of the fields in this tab are populated, and copy them for use in crawling:Ā Server Hostname,Ā Port, and HTTP Path.

Generate a personal access token

To generate a personal access token:

  1. From the left menu of any page of your Databricks instance, at the bottom, clickĀ Settings and thenĀ User Settings.
  2. Under the Access tokens tab, click the Generate new token button.
  3. In theĀ Generate new token dialog:
    1. ForĀ Comment enter a description of the token's intended use (for example, Atlan crawler).
    2. ForĀ Lifetime (days) consider removing the number. This will allow the token to be used indefinitely ā€” it will not need to be refreshed. If you do enter a number, remember that you will need to generate a new token periodically and enter
      šŸšØ Careful! If you do enter a number, remember that you will need to periodically regenerate it and update Atlan's crawler configuration with the new token each time.
    3. At the bottom of the dialog click Generate.
  4. Copy and save the generated token somewhere, and then clickĀ Done.

SQL warehouse (formerly SQL endpoint)

Confirm warehouse setup

To confirm a SQL warehouse is configured:

  1. From the left menu of any page of your Databricks instance, open the drop-down just below theĀ databricks logo and change toĀ SQL.
  2. From the refreshed left menu, clickĀ SQL Warehouses.
  3. Click the link under theĀ Name column of the table to open your SQL warehouse.
  4. Under theĀ Connection details tab, confirm that all of the fields are populated and copy them for use in crawling:Ā Server hostname,Ā Port,Ā HTTP path.

Generate a personal access token

To generate a personal access token:

  1. In the lower-right corner of theĀ Connection details tab of the SQL warehouse, click the link to Create a personal access token.
  2. In the resultingĀ User Settings page, under theĀ Personal access tokens tab, click theĀ Generate new token button.
  3. In theĀ Generate token dialog:
    1. ForĀ Comment enter a description of the token's intended use (for example,Ā Atlan crawler).
    2. ForĀ Lifetime (days) consider removing the number. This will allow the token to be used indefinitely ā€” it will not need to be refreshed. If you do enter a number, remember that you will need to generate a new token periodically and enter
      šŸšØ Careful! If you do enter a number, remember that you will need to periodically regenerate it and update Atlan's crawler configuration with the new token each time.
    3. At the bottom of the dialog clickĀ Generate.
  4. Copy and save the generated token somewhere, and then clickĀ Done.

(Optional) Set permissions for Unity Catalog

If you're managing permissions selectively through Unity Catalog, then Atlan will require certain privileges to crawl metadata for tables and views.

The caller must be a metastore admin or an owner of (or have the SELECT privilege on) the tables. For the latter case, the caller must also be the owner or have the USE_CATALOG privilege on the parent catalog and the USE_SCHEMA privilege on the parent schema.

Related articles

Was this article helpful?
1 out of 1 found this helpful