What are the known limitations of the Databricks connector?
Primary and Foreign Keys
Atlan does not crawl primary keys (PK) and foreign keys (FK) for Databricks. For more information, see What column keys does Atlan crawl?
Does Atlan consider expensive queries and compute costs?
No, Atlan does not factor in expensive queries or compute costs due to limitations in the Databricks APIs, which do not expose this information.
How does Atlan calculate popularity for Databricks assets?
Atlan calculates popularity for tables, views, and columns in Databricks by analyzing query execution data. It retrieves query history from the system.query.history table and specifically filters for execution_status = 'FINISHED' and statement_type = 'SELECT' to determine how frequently assets are accessed.
How to debug test authentication and preflight check errors?
Hostname resolution error
Provided Host name cannot be resolved via DNS, please check and try again.
- The hostname you have provided cannot be resolved through DNS. Ensure that the hostname is correct.
- Verify that the DNS settings have been configured properly.
Invalid client ID or secret
Provided Client ID is invalid, please check and try again.
- The client ID or secret you have provided is either invalid or no longer working. Follow the steps for AWS or Azure setup to generate new credentials.
Invalid tenant ID
Provided tenant ID is invalid, please check and try again.
- The tenant ID you have provided is incorrect.
- Ensure that the tenant ID you have provided corresponds to the one in your Microsoft Entra ID application.
Unity Catalog not linked
Configured Databricks instance doesn't have Unity Catalog linked. Please choose
JDBC extraction instead of REST API in Atlan.
- If you have not set up Unity Catalog in your Databricks workspace, you can change the extraction method to JDBC instead of REST API to crawl your Databricks assets in Atlan.
Connection timeout
Failed to connect to Databricks (connection timed out). Please check your host
and port and try again.
- The connection to the Databricks instance has timed out.
- Verify that the host and port are correct.
- Ensure that no firewall rules or network issues are blocking the connection.
Invalid HTTP path
Provided HTTP path is invalid, please check and try again.
- The HTTP path you have provided is invalid.
- Ensure that the endpoint is properly configured and accessible, and the warehouse ID in the HTTP path is correct.
Invalid personal access token
PAT token is invalid, please check and try again.
- The personal access token used for authentication is invalid.
- Ensure that the token is valid and neither deleted nor expired.
- You can also generate a new personal access token, if needed.
General connection failure
Unable to connect to the configured Databricks instance, please check your credentials
and configs and then try again. If the problem persists, contact
support@atlan.com.
- Ensure that you have entered the host and port correctly.
- Verify that the credentials for the connection are correct.
- Ensure that your Databricks instance is properly configured and accessible.
- If the problem still persists after verifying all of the above, contact Atlan support.
Why does the workflow take longer than usual in the extraction step?
- Certain Databricks runtime versions do not have an easy way to extract some metadata (for example partitioning, table_type, and format). Thus we must perform extra operations to retrieve these, resulting in slower performance.
- If you are not already, you may want to try the Unity Catalog extraction method.
Why is some metadata missing?
- We currently cannot extract some metadata from Databricks:
Metadata JDBC REST API ViewCountandTableCount(on schemas)❌ ✅ RowCount(on tables and views)❌ ❌ TABLE_KIND(on tables and views)❌ ❌ PARTITION_STRATEGY(on tables and views)❌ ❌ CONSTRAINT_TYPE(on columns)❌ ❌ Partition key (on columns) ❌ ✅ Table partitioning information ✅ ❌ BYTES,SIZEINBYTES(table size)❌ ❌ - We'll be exploring ways to bring this metadata into Atlan if Databricks supports extraction of the metadata.
Why doesn't my SQL work when querying Databricks?
- We currently support SparkSQL on Databricks runtime 7.x and above.
Can I use Atlan when the Databricks compute engine is not running?
- Atlan needs the Databricks compute engine to be running for two activities:
- Crawling assets (normal and scheduled run)
- Querying assets (including data previews)
- If you do not need to perform the activities above, your experience shouldn't be affected.
- In any other case, you'll get a downgraded experience on Atlan if the compute engine is not running. Queries won't work as expected and a scheduled workflow might fail after a couple of retries.
- We recommend turning off the Terminate after x minutes of inactivity option in your cluster to avoid these problems. If you have this turned on, any of the above activities should trigger the cluster to come back online within about 30 seconds.
Why can't I see all the assets on Atlan that are available in Databricks?
- Have you excluded the database or schema when crawling?
- Does the Databricks user you configured for crawling have access to these other assets?
Why is the test authentication taking so long?
- Please check the state of the compute engine. It must be in a running state for all operations, including authentication.
What limitations are there with the REST API (Unity Catalog) extraction method?
- We currently do not support schema-level filtering and retrieving table partitioning information.
Why has my workflow started to fail when it worked before?
- This can happen if the PAT you configured the workflow with has since expired.
- You will need to create a new PAT in Databricks, and then modify the workflow configuration in Atlan with this new PAT.
- If you are unable to update the PAT, pause the workflow and reach out to us.
How do I migrate to Unity Catalog?
- Currently Unity Catalog is in a public preview state.
- The Databricks team is working on an automated migration to Unity Catalog.
- Currently you must migrate individual tables manually.